AJ Agent Crawl Optimizer – AI Readiness Scanner & llms.txt for WordPress

Run the built-in readiness scanner. AJ Agent Crawl Optimizer checks your live site against 21 AI-agent-readiness checks — the same set as Cloudflare's isitagentready.com — and grades you on a Level 0–5 ladder from Not Ready to Agent-Native. Every verdict shows the real request and response behind it, and failing checks get a one-click Fix now button that re-scans to prove it passed.

llms.txt is a plain-Markdown file at /llms.txt that gives AI agents a curated index of your most important content, proposed at llmstxt.org. AJ Agent Crawl Optimizer lets you hand-curate it: write the intro, pick which post types (and WooCommerce products) appear, and set a per-post "Summary for AI agents" that overrides the excerpt. It is a surface AI agents can route on — not a guaranteed path to AI citations, but a machine-readable map of what your site is for.

The plugin writes explicit robots.txt rules for the 15 AI crawlers readiness scanners check — GPTBot and ChatGPT-User (OpenAI), Claude-Web (Anthropic), PerplexityBot, Google-Extended, CCBot and Bytespider among them — each allowed or blocked per a per-bot policy. Presets like "Allow search & user requests, block training" make it a single choice. Remember: to be cited by an AI engine you must allow its crawler.

There is no guaranteed path, but three things help and this plugin does all three: allow the AI search crawlers (OAI-SearchBot, PerplexityBot, Google-Extended), publish the machine-readable surfaces agents look for (llms.txt, JSON-LD, Markdown negotiation), and write a clear "Summary for AI agents" on key pages. The readiness scanner then verifies each of these is actually live over real HTTP — so you fix what's broken instead of guessing.

/llms.txt is a curated index — a Markdown list of your key pages with links and short summaries, so an agent can decide what to fetch. /llms-full.txt serves the full Markdown content of those same curated entries in one file, for agents that want everything inline. The plugin serves both, and password-protected content is always excluded from each.

Most plugins only generate an llms.txt file from fixed rules. AJ Agent Crawl Optimizer is an agent-readiness scanner and fixer: it audits your live site over real HTTP against 21 checks, scores a Level 0–5 ladder, shows the request/response evidence behind every verdict, fixes failures in one click, and re-verifies. It covers the answer engine optimization (AEO) and generative engine optimization (GEO) groundwork — structured data, AI crawler rules, and MCP discovery — that plain generators and content-writing SEO plugins skip.

Yes. WooCommerce products are detected automatically and can appear as their own section in your curated llms.txt, so AI agents can see your catalogue. You control whether products appear, how many, and in what order, and you can exclude individual products or give any product a per-product "Summary for AI agents."

Dashboard is the verifier: it scans your live site over real HTTP, shows your Level 0–5 with evidence for every check, and fixes failures with one click. Settings is the switchboard: the per-feature toggles, wizard, and testing tools. A toggle can be ON in Settings while the Dashboard still shows a fail — that gap usually means a page cache or server rule is intercepting the endpoint, and the evidence view shows exactly what happened.

The scanner runs the same 21 checks (and the same Level 0–5 ladder) as Cloudflare's isitagentready.com: Level 1 needs 2 of robots.txt/sitemap/Link headers; Level 2 adds AI bot rules and Content Signals; Level 3 adds Markdown negotiation; Level 4 adds one of MCP card/Agent Skills/API catalog/A2A card; Level 5 adds two of Web Bot Auth, all four integrations, and auth metadata. Nothing counts unless the scan verifies it — enabling a toggle earns nothing until the endpoint actually responds. Commerce checks are informational and never scored.

The plugin never creates files: every endpoint (/.well-known/, /llms.txt, /openapi.json, /auth.md) is served virtually by WordPress, so hosts that forbid file creation are fully supported. What can break things is the web server intercepting requests before WordPress runs — typically an nginx dot-path deny rule returning 403 on /.well-known/, or a static-file rule returning 404 for /llms.txt-style paths. The readiness scan detects exactly this (feature enabled + server blocking) and the Dashboard shows a hosting notice with copy-paste nginx and Apache fixes you can apply yourself or forward to your hosting support.

Yes — it is curated, not auto-generated from fixed rules. Under Agent Ready → llms.txt you write the intro, pick which post types appear (custom post types and WooCommerce products show up automatically), and set each section's heading, item count and order, plus your own Markdown block. A live preview shows the file as you edit, before you save. Individual entries are steered from the editor: the "Agent Ready (llms.txt)" panel has an Include in llms.txt toggle (excluded pages stay out of both /llms.txt and /llms-full.txt) and a Summary for AI agents field that overrides the excerpt for that entry — write it for a model deciding whether to fetch the page. The defaults keep the same sections and ordering as before, so upgrading needs no attention until you decide to curate.

Yes: wp agent-ready scan (add --format=json for machines or --format=agent for a markdown fix report), wp agent-ready fix <check> or --all-safe, and wp agent-ready status. The same operations are available over REST at /wp-json/ajaco/v1/scan and /wp-json/ajaco/v1/fix (admin capability required; /wp-json/ajaco/v1/health is public).

No. JSON-LD output auto-suppresses when Yoast SEO, Rank Math, All in One SEO, SEOPress, The SEO Framework, Slim SEO, Squirrly SEO, Schema Pro, or SASWP is active. The settings page shows a clear notice when this happens. Our robots_txt filter runs at PHP_INT_MAX priority, so any rules your SEO plugin adds (sitemap URLs, custom Disallow, etc.) are preserved — our Content-Signal line is appended last.

No. Browsers send Accept: text/html and get the normal HTML response. The .well-known/..., /llms.txt, and ?format=openapi endpoints are paths a normal user never visits. The only thing added to the page HTML is a single JSON-LD <script> block (when no SEO plugin is active) and a small WebMCP script tag (which no-ops in browsers without the experimental flag).

Yes. Every endpoint resolves at both the root and per-subsite paths automatically — /llms.txt, /blog/llms.txt, /.well-known/api-catalog, /blog/.well-known/api-catalog, etc. Each subsite has its own settings.

Use the ajaco_required_capability filter. The plugin defaults to manage_options (admin-only); change it to any capability of your choice, e.g. edit_pages for editors: add_filter( 'ajaco_required_capability', function () { return 'edit_pages'; } );

Check three things: (1) the IndexNow API Key field is filled in, (2) you're publishing a post of a public post type — revisions and autosaves are skipped, (3) you're on production. Pings are non-blocking, so failures are silent. Tail your server log for outbound requests to api.indexnow.org.

Yes. Use the relevant filter:

• ajaco_skill_definitions — register custom Agent Skills.
• ajaco_json_ld_graph — add custom Schema.org entries (Product, Recipe, Event, etc.).
• ajaco_llms_txt_content — append sections to or replace the llms.txt body.
• ajaco_openapi_spec — add securitySchemes, custom tags, additional servers.
• ajaco_api_catalog_linkset — add anchors or rels (e.g. for a GraphQL endpoint).
• ajaco_mcp_server_card — override transport / capabilities for a real MCP implementation.

Use the Reset to Defaults button under the Save Changes button on the settings page. It clears every plugin option, wipes cached endpoint outputs, and shows a confirmation dialog before wiping anything.

Minimal. The OpenAPI document is cached for a day in a transient and only regenerates on plugin activation/deactivation or theme switch. The llms.txt body is cached for an hour and invalidates on post edits, site name/description changes, and setting toggles. Endpoint handlers run on init only when the request path matches; on every other request they early-return after one regex check. The Markdown handler runs at PHP_INT_MAX priority on shutdown so it never breaks object-cache writes or other shutdown hooks.

The plugin doesn't include a built-in access log. To verify activity, check your server access log for User-Agents like GPTBot, ClaudeBot, OAI-SearchBot, Google-Extended, PerplexityBot, CCBot, Bytespider, etc. hitting any of the plugin endpoints.

Yes. Every feature is an independent toggle. Uncheck what you don't want and Save Changes. The corresponding behavior reverts immediately on the next request — no rewrite flush, no cache flush needed (the plugin handles cache invalidation itself).