Grubbomatic Crawl Readiness – AI Crawler Check and Fixes

ChatGPT, Claude, Perplexity and Google AI answer questions by reading websites. If they cannot read yours, you are not in the answer. Grubbomatic Crawl Readiness runs the free readiness check from crawlreadiness.com against your site and shows the result in wp-admin: a score out of 100, a plain-English verdict, and the list of things AI crawlers need that your site does not have yet. Then it fixes most of them for you, from inside WordPress:

• llms.txt — a Markdown summary of your site, built from your own pages and posts and served at /llms.txt. Nothing is written to disk and nothing goes stale.
• AI crawlers in robots.txt — a rule for every known AI crawler (OpenAI, Anthropic, Google, Perplexity, Meta, Apple and the rest) plus a Content-Signal line. Choose "allow all" or "allow AI search and assistants, block AI training".
• Meta description and Open Graph tags — a description, title, image and canonical address for every page, when you do not already run an SEO plugin.
• Structured data (JSON-LD) — Organization and WebSite for the site, Article for posts, again only when no SEO plugin owns it.
• agents.json — an optional card at /.well-known/agents.json pointing AI systems at your llms.txt, sitemap and feed.

The check itself takes about ten seconds and can be run again any time. The dashboard shows your latest score. Check first, then fix Most AI plugins write a file and hope it helps. This one looks first, from the outside, the way an AI crawler does. It tests 51 AI crawlers and scores them by what each one is for, so blocking the ones that only collect training data doesn't cost you points. Every finding is matched to the switch that fixes it, and checking again shows whether it worked. llms.txt is one of five fixes, not the point of the plugin. Works with your SEO plugin If Yoast SEO, Rank Math, All in One SEO, SEOPress, The SEO Framework or Slim SEO is active, the meta tag and structured data switches stay off so nothing is written twice. llms.txt and the robots rules still apply. Going further Being readable is step one. LLM Monitor asks ChatGPT, Claude, Perplexity and Google AI the questions your customers ask and tracks, week by week, whether they name you or a competitor. Paid plans start at $29 a month; the check and this plugin are free.

1. Install and activate the plugin.
2. Go to Tools → AI Readiness and press Check this site.
3. Turn on the fixes you want and press Save settings. Check again to see the difference.

An API key is optional. Without one, the check shares a free daily allowance with other sites on your network. A free account at crawlreadiness.com gives you a key with 50 checks a month.

1.0.0

• First release: the check, llms.txt, AI crawler rules with a Content-Signal line, meta and Open Graph tags, JSON-LD, agents.json, dashboard widget.