AumCrawl – AI Crawler Control: See and Block AI Bots

Every publisher is now asking the same question and nobody can answer it: is ChatGPT actually reading my site? This plugin answers it. It records every known crawler that reaches your site - AI, search engine, SEO tool, link preview - and shows you who came, when, and which pages they read. Then it lets you turn away the ones that give you nothing in return. AI crawlers are not one thing Most "block AI" plugins block everything, and that is the wrong move. AI crawlers split into two groups that behave in opposite ways. Some send traffic back. OAI-SearchBot, PerplexityBot, Claude-SearchBot and Google-Extended fetch a page because a person asked a question, and the answer credits you with a link. Blocking them costs you visits. Some take and give nothing. GPTBot, ClaudeBot, CCBot, Bytespider and the SEO backlink crawlers pull your content into a training corpus or a commercial dataset. No link, no citation, no visitor. The plugin sorts every crawler into those groups and puts the switch next to the evidence, so you are deciding with numbers in front of you rather than in the abstract. What it will not let you do There is deliberately no switch for Googlebot, Bingbot, or the crawlers that build link previews on Facebook, X and LinkedIn. Turning one of those off does real damage, and that belongs in your own robots.txt rather than behind a button you can hit by accident. They are recorded, so you can still see when Google last came by. Honest about what each control does

• Rules in robots.txt - honoured by every crawler listed in this plugin, and ignored by anything that chooses to. It is a request, not a fence.
• Turning blocked crawlers away - the strongest option. The page is never served. Only works against crawlers that say who they are.
• A "do not train on this" note - the X-Robots-Tag: noai convention. Weak. Some tools read it, many do not. It costs nothing, so it is there, labelled for what it is.

The settings screen shows which of these is genuinely in effect right now. It does not assume a filter worked: it fetches your live robots.txt and looks. If another plugin is replacing the file, or a robots.txt exists on disk, you are told so, told which plugin is responsible, and given the exact lines to paste instead. Are they really who they say they are? Anyone can send User-Agent: GPTBot. The plugin checks the claim against the hostname each operator publishes, using forward-confirmed reverse DNS, and shows you which visits could not be verified. No IP addresses are stored. The address is used inside that one request and then discarded. If a check fails, only a shortened network range is kept - the last part of the address is removed first - which is enough to see that a range is impersonating a crawler without ever recording who connected. Works alongside your SEO plugin robots.txt is crowded, so this plugin adds to it rather than taking it over.

• Yoast SEO - rules are appended after Yoast's, never replacing them
• Rank Math - same, and if Rank Math is set to replace robots.txt entirely you are told, and given the block to paste into its editor
• All in One SEO - same
• SEOPress, Slim SEO, Squirrly, Better Robots.txt - appended
• If any plugin already writes a rule for a crawler, this one stands aside and says so rather than writing a second, conflicting rule

It does not generate llms.txt. That is a different job, and other plugins already do it. Crawlers covered Sends traffic back: OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User, Claude-User, Claude-SearchBot, Google-Extended. Takes and gives nothing back: GPTBot, ClaudeBot, CCBot, Bytespider, Applebot-Extended, Meta-ExternalAgent, Amazonbot, Diffbot, omgilibot, AhrefsBot, SemrushBot, MJ12bot, DotBot, DataForSeoBot. Recorded only: Googlebot, Bingbot, DuckDuckBot, Applebot, facebookexternalhit, Twitterbot, LinkedInBot. New crawlers appear faster than plugin releases, so the list is filterable. See the FAQ. No account, no service, no cost Nothing is sent anywhere. There is no API key to obtain, no external service, and no paid tier. The plugin reads your own traffic and writes its own robots.txt rules. Source code The released source is on GitHub at github.com/aumcreate/aumcrawl — bug reports and pull requests are welcome there. More from AumCreate The full list of crawlers it knows, how verification works, and what blocking actually changes: aumcreate.com/plugins/aumcrawl Also free from AumCreate: llms.txt and schema for AI search, AI translation for WordPress, digital-goods checkout for WooCommerce, and a hosted chat widget: aumcreate.com/plugins

1. Install and activate.
2. Go to Settings → AI Crawlers.
3. Leave it for a day or two, then come back and see who visited.

Nothing is blocked until you switch something off, so activating the plugin changes nothing about how your site is crawled.

1.1.0

• Twelve crawlers can now be confirmed that could not be before. The identity check knew one method — looking up the name an address resolves back to — and that only works for operators who publish such a name. The older search engines do; nobody newer does. OpenAI, Anthropic, Perplexity and DuckDuckGo publish lists of their addresses instead, so every visit from them was reported as "unknown" forever: the honest answer, and a useless one. Those lists are now read and checked against, so a real Claude or ChatGPT visit is confirmed and a forged one is caught.
• The lists are fetched in the background on a schedule, never while a visitor is being served. Fetching a remote file during a page load would put somebody else's server in the path of yours.
• An address missing from a list is not treated as forged unless the copy of the list is recent. Operators add addresses without announcing it, and a list we fetched yesterday is a snapshot. Saying "cannot tell" about a real crawler costs nothing; saying "forged" about one is what this plugin did until 1.0.9.
• Where an operator publishes both a hostname and a list, the hostname decides, and the list settles the cases where an address has no hostname at all — which is most of Bing's.

1.0.10

• AhrefsBot was reported as an impostor whenever it came from one half of its own network. Ahrefs documents two hostname suffixes — ahrefs.com and ahrefs.net — and only the first was recorded here, so every visit from the second failed the identity check. Both are recorded now.
• SemrushBot's entry had no basis. Semrush's crawler page does not document a hostname to check against, so there was nothing behind the one recorded here. It is empty now and the check reports "unknown", which is the truth.
• Each remaining entry now carries the address of the operator's own documentation and the date it was read. A fact about a crawler with no date on it stops being trustworthy within a year, because operators change how they identify themselves; and an entry with no source is one nobody can check.

1.0.9

• Eight crawlers were being reported as impostors when they were genuine. The identity check confirms a crawler by looking up the hostname its address resolves back to, which only works for operators that publish such hostnames. OpenAI, Anthropic and Perplexity do not — they publish lists of addresses instead — and this plugin had hostnames recorded for them anyway. Their crawlers run on cloud infrastructure whose addresses resolve back to the cloud provider, so the check saw a hostname that did not match and concluded the visit was forged. Anthropic's and Perplexity's crawlers were listed under "Failed the identity check" on every site running this plugin, and Perplexity's are in the group this plugin tells you not to block because they send readers back to you. Those entries are now empty, so the check reports "unknown" for them, which is the truth: these operators do not publish what the check needs. Verification from their published address lists is a separate piece of work.
• No crawler was blocked by this. The check only labels what it sees.

1.0.8

• The plugins list now carries a Guide link beside Deactivate, pointing at this plugin's manual in the language that administrator reads the admin in. The manual is the product's full documentation; it is a separate thing from the plugin's own help text.

1.0.7

• Fixed: on a site with no AumViso history to import, the one-off import check queried three tables that do not exist on every admin page load, filling debug.log with database errors. It now asks whether the tables exist first, and records that there was nothing to import.

1.0.6

• The links in the description are now real links. They were plain text, because wordpress.org does not turn a bare address into a link.

1.0.5

• Added a short section pointing to the plugin's own page on aumcreate.com and to the other free AumCreate plugins. No code changes.

1.0.4

• Crawler settings and visit history from AumViso are imported automatically the first time this plugin runs on a site that had them — AumViso no longer carries the crawler feature, and without this its blocking rules would simply stop applying. Nothing is imported if you have already configured this plugin, and the notice that reports the import offers to undo it.
• The link at the foot of the settings screen now goes to the plugin's own page on aumcreate.com instead of the site's front page.

1.0.3

• Added a link from the Plugins list to the plugin's page on aumcreate.com.

1.0.2

• Tags and summary now name what people search for. No functional change to the plugin.
• A line at the foot of the settings screen linking to the rest of the AumCreate ecosystem. Older entries are in changelog.txt, which ships with the plugin.