Softglaze AI Crawler Monitor

Softglaze AI Crawler Monitor records every request from a known AI crawler that reaches WordPress. It shows you which crawlers came, what they asked for, and what your site told them at the time. Most plugins in this area stop at writing a rule. This one is built around the gap between writing a rule and the rule working, because on the open web those are not the same thing. You can find out:

• Which AI crawlers requested pages on your site, how often, and which pages.
• What each crawler was told, and whether it took any notice.
• Which requests were refused, if you chose to refuse them.

What it can and cannot do There are three ways to tell a crawler to stay away. Only one of them is enforced, and the plugin repeats that on every screen where you make the choice.

1. A robots.txt rule. This is a request, not a barrier. Well behaved crawlers honour it. One that ignores it can still fetch your pages, and nothing on your server stops it.
2. An X-Robots-Tag: noai, noimageai header, sent only on requests from that crawler. Also a request, and a weaker one, because noai is not an agreed standard and most crawlers ignore it.
3. Refusing the request with a 403. This is the only setting that is actually enforced. A matching request gets an empty "403 Forbidden" and the page is never sent.

The header never contains noindex, nofollow or none under any setting. Those would take your pages out of ordinary Google and Bing results, which is not what anyone means when they say they want to block AI. Three crawlers in the list are documented by their own operators as fetching pages when a person asks a question, with robots.txt rules that may not apply to them: ChatGPT-User, Perplexity-User and Meta-ExternalFetcher. All three are marked on the crawler screen, right next to the setting, so you find out before you rely on it rather than afterwards. Your robots.txt is always served, even to a crawler you are refusing. Blocking a crawler from reading robots.txt would leave it no way to learn that you want it to stay away. Per-content control The site-wide setting is the default. Any single post, page or product can override it from the edit screen, from quick edit, or from bulk edit. A column in the posts list shows which content has its own setting, and you can filter by it. A per-post setting sends the X-Robots-Tag header and does nothing else. It adds no lines to robots.txt, it cannot refuse a request, and it cannot lift a refusal you set site-wide. The edit screen says so in those words. What it cannot see The plugin records requests that reach WordPress. Plenty of requests never get that far, and rather than quietly under-report, it tells you so:

• Requests your CDN or firewall answers before they reach your server.
• Requests served from a page cache. On a test server with page caching on, only one request in five reached WordPress at all.
• Static files. Images, stylesheets, scripts and fonts are served without running WordPress.
• Addresses on your domain that WordPress does not handle.
• Any crawler that is not in this plugin's list.

There is a banner above every figure in the plugin saying this. You cannot dismiss it, because it changes what the numbers mean. Crawler identity checking A user agent is self declared, so anything can claim to be GPTBot. The plugin can check whether a crawler really is what it says. Both checks are switched off until you turn them on, because both mean contacting a server outside your site. Left off, the plugin still does everything else, and crawlers are recorded as "not checked", which is what they honestly are. A crawler is only ever recorded as having failed a check when a check actually ran and gave a negative answer. A check that could not run is never treated as a failure. Checking only works where the operator publishes something to check against. Eleven of the crawlers in the list can be checked. The rest stay "not checked" however you set these options, because their operators publish nothing to check them with, and the crawler screen tells you which is which. Crawlers included in this version GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Claude-SearchBot, Google-Extended, PerplexityBot, Perplexity-User, meta-externalagent, Meta-WebIndexer, Meta-ExternalFetcher, Applebot, Applebot-Extended, Bytespider, CCBot, Amazonbot, Amzn-SearchBot, Amzn-User, cohere-ai, Diffbot, Diffbot-User, Timpibot, YouBot. Every one of these was checked against the operator's own published documentation, and each entry records the page it came from. The crawler screen links to that page for each one. Three of them have no published documentation at all: Bytespider, cohere-ai and Timpibot. They are in the list because the requests are real and you are entitled to see them, and each is labelled on screen as undocumented, since nobody has stated whether they read robots.txt. Crawler names and robots.txt tokens change over time. The list ships with the plugin, it is versioned, and it is never fetched from a remote server. Other plugins by SoftGlaze

• SoftGlaze PDF Invoices for WooCommerce — invoices, packing slips, proformas and credit notes from your WooCommerce orders, rendered locally.
• SoftGlaze Maintenance Mode & Coming Soon — coming soon and maintenance pages with templates, countdowns and access control.
• SoftGlaze Click to Chat — a WhatsApp chat button with multiple agents, schedules and WooCommerce support.

1. Install through Plugins, Add New, Upload Plugin, or copy the folder to /wp-content/plugins/.
2. Activate it on the Plugins screen.
3. Open "AI Crawlers" in the admin menu.

Nothing is blocked and nothing is disallowed when you activate it. The plugin starts by recording what happens and changes nothing about your site until you pick a setting, so you can look before you decide anything. On activation it writes a small helper file into wp-content/mu-plugins/ so it can identify a crawler at the earliest point WordPress allows. It removes that file again when you deactivate the plugin. If the folder cannot be written to, the plugin detects slightly later in the page load instead and tells you which mode is running. Nothing is lost either way.

0.7.0

• Renamed to Softglaze AI Crawler Monitor. The previous name described the plugin rather than identifying it, so it now leads with the Softglaze name. Every identifier moved with it, including the database tables, so this version installs as a new plugin rather than upgrading the old one.
• The helper file in wp-content/mu-plugins/ no longer assumes the plugin folder's name when locating the plugin. It uses the real path recorded when the file was written, and the plugin rewrites that path by itself if the folder is ever moved or renamed.
• The "External services" section of this readme now states, for each outbound request, exactly what is sent, when, and under what condition, and separates the operator documentation links shown on screen — which the plugin never fetches — from the addresses it actually requests.

0.6.0

• Per-content control. Set AI access on a single post, page or product from the edit screen, quick edit or bulk edit, with a filterable column in the posts list.
• The URL checker now reports conflicts, meaning which crawlers requested an address after being asked not to, and which were refused.
• The URL checker also shows what each crawler is currently told about that one address, and whether the address is reachable at all.
• Six more crawlers, each checked against its operator's documentation: Meta-WebIndexer, Meta-ExternalFetcher, Applebot, Amzn-SearchBot, Amzn-User and Diffbot-User.
• The crawler screen now warns you before you disallow a crawler that feeds a product which links back to its sources, because there is no way to submit your site to those products afterwards.

0.5.0

• Every crawler in the list checked against its operator's own documentation, with the source page recorded and linked from the crawler screen.
• Crawlers whose operators say robots.txt may not apply to them are now marked as such, next to the setting.
• Crawlers whose operators publish no documentation at all are now marked as such.
• Identity checking now works wherever an operator publishes a way to check, covering ten crawlers by published address list or reverse DNS.
• FacebookBot removed, because Meta's crawler documentation no longer lists it.
• Applebot-Extended corrected to a robots.txt name only, which is what Apple documents it as. It could never have appeared in reports, and it no longer looks as though it might.

0.4.0

• Violation reporting, with separate panels for crawlers that ignored your robots.txt and requests that were refused.
• Crawler identity checking by reverse address lookup and published address lists. Both switched off by default.
• Optional removal of all data when the plugin is deleted. Off by default.

0.3.0

• Choose what each crawler is told: allow, monitor, or disallow.
• robots.txt rules and the X-Robots-Tag header for disallowed crawlers.
• Optional refusal of matching requests with a 403.
• Detection and handling of an existing robots.txt file at the site root.

0.2.0

• Dashboard showing requests per crawler over 7, 30 and 90 days, with a trend and the most requested addresses.
• URL checker for exact per-address counts.
• Cache state banner.

0.1.0

• Crawler detection and request recording.
• Daily totals and retention cleanup.