| 开发者 | hitmind |
|---|---|
| 更新时间 | 2026年11月18日 00:14 |
| PHP版本: | 7.4 及以上 |
| WordPress版本: | 7.2 |
| 版权: | GPLv2 or later |
| 版权网址: | 版权信息 |
User-agent: *, including the paths you blocked there (RFC 9309, section 2.2.1).
It is an easy mistake to make. In HitMind's 2026 study of the top 10,000 websites, 29.8% of the robots.txt files that block paths and name an AI crawler left those paths open to that crawler.
AI Crawler Check reads your live robots.txt the way crawlers do and shows:
User-agent: * that a named AI crawler can still fetch, and the exact lines to add to its group.__usage_ai-training line (IETF draft, October 2026), read the way the draft says: User-agent: * does not count, and a Google-Extended group that allows a path wins for Google. If you have not stated one, it shows the two lines that say no.No. It reads your robots.txt and reports what it finds. It never writes to it.
Because GPTBot has its own group in your robots.txt. A crawler reads only the group that names it and ignores User-agent: * completely once it has one. Add the listed lines to its group, or remove the group if you meant GPTBot to follow your general rules.
The table marks crawlers whose operators say they don't follow robots.txt, such as Perplexity-User, which fetches pages when a person asks for them. Rules for them state your position but are not enforced. Stopping them takes server or firewall rules.
The plugin asks your own site for /robots.txt, from your own server. Some hosts block that, and some firewalls and CDNs answer with a challenge page. Paste your file into the box at the bottom of the page to check it instead.
No. The only request it makes is to your own site's /robots.txt. See "External services" for the two links on its page.
A shared robots.txt line for saying whether your pages may be used to train AI models, from an IETF draft published in October 2026. Bing applies it from 1 February 2027; Google and OpenAI say they will support it. It is not a crawl rule: it does not stop any crawler from fetching pages, and a User-agent: * group does not count as a preference. Until each company supports it, blocking training crawlers such as GPTBot by name is what stops the fetch.
robots.txt always lives at the root of the domain, so the plugin checks that file and tells you it is not served by WordPress.
__usage_ai-training line (IETF draft, October 2026) the way the draft says, warns when a Google-Extended group lets Google train anyway, and shows the two lines that opt out.