Linux 软件免费装
Banner图

HitMind – AI Crawler Check

开发者 hitmind
更新时间 2026年11月18日 00:14
PHP版本: 7.4 及以上
WordPress版本: 7.2
版权: GPLv2 or later
版权网址: 版权信息

标签

robots.txt ai crawlers gptbot ai bots ai training

下载

1.0.0 1.1.0

详情介绍:

A crawler follows exactly one group in your robots.txt: the one that names it. The moment you give GPTBot a group of its own, it stops reading everything under User-agent: *, including the paths you blocked there (RFC 9309, section 2.2.1). It is an easy mistake to make. In HitMind's 2026 study of the top 10,000 websites, 29.8% of the robots.txt files that block paths and name an AI crawler left those paths open to that crawler. AI Crawler Check reads your live robots.txt the way crawlers do and shows: What it does not do: The rules engine is the one behind HitMind's free robots.txt generator and checker and its AI crawler study. It gives the same verdicts on all 5,539 robots.txt files from that study.

安装:

  1. Install and activate the plugin.
  2. Go to Tools → AI Crawler Check.

屏幕截图:

  • Every AI crawler: what it is for, whether it follows robots.txt, the group it reads, and the verdict for your home page.
  • The Site Health check.
  • AI training preference: whether your robots.txt states one with the `__usage_ai-training` line, and the two lines to add if it doesn't.

常见问题:

Does this plugin block AI crawlers?

No. It reads your robots.txt and reports what it finds. It never writes to it.

Why is a path I blocked still open to GPTBot?

Because GPTBot has its own group in your robots.txt. A crawler reads only the group that names it and ignores User-agent: * completely once it has one. Add the listed lines to its group, or remove the group if you meant GPTBot to follow your general rules.

Some crawlers say they ignore robots.txt. Does blocking them help?

The table marks crawlers whose operators say they don't follow robots.txt, such as Perplexity-User, which fetches pages when a person asks for them. Rules for them state your position but are not enforced. Stopping them takes server or firewall rules.

It says it could not read my robots.txt.

The plugin asks your own site for /robots.txt, from your own server. Some hosts block that, and some firewalls and CDNs answer with a challenge page. Paste your file into the box at the bottom of the page to check it instead.

Does it send my data anywhere?

No. The only request it makes is to your own site's /robots.txt. See "External services" for the two links on its page.

What is the __usage_ai-training line?

A shared robots.txt line for saying whether your pages may be used to train AI models, from an IETF draft published in October 2026. Bing applies it from 1 February 2027; Google and OpenAI say they will support it. It is not a crawl rule: it does not stop any crawler from fetching pages, and a User-agent: * group does not count as a preference. Until each company supports it, blocking training crawlers such as GPTBot by name is what stops the fetch.

WordPress runs in a subfolder on my site.

robots.txt always lives at the root of the domain, so the plugin checks that file and tells you it is not served by WordPress.

更新日志:

1.1.0 1.0.0