Forge AI Bot Log – Block AI Bots, Crawler Detection & Verification
| 开发者 | forgeonline |
|---|---|
| 更新时间 | 2026年7月31日 11:14 |
| PHP版本: | 7.4 及以上 |
| WordPress版本: | 7.0 |
| 版权: | GPLv2 or later |
| 版权网址: | 版权信息 |
详情介绍:
User-agent: GPTBot gets logged as OpenAI. Forge AI Bot Log verifies each visit against the crawler's own published infrastructure using forward-confirmed reverse DNS — the technique Google, OpenAI and Anthropic recommend for confirming their own crawlers — and labels every hit Verified, Spoofed or Unverified. You find out which AI traffic genuinely came from OpenAI, and which was something else wearing its name.
Verification covers every crawler that publishes a way to check it, including OpenAI, Anthropic, Google, Perplexity, Amazon, Apple and Meta. Bots with no published verification method are labelled Unverified rather than guessed at.
Beyond verification, Forge AI Bot Log shows you exactly which AI crawlers and assistants are visiting your WordPress site and explains what each one actually does. The bundled directory covers 23 bots from 15 companies, including GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User, Google-Extended, GoogleOther, PerplexityBot, CCBot, Bytespider, Amazonbot, Applebot, Applebot-Extended, meta-externalagent, cohere-ai, Diffbot, YouBot, MistralAI-User and DuckAssistBot.
Key features
- Live dashboard — bot visits for the last 24 hours, 7 days, 30 days, plus a complete searchable log.
- Bot directory — click any bot for a plain-English explanation of who owns it, whether it trains AI models on your pages, powers AI search, or fetches pages live, and whether it obeys robots.txt.
- Most-crawled content — see which of your pages AI bots are reading most.
- Verified / Spoofed labelling — forward-confirmed reverse DNS on every tracked hit, so impersonation is flagged instead of trusted. Results are cached and throttled to keep it off the critical path.
- Access control — block any bot with one click. Blocking adds a
Disallow: /directive for that crawler to your robots.txt, and there is a hard-block.htaccesssnippet for the bots that ignore robots.txt. - Monthly email summary — sent to the site admin on the 1st of each month at 9am, covering the previous month.
- New-bot alerts — optional email the first time a brand-new AI bot appears.
- Reliable capture — an in-PHP logger, an optional must-use early logger, and a server access-log importer for sites behind full-page caching or a CDN.
- Privacy-friendly — only logs bots, not human visitors, with optional IP anonymisation and configurable data retention.
- CSV export — download the full or filtered activity log for your own analysis.
安装:
- Upload the
forge-ai-bot-logfolder to/wp-content/plugins/, or install via Plugins → Add New. - Activate the plugin.
- Visit Forge AI Bot Log in the admin menu.
- (Optional, for cached sites) Copy
mu-plugin/forge-ai-bot-log-mu.phpinto/wp-content/mu-plugins/for earlier capture, or use Settings → Import from server access log.
常见问题:
How do I stop ChatGPT from using my content?
Block GPTBot to stop OpenAI collecting your pages as training data for its models, and block OAI-SearchBot and ChatGPT-User if you also want to keep your content out of ChatGPT's search and live page-fetching. The plugin lists all three separately, because most people want to opt out of training without disappearing from AI search. Blocking writes the relevant robots.txt directives for you.
Can I tell if a bot claiming to be GPTBot is fake?
Yes — that is the main reason this plugin exists. A user-agent string is trivial to forge, so any scraper can send GPTBot and appear in your logs as OpenAI. On each tracked visit the plugin performs a reverse DNS lookup on the visiting IP, then re-resolves the hostname it gets back and confirms it maps to the same address and to the crawler's published domain. Genuine crawlers pass; spoofing does not. Hits are labelled Verified, Spoofed or Unverified so you can see the difference.
How do I block AI bots in robots txt?
Tick the bots you want to block on the plugin's Access page. The directives are merged into the robots.txt WordPress serves, so you never edit a file by hand. One caveat worth knowing: if a real robots.txt file exists in your site root, your web server serves that file directly and WordPress never runs — so the plugin's rules will not appear. Delete or fold in that physical file if you want the plugin to manage your AI crawler directives.
Does blocking a bot really stop it?
robots.txt is advisory. Well-behaved bots obey it; some (Bytespider and Perplexity-User, in the bundled list) are reported to ignore it. For those the plugin generates an .htaccess hard-block snippet that rejects them at the server level, on Apache.
Does blocking Google-Extended remove me from Google Search?
No. Google-Extended controls whether your content is used to train and ground Google's Gemini models. Normal Google Search crawling and indexing is done by Googlebot, which this plugin does not touch, so your search rankings are unaffected. Note also that blocking Google-Extended does not remove you from AI Overviews, which are generated from Google's regular search index.
Which AI crawlers can it detect?
23 bots from 15 companies — OpenAI, Anthropic, Google, Perplexity, Amazon, Apple, Meta, ByteDance, Cohere, Mistral AI, DuckDuckGo, You.com, Diffbot, Timpi and Common Crawl — split into AI training crawlers, AI search indexers and live retrieval agents. Each has its own entry explaining what it does with your pages. The crawlers behind the big assistants are all covered, so ChatGPT, Claude, Gemini and Perplexity traffic is identified by name rather than lumped together.
Does this slow down my site?
No. Logging is limited to matched AI bots and runs on an early hook, and reverse-DNS verification is throttled and cached. LLMs are not consulted at any point — all detection is local pattern matching against the bundled list plus a DNS lookup.
Will it catch bots if I use a caching plugin or CDN?
Full-page caches can serve bots without running PHP. Use the included must-use early logger and/or the access-log importer to capture those hits.
更新日志:
- Documentation: expanded and clarified the readme, with new FAQs on blocking, verification and robots.txt handling.
- Documentation: corrected the verification description — hits are labelled Verified, Spoofed or Unverified, not only the first two.
- Documentation: clarified that robots.txt directives are merged into the robots.txt WordPress serves, and that a physical robots.txt file in the site root overrides them.
- Internal version constant bumped to match the plugin header, so browsers re-fetch the plugin's CSS and JavaScript once. No other functional changes.
- Fixed: genuine bots visiting over IPv6 are no longer incorrectly marked "Spoofed" by DNS verification.
- Removed the remote bot-list auto-updater; the bundled bot list now updates via plugin releases.
- Added an External services disclosure for the optional promo feed.
- Added a Settings shortcut to the Plugins screen.
- New-bot alert emails are now sent on a background schedule instead of during the visit.
Older entries (1.0.0 – 1.1.1) are in
changelog.txtin the plugin folder.