FocusWeb AI Crawler Monitor – Track, Verify & Block AI Bots

Yes -- per bot, and only when you opt in. On the Crawl Control page each bot can be set to Monitor (default, log only), Disallow (adds a robots.txt rule), Block (403 every matching request), or Block only fakes (403 only visits that claim the bot's User-Agent but fail verification). Updating the plugin never changes behavior: every bot stays on Monitor until you choose otherwise.

Where the vendor publishes official IP ranges (OpenAI for GPTBot and ChatGPT-User, Perplexity for PerplexityBot), each visit's IP is checked against those ranges -- instantly, using a locally cached copy refreshed daily. Where the vendor documents reverse-DNS rules instead (Google, Apple, Amazon, ByteDance), visits are logged as Pending and checked in an hourly background batch (reverse DNS plus forward confirmation), so DNS lookups never slow down your visitors. Bots whose vendors publish no method (e.g. Anthropic's ClaudeBot, Common Crawl's CCBot as of today) are honestly labeled Unverifiable rather than guessed at.

It blocks requests that pretend to be a known bot but fail verification -- for example a scraper sending "GPTBot" from a random datacenter IP -- while the genuine crawler from the vendor's verified infrastructure is untouched. It never adds a robots.txt rule, because the real bot remains welcome. For bots with no published verification method it never blocks anything (there is no reliable way to tell real from fake, so the plugin refuses to guess).

It can, depending on which bots you block. Blocking AI training crawlers (GPTBot, CCBot, Bytespider...) stops future model training with little effect on current traffic. Blocking AI search crawlers (PerplexityBot, and OpenAI's search crawling) may reduce your citations in AI answers and the human referral visits they generate. The Crawl Control page shows, per bot, how many referral visits the related platform sent you in the last 30 days, so you can decide with your own numbers.

No. The registry targets AI-specific crawlers only. Google-Extended is Google's AI-training opt-out token -- controlling it does not affect Googlebot, ordinary crawling, indexing, or rankings. As an extra safety rail, blocking never applies to logged-in users.

The plugin adds its rules through the standard WordPress robots_txt filter (the virtual robots.txt), alongside rules from other plugins. The Crawl Control page shows a live preview of the final merged robots.txt and warns you about the cases that need attention -- in particular a physical robots.txt file at your site root, which overrides the virtual one entirely.

No. Detection happens via a WordPress init hook reading the User-Agent header on each request, so it works identically on shared and managed hosting.

The plugin skips wp-admin, wp-login.php, admin-ajax.php, REST API requests, and static asset requests (images, CSS, JS, fonts, etc.) to keep the log focused on actual content visits. You can add further exclusions (e.g. /checkout/*) on the Settings page.

It stores the requesting IP address and User-Agent string for each logged visit. IP anonymization (masking the last octet/segment) is enabled by default. Verification checks the real IP at request time, but with anonymization enabled only the anonymized address is stored in the log; addresses queued for background reverse-DNS checks are held temporarily and deleted as soon as the check completes. You can also configure automatic log deletion after a set number of days.

No. Range verification is pure in-memory math against a cached list. Reverse-DNS lookups and vendor range downloads only ever run in background cron jobs, never while a visitor (or bot) is waiting for a response.

Yes. Three filters are provided: aicm_known_bots (add bots with a label, User-Agent pattern, and category), aicm_verification_config (add or override per-bot verification methods -- IP-range URL or rDNS suffixes), and aicm_block_decision (final say over any block, receiving the bot key, IP, and verification verdict).

No. It only generates suggested content in a textarea, with a button to download it as llms.txt. You place the file at your site root manually, avoiding filesystem permission issues on managed hosting.

Not yet. llms.txt is a community-proposed convention (introduced in 2024) suggesting a simple Markdown file at your site root to help AI systems find your key content. As of today, no major AI crawler or assistant (OpenAI, Anthropic, Google, Perplexity, etc.) has publicly confirmed that it reads or prioritizes llms.txt in production. Publishing one is a reasonable, low-effort bet on where things may be heading -- not a guaranteed way to influence how AI systems treat your site. This plugin only generates suggested content for you to review and place manually; it never assumes llms.txt is authoritative or required.