AI Crawler Control by Citlyze – robots.txt, llms.txt & AEO Audit

Open Citlyze > AI crawler access and choose "Block AI training, stay in AI search". It blocks GPTBot and the other training crawlers and leaves OAI-SearchBot and ChatGPT-User alone, so ChatGPT search can still read and cite your pages. The preview shows the exact lines before you save.

Start with Citlyze > AI crawler access. It shows whether your robots.txt blocks OAI-SearchBot, PerplexityBot or the assistants that fetch pages, and which plugin, theme or setting wrote the rule. It also flags the "Discourage search engines" setting and hosts or firewalls that can block crawlers before robots.txt is read. Being readable is the first requirement; whether AI answers then cite you depends on your content and your competitors, which is what Citlyze measures.

Only when you choose AI crawler rules and save them. The plugin then adds a marked block of rules to the end of the robots.txt WordPress serves, after anything other plugins write. Until then it changes nothing, and "Undo last change" puts back your previous choice.

Yes. Under "Parts of your site", add a path such as /members/ and choose whether to keep out AI training crawlers or all AI crawlers. The rest of the site keeps your choice for the whole site. Crawlers that get rules for a section keep following your site's other robots.txt rules too, because the plugin repeats them in their group.

From the AI provider set up in WordPress 7.0 and later, such as OpenAI, Anthropic or Google, never from Citlyze. A suggestion is only requested when you click, you see it before anything changes, and Insert adds it as plain text that Undo removes. Without a provider set up in WordPress, the panel shows no suggestions.

Yes. Once a day it checks the robots.txt crawlers receive. If another plugin, a theme, a file on the server or the "Discourage search engines" setting starts blocking AI search crawlers, AI assistants or search engines, you get a notice in wp-admin, a Site Health result, and, if you turn it on, an email. Rules you chose on the AI crawler access screen never raise an alert.

Yes. Their robots.txt rules stay as they are, and the check names them when they block a crawler. The plugin's own rules are added after theirs. If one of them already generates llms.txt, this plugin checks that file instead of serving its own.

A robots.txt file on the server is sent directly by your web server, so WordPress and its plugins cannot add to it. The check reads that file, and the plugin shows the rules you chose so you can copy them into it.

No. The AI crawler tools work without an account. An account is only needed to track AI crawler visits and visits from AI answers in Citlyze.

Yes. The editor panel uses the same checks and scoring as the Citlyze Chrome extension and the free AEO grader on citlyze.com. A published page gets the same score on all three.

No major AI engine has documented reading llms.txt, and public log studies show little evidence that they request it. It costs nothing to have, so the plugin helps you keep a valid one, but robots.txt matters far more: it decides whether AI crawlers can read your site at all. Our read of the evidence: https://citlyze.com/blog/does-llms-txt-work

No. The checks run only on the plugin's admin screens. On the public site, the plugin only adds its rules when robots.txt is requested, and, if you connect Citlyze, notes AI crawler requests without loading anything in visitors' browsers.

Only when the request reaches WordPress. A full-page cache or CDN can respond without running PHP, so those crawler visits are not visible to this plugin. The connection screen tells you when it detects a page cache. Use the Citlyze Cloudflare Worker, or upload your server logs in Citlyze, for sites where most traffic is served from a cache or the edge.

Anything that never reaches WordPress: redirects done by your web server or CDN (for example rules in .htaccess or nginx, or a host-level redirect to https or www), requests a firewall blocks before WordPress loads, and pages served from a full-page cache. The Citlyze Cloudflare Worker or a log upload covers those.

Citlyze confirms a crawler by its IP address, so the plugin needs the real visitor address. Behind Cloudflare this works automatically: the plugin trusts Cloudflare's visitor header only for requests that come from Cloudflare's own network. Behind another proxy, choose the header it sets and list its addresses under Citlyze > Connect Citlyze > Visitor IP behind a proxy.

The plugin recognises crawlers that issue live page requests, including GPTBot, ChatGPT-User, OAI-SearchBot, ClaudeBot, PerplexityBot, Googlebot, Google Gemini Notebook, Google-Agent, Atlassian Rovo, Amazonbot, Applebot, Bingbot, Meta crawlers, and others. The list is updated daily from Citlyze and with each release. Citlyze independently confirms that a request genuinely came from the crawler it claims to be before counting it as verified. A user-agent string on its own is only a claim, so unconfirmed requests are reported separately as diagnostics and never inflate verified crawler totals.

The free tools send nothing anywhere. With the Citlyze connection configured, events go to the Citlyze endpoint entered in Citlyze > Connect Citlyze. See the External services section for the exact fields, timing, retention summary, and policy links.

The secret you paste is never stored. It is reduced to a derived signing key immediately, and that key is encrypted using your site's own WordPress salts before it is written to the database. The field is write-only: once saved, the value cannot be read back through the admin screen. You can avoid database storage entirely by defining CITLYZE_CRAWLER_SIGNING_KEY in wp-config.php instead. If a database backup leaks, rotate the site key in Citlyze under AI Crawlers > Install tracking. Rotation invalidates the old key immediately.