| 开发者 | maksymkotenko |
|---|---|
| 更新时间 | 2026年11月14日 19:50 |
| PHP版本: | 7.4 及以上 |
| WordPress版本: | 7.1 |
| 版权: | GPLv2 or later |
| 版权网址: | 版权信息 |
/.well-known/tdmrep.json, the tdm-reservation header and meta tag, with an optional licensing policy URL.Content-Signal: search=yes, ai-input=yes, ai-train=no line to robots.txt, with each value your choice.X-Robots-Tag header.latchbot_bots filter - add, change or remove crawlers; the list is built once per request, so add the filter from a plugin or your theme's functions.php. Each entry has token, name, operator, purpose (training, search or user) and ua (the User-Agent substring; empty for robots.txt-only tokens).latchbot_refuse filter - decide whether a request from a known crawler gets a 403. Arguments: the decision, the crawler id and the kind of page.latchbot_visit action - runs after a crawler visit is counted, with the crawler id, the kind of page and whether it was refused.No. Google-Extended only controls whether Google may use your pages for its AI models; Google Search keeps crawling with Googlebot, which Latchbot never touches. The same goes for Applebot-Extended and Applebot.
Training crawlers collect pages to train AI models. AI search crawlers build the index that AI answers cite and link to. User fetches happen when someone asks an assistant about your page. Many sites block training and allow the other two, which is the first preset.
The robots.txt rules, the TDMRep file and the meta tags are part of the page, so cached copies keep them. The X-Robots-Tag and tdm-reservation headers, the visit counts and the 403 answers need WordPress to run, so they cover only requests your cache passes through; some caches store headers too.
Crawlers read robots.txt and /.well-known/tdmrep.json only at the root of the domain. Latchbot shows a warning; copy the rules from the preview into the files at the root.