All-IN Crawler & AI Agent Manager
| 开发者 | elis85 |
|---|---|
| 更新时间 | 2026年11月6日 14:34 |
| PHP版本: | 7.4 及以上 |
| WordPress版本: | 7.1 |
| 版权: | GPLv2 or later |
| 版权网址: | 版权信息 |
详情介绍:
- About 120 known bots in 12 categories: search engines, AI search & assistants, AI training crawlers, SEO tools, social previews, monitoring, feed readers, archives, scripts & HTTP libraries, vulnerability scanners, unidentified bots and empty User-Agents.
- Allow, rate-limit or block per category, with per-bot overrides and custom User-Agent rules (text or regex).
- One-click presets: Balanced, Maximum AI visibility, Protect content from AI training, Lockdown.
- IP / CIDR allow and block lists (IPv4 and IPv6), and excluded paths for webhooks and payment callbacks.
- Protect content from AI training, stay in AI answers: training crawlers (GPTBot, ClaudeBot, CCBot…) are handled separately from AI search and assistant agents (OAI-SearchBot, ChatGPT-User, Claude-User, Perplexity-User…).
- Real search engines only: forward-confirmed reverse DNS plus optional official crawler IP lists (Google, Bing, Apple, OpenAI, Perplexity; off until you turn them on) catch fake Googlebots and GPTBots.
- Overload protection: rate limits per IP or per bot with 429 + Retry-After, load shedding with 503 + Retry-After when server load per CPU core crosses your threshold, and an optional per-IP limit for "human" visitors (logged-in users are exempt). SEO-safe status codes only, never cached.
- robots.txt and llms.txt: manage robots.txt rules (including Google-Extended, Applebot-Extended and an optional Crawl-delay) and generate an llms.txt that skips noindex content.
- Block before PHP: export rules to .htaccess (Apache/LiteSpeed) or nginx. Inside WordPress, requests are decided while plugins load, before the theme runs.
- Dashboard and reports: 7/14-day statistics, top denied IPs with one-click blocking, a denied-request log, a User-Agent tester and weekly or monthly email reports.
- Optional bot-list feed that adds new AI crawlers between releases (add-only and safety-checked). Off by default, and it only downloads from a URL you enter.
- Fast storage: APCu, then Redis/Memcached, then the database.
- Fails open: if anything goes wrong inside the plugin, visitors are let through and the error is logged.
- Full Hebrew translation, RTL ready.
docs/USER-MANUAL.md (site owners) and docs/DEVELOPER.md (hooks & architecture).
More from A-in
This plugin is complete on its own: no licence key, no locked features. Premium plugins, add-ons and support from the same developer are available at a-in.co.il.
安装:
- Upload the plugin ZIP via Plugins → Add New → Upload Plugin, then activate it.
- Open All-IN Crawler → Dashboard and check that "Your IP" shows your real public IP. If it doesn't, configure Advanced → IP source.
- Optionally switch Protection → Mode to Monitor for a few days to review decisions safely.
- Pick a preset on Bot Rules.
- Optionally export server rules (.htaccess / nginx) on the Server Rules tab so blocked bots are rejected before PHP runs.
屏幕截图:
常见问题:
Will this hurt my SEO?
No, with the default configuration. Search engines are allowed and DNS-verified, and throttling uses 429/503 with Retry-After, which search engines treat as "come back later".
Can I block AI training but stay in ChatGPT / Claude / Perplexity answers?
Yes. Apply the "Protect content from AI training" preset. AI search and assistant agents stay allowed.
Can I block Google's AI Overviews?
Not without blocking Google Search. AI Overviews use regular Googlebot. Google-Extended only controls Gemini training/grounding use.
Does it replace Cloudflare or a server firewall?
No. It complements them. Large DDoS attacks must be stopped before they reach PHP.
Does All-IN Crawler & AI Agent Manager contact any outside service by default?
No. Out of the box it makes no outgoing requests: search engine bots are verified with DNS lookups through your own server's resolver. Two features can download data, and both are off until you turn them on in the Bot Lists tab: the official crawler IP lists (from Google, Bing, Apple, OpenAI and Perplexity) and a bot-list feed URL of your own. See "External services" below for exactly what is requested.
Why would I turn on the official crawler IP lists?
OpenAI and Perplexity bots have no reverse DNS, so without their published IP lists All-IN Crawler & AI Agent Manager can't tell a real GPTBot or PerplexityBot from a scraper using the same name; they stay "unverified". With the lists on, a request claiming to be one of those bots from an unlisted IP is treated as an impostor. A list is only used while it is fresh (under 7 days old), so a failed download never blocks a genuine bot.
Does a page cache bypass it?
Pages served from a full-page cache don't run WordPress, so All-IN Crawler & AI Agent Manager doesn't see them. Those requests are cheap anyway. Denials are sent with Cache-Control: no-store and are never cached.
更新日志:
- Changed: new name, All-IN Crawler & AI Agent Manager (slug and text domain all-in-crawler-ai-agent-manager). Settings, statistics and logs are kept: the internal botctl_ option and table names are unchanged, and settings exports from earlier versions can still be imported.
- Removed: the optional early loader. The plugin no longer writes any file to wp-content/mu-plugins; a loader created by an older build is removed on upgrade, deactivation and uninstall (only if All-IN Crawler & AI Agent Manager generated it).
- Security: the request URI, User-Agent, method and cookie names are sanitized as soon as they are read; imported settings and admin form input are whitelisted, type-cast and sanitized field by field.
- Changed: translations are loaded by WordPress from translate.wordpress.org; no translation files are bundled.
- Changed: extra bot definitions (
data/bots.json) are bundled and always used; no network request is needed for them. - Changed: there is no default remote feed any more. The feed downloads only from a URL you enter.
- Changed: official crawler IP list downloads are opt-in and off by default. Nothing is downloaded until you enable them on the Bot Lists tab, and a setting saved by an older version never turns them on. The IP-list requests no longer include your site URL.
- Changed: requires WordPress 6.1 (object-cache group flush).
- Docs: External services section, GPL LICENSE file, Tested up to 7.1. Description condensed, Donate link, 5 screenshots with captions.
- New: official crawler IP lists (Google, Bing, Apple, OpenAI, Perplexity) with fail-safe freshness checks.
- New: remote bot-list feed (add-only, browser-safe validation, optional access key).
- New: .htaccess / nginx server rules with auto-sync.
- New: weekly/monthly white-label email reports.
- New: llms.txt generator.
- New: Hebrew translation.
- Initial release: bot detection (~120 bots, 12 categories), per-category/per-bot/custom rules, presets, rate limiting (APCu / object cache / DB), DNS verification of search engines, load shedding, human rate limiting, IP allow/block lists, robots.txt management, dashboard, logs, UA tester, import/export, early loader, debug logging.