AI Crawlers Visibility, Analytics & Control
| 开发者 | cyberlord92 |
|---|---|
| 更新时间 | 2026年8月3日 17:13 |
| PHP版本: | 7.4 及以上 |
| WordPress版本: | 7.0.1 |
| 版权: | Expat |
| 版权网址: | 版权信息 |
详情介绍:
llms.txt, run an AI Visibility Score, and export reports — without replacing your SEO plugin, firewall, or server-level bot protection.
AI SEO, GEO, and AEO for WordPress
Traditional SEO helps search engines index a site. AI SEO, Generative Engine Optimization (GEO), and Answer Engine Optimization (AEO) focus on making content easier for AI crawlers and answer engines to discover and understand.
This plugin supports the technical foundations of AI visibility by helping you:
- Measure which AI crawlers and MCP clients access your content.
- Inventory the agents, MCP servers, and abilities touching your site.
- Check discovery, structured-data, and content-readiness signals.
- Publish AI-readable discovery files and Markdown versions of public content.
- Control crawler and MCP access without changing the content human visitors see.
- See which AI crawlers visit your site and which pages they request.
- See which MCP servers, abilities, and clients interact with your site.
- Compare traffic across date ranges and spot trends over time.
- Block AI crawlers (robots.txt / PHP 403) and MCP tools or servers (MCP Control).
- Keep logs for compliance, debugging, and content licensing decisions.
- See a 7-day crawler summary on the WordPress admin dashboard.
- Enable crawler tracking (and optionally MCP tracking) and choose log retention.
- Review Analytics, AI Inventory, and the Activity Log.
- Use Crawler Control and/or MCP Control for traffic you do not want.
- Run an AI Visibility Score scan and enable Discoverability features that fit your site.
- Dual crawler + MCP summary cards on one dashboard.
- Charts for crawler requests over time and traffic by category; MCP timeline and server charts.
- Top AI crawlers, top MCP tools, and most-crawled pages.
- Download CSV reports for crawler and MCP activity (preset or custom date ranges).
- Agents — crawlers and MCP clients with last seen, requests (7d), and status.
- MCP Servers — namespaces with trust tier (core / known / unknown).
- Abilities / Tools — tools seen in MCP traffic with a simple read/write risk heuristic.
- Findings — items that may need review (for example unknown servers).
- Detail pages per agent, server, or tool with entity-specific activity (paginated).
- Block / unblock from the inventory list or detail page (crawler PHP block or MCP Control rules).
- MCP detail activity includes an eye-icon request details modal (request/response JSON), same idea as MCP Tracker.
- Live refresh of the inventory catalog while the page is open.
- Dedicated recent-activity view with Crawlers and MCP tabs.
- Filters for crawler, status, content type, MCP server, and MCP outcome.
- Blocked attempts for crawlers and MCP.
- Click a row to open the matching AI Inventory detail page.
- Analyze any page or your entire site for AI readiness.
- 28 automated checks across discovery, structured data, content structure, and technical signals.
- 0–100 score with letter grade and prioritized recommendations.
- Site-wide batch scan, scan history, comparison diff, and PDF export.
- Auto-generate
/llms.txtand optional/llms-full.txtfor AI systems. - Choose public post types, set URL limits, and include or exclude individual content.
- Exclude content by URL pattern and automatically exclude noindex content.
- Respect noindex metadata from Yoast SEO, Rank Math, All in One SEO, SEOPress, and Genesis.
- Regenerate discovery files when content is published or on a schedule.
- Optionally add
LLMs:andLLMs-full:references to robots.txt. - Serve
.mdURL endpoints for public content and taxonomy archives. - Respond to
Accept: text/markdownrequests from compatible AI agents. - Output Organization, WebSite, and optional Article JSON-LD with AI discovery pointers.
- Include products in
llms.txtwith price, stock status, ratings, and attributes. - Variable product price ranges and sale price display.
- Product Markdown endpoints with YAML commerce frontmatter (SKU, price, availability).
- Optional Product JSON-LD schema (skipped when major SEO plugins already output it).
- Product data panel tab: exclude from AI and set a short AI description.
- HTTP
Linkheader pointing to/llms.txt(rel="ai-content-index"). - Product-focused Visibility Score checks (schema, SKU, price).
- Pre-configured detection for major AI, search, and scraper bots.
- Category labels (training, search, assistant, agentic, scraper, and more).
- Per-crawler info with user-agent pattern details.
- Enable or disable monitoring for individual AI crawlers.
- Includes crawlers associated with OpenAI, Anthropic, Google, Perplexity, Meta, Amazon, Apple, ByteDance, Common Crawl, and others.
- Add your own user-agent patterns for bots not in the built-in list.
- Assign company name and category for consistent reporting.
- Add Disallow rules for selected AI crawlers.
- Bulk templates to disallow or allow entire categories (training, scraper, and more).
- Compliance monitor shows top non-compliant crawlers that still visit.
- Rules are appended to your WordPress robots.txt output.
- Live preview before you publish changes.
- Hard-block selected AI crawlers before WordPress serves a response.
- Block specific IP addresses with the same 403 PHP response.
- Customize the plain-text message returned to blocked bots.
- Blocked requests are logged in analytics / Activity Log.
- Works alongside robots.txt for stronger enforcement when needed.
- Automatic MCP detection via
Mcp-Session-Idand tracked namespaces (wp-abilities,mcp,mcp-adapter). - Logging: route, HTTP method, headers (sensitive values redacted), request payload, response status, and response body.
- User attribution — which WordPress user (or guest) initiated the request.
- Filters by date range, ability/tool, user, and server namespace.
- Detail modal with formatted JSON request/response bodies.
- Block / unblock tool or server rules from the tracker row actions.
- Stats for total requests, unique abilities, errors, and last-24-hour activity.
- Enable or disable tracking from the MCP Tracker page header.
- Privacy focused — logs stay in your WordPress database.
- Shared retention with crawler logs; delete all MCP logs from Settings → Danger zone.
- Export MCP activity as CSV from Settings (Reports section).
- Optional MCP summary in scheduled email reports.
- Block or allow by tool, server, user, or IP.
- Rules apply at the WordPress REST layer for matching MCP traffic.
- Manage rules from Manage MCP → MCP Control, or add/remove common tool/server rules from MCP Tracker and AI Inventory.
- Configurable log retention (7 days to 1 year, or keep all data) for crawler and MCP logs.
- Optional delete-all crawler data and delete-all MCP logs from Settings.
- Delete plugin data on uninstall (optional setting).
- Multi-select filters on Activity Log and analytics tables.
- Download reports for preset or custom date ranges.
- Daily, weekly, or monthly HTML email reports with configurable recipients and alert types.
- Email summaries cover crawler activity, traffic changes, blocked attempts, new crawlers, robots.txt non-compliance, and MCP activity.
- Test email delivery and view the next scheduled report.
- Authenticated REST API for analytics summaries, recent activity, CSV exports, visibility scans, and blocked attempts.
- WordPress admin menu: AI Crawlers (by miniOrange).
- Grouped sidebar navigation and dark mode (theme toggle in Settings → General).
- Overview: Analytics, AI Inventory, Activity Log.
- Manage Crawlers: Crawlers, Control.
- Manage MCP: MCP Tracker, MCP Control.
- Optimize: Visibility Score, Discoverability.
- System: Settings, Support (contact popup).
- Built for site administrators — no code required.
- Bloggers and publishers who want to know when AI crawlers access their articles.
- Business and marketing sites tracking AI bot traffic to key landing pages.
- Developers and agencies managing AI crawler and MCP policy across client WordPress sites.
- Site owners exploring robots.txt, PHP blocking, and MCP Control.
- Teams using Abilities API / MCP who need inventory, audit logs, and allow/deny for tool calls.
- AI crawler and MCP activity logs are stored locally in your WordPress database.
- Analytics tracks detected crawler and MCP requests, not general human visitor analytics.
- MCP Tracker / Control do not send logs to external services.
- Log retention and deletion are controlled from the plugin settings.
- Scheduled reports are sent through the standard WordPress
wp_mail()function. - Core monitoring does not require a separate analytics account.
安装:
- Upload the
aI-visibility-suitefolder to/wp-content/plugins/, or install via the WordPress Plugins screen. - Activate AI Crawlers Visibility, Analytics & Control through the Plugins menu.
- Go to AI Crawlers → Crawlers and enable crawler tracking; set retention under Settings.
- Open AI Crawlers → Analytics after bots visit your site.
- Optional: open AI Crawlers → AI Inventory for a live catalog of agents, servers, and tools.
- Optional: open AI Crawlers → MCP Tracker, enable tracking, and configure MCP Control if you need allow/deny rules.
屏幕截图:
常见问题:
How do I monitor AI crawlers on WordPress?
Install and activate the plugin, enable crawler tracking under AI Crawlers → Crawlers, then open AI Crawlers → Analytics. Use Activity Log for recent crawler and MCP requests, and AI Inventory for a catalog of agents and tools.
Which AI crawlers does this plugin detect?
The plugin ships with 78+ built-in AI crawlers including GPTBot, ClaudeBot, Google-Extended, Amazonbot, Bytespider, CCBot, PerplexityBot, Applebot-Extended, Meta-ExternalAgent, and more. You can also add custom user-agent patterns for bots not in the list.
How do I block AI crawlers?
Blocking is opt-in. Use robots.txt rules under AI Crawlers → Control to disallow specific AI crawlers, or enable PHP blocking to return a 403 response. You can also block/unblock from AI Inventory agent rows. Blocked attempts appear in Analytics / Activity Log.
Does this plugin block AI crawlers automatically?
No. Tracking is opt-in when you enable it, and blocking is always opt-in. Configure robots.txt rules or PHP blocking per crawler from AI Crawlers → Control.
Will blocking AI crawlers hurt my SEO?
Blocking AI training or retrieval crawlers is separate from traditional search engine indexing. Disallowing GPTBot or similar bots in robots.txt does not block Googlebot or Bingbot by default. Review your robots.txt preview and allow crawlers you still want to reach your site.
Does the plugin modify my physical robots.txt file?
No. robots.txt rules are injected via the WordPress robots_txt filter. Use the live preview in Control to see the effective output.
What is the difference between robots.txt control and PHP blocking?
robots.txt publishes a preference that compliant crawlers should follow. PHP blocking returns an HTTP 403 before WordPress serves the content. You can use either or both.
What is AI Inventory?
AI Inventory is a live catalog of AI agents (crawlers and MCP clients), MCP servers, abilities/tools, and findings for this site. Open any row for a detail page with that item’s recent activity, pagination, and block/unblock controls. MCP rows can open a request details modal (request/response bodies).
What is the AI Visibility Score?
The AI Visibility Score analyzes a page or your site across 28 discovery, structured-data, content-structure, and technical checks. It returns a score from 0 to 100, a letter grade, and prioritized recommendations. It measures technical readiness; it does not guarantee AI citations or rankings.
What are llms.txt and llms-full.txt?
llms.txt is an AI-readable index of selected public content. The optional llms-full.txt can provide a fuller content representation. The plugin can generate both, update them when content changes or on a schedule, reference them in robots.txt, and exclude noindex or manually excluded content.
Does this replace Yoast SEO, Rank Math, or another SEO plugin?
No. This plugin focuses on AI crawler/MCP monitoring, access control, inventory, and AI discoverability. It works alongside an SEO plugin and can respect noindex metadata from Yoast SEO, Rank Math, All in One SEO, SEOPress, and Genesis.
Where is AI crawler and MCP data stored?
In your WordPress database. You can choose a retention period, delete collected data from Settings, and optionally remove plugin data during uninstall.
What is MCP Tracker?
MCP Tracker logs Model Context Protocol (MCP) and Abilities API related REST requests. Open AI Crawlers → MCP Tracker to filter the log, inspect JSON payloads, and block/unblock tools or servers.
What is MCP Control?
MCP Control is the policy layer for MCP traffic: allow or deny by tool, server, user, or IP. Open AI Crawlers → MCP Control to manage rules. Common tool/server blocks can also be triggered from MCP Tracker or AI Inventory.
What is MCP?
MCP (Model Context Protocol) is a standard that lets AI clients call tools/abilities over a consistent protocol. On WordPress this often appears as REST traffic to Abilities / MCP endpoints, commonly identified with an Mcp-Session-Id header.
Does MCP Tracker modify REST requests?
No. Logging is non-intrusive. It observes requests through WordPress REST hooks and does not change routing, authentication, or response content. MCP Control may deny matching requests when rules are configured.
Where are MCP logs stored?
In your WordPress database (a dedicated MCP requests table). Logs are not sent to external services. Retention follows the shared data-retention setting; you can delete all MCP logs from Settings → Danger zone.
Will MCP Tracker slow down my site?
Impact is minimal. Logging runs only for matching MCP / Abilities-related REST requests. You can disable tracking anytime from the MCP Tracker page.
Who can access the admin interfaces?
Only administrators (users with the manage_options capability). AJAX actions use capability checks and nonces.
Can I delete old logs?
Yes. Set shared Data retention under Settings, or use delete-all actions in Settings → Danger zone.
Will crawler monitoring slow down my site?
The plugin checks request user-agent strings and records matching AI crawlers. It does not run general human-visitor analytics. Choose an appropriate retention period for busy sites. Full-page caches or reverse proxies that serve requests before WordPress runs may prevent those requests from being logged.
Can I export logs?
Yes. Download crawler and MCP CSV reports from Settings (Reports section), with preset periods or a custom From/To range. Analytics also offers report download actions.
How do scheduled email reports work?
Open AI Crawlers → Settings, go to the Reports section, add recipients, choose daily/weekly/monthly delivery, and select alerts. WordPress sends mail through wp_mail(); configure SMTP if delivery is unreliable.
Does the plugin provide a REST API?
Yes. Authenticated endpoints under moai-av/v1 cover analytics summaries, recent activity, CSV exports, visibility scans, and blocked attempts. Requests require a logged-in administrator with manage_options.
Is this plugin compatible with caching and security plugins?
Yes. Detection runs on front-end / REST requests as documented. Test on staging if you use aggressive caching or a reverse proxy.
更新日志:
- Added AI Inventory — live catalog of agents (crawlers + MCP clients), MCP servers, abilities/tools, and findings.
- Inventory detail pages with entity-specific activity (paginated), block/unblock, and MCP request-details modal (eye icon).
- Inventory list actions: view activity (detail page), block/unblock icons.
- Added MCP Control — allow/deny MCP traffic by tool, server, user, or IP.
- MCP Tracker: block tool/server row actions and blocked status badges.
- Evolved Overview IA: Analytics, AI Inventory, and Activity Log.
- Analytics: dual crawler + MCP summaries, chart tabs, top MCP tools; Activity Log for recent + blocked crawler/MCP traffic.
- Activity Log rows open the matching AI Inventory detail page.
- Discoverability: in-page tabs for llms.txt, WooCommerce, Markdown for Agents, and JSON-LD.
- Settings: Reports embedded with General and Danger zone; Support opens as a sidebar popup.
- Sidebar System group: Settings + Support (theme toggle remains in General).
- Added MCP activity CSV export on the Reports page (same date range controls as crawler reports).
- Added MCP activity summary to scheduled and test email reports (optional alert toggle).
- Added MCP Tracker under Manage MCP to monitor MCP / Abilities API REST requests without changing request flow.
- Logs route, HTTP method, MCP method, ability name, user, session ID, headers (redacted), request/response bodies, and execution status.
- Added filters (date, ability, user, server), detail modal, stats cards, and SQL-paginated request log.
- Added header-level tracking toggle; shared retention cleanup; delete all MCP logs from Settings.
- Sensitive headers are redacted from stored logs.
- Added WooCommerce AI Search Optimization under Discoverability (optional when WooCommerce is active).
- Enriched product entries in llms.txt with price, stock, ratings, attributes, and variable/sale pricing.
- Added product Markdown endpoints with commerce YAML frontmatter.
- Added Product JSON-LD schema (auto-skipped when Yoast, Rank Math, AIOSEO, or SEOPress already output Product schema).
- Added Product data → AI Visibility tab to exclude products or set an AI short description.
- Added HTTP Link header to llms.txt (
rel="ai-content-index"). - Added product Visibility Score checks for Product schema, SKU, and price.
- Added site-wide Visibility Score scan from the Visibility page with batch progress and per-URL results.
- Added 5 built-in crawlers: Cotoyogi, ExaBot, VelenPublicWebCrawler, Omgilibot, and Thinkbot (78 total).
- Added a new AI agent category for custom crawlers, filters, and bulk robots templates; reclassified Google-Agent, Devin, FirecrawlAgent, and NovaAct.
- Expanded AI Visibility Score from 26 to 28 checks (semantic HTML5 landmarks and JavaScript-dependent content detection).
- Added a complete Reports page with report downloads, email delivery settings, schedule status, test email support, and REST API information.
- Added scheduled daily or weekly HTML email reports with configurable recipients and alert types.
- Added email alerts for crawler activity, traffic changes, blocked attempts, newly detected crawlers, and robots.txt non-compliance.
- Added professional grayscale email summaries with key metrics, the top crawler, and most-crawled pages.
- Added authenticated
moai-av/v1REST endpoints for analytics summaries, recent activity, CSV exports, visibility scans, and blocked attempts. - Added compact multi-select crawler, category, HTTP status, and content-type filters to Recent Activity.
- Added downloadable reports with preset periods and custom From/To date ranges.
- Improved report settings with immediate schedule-status updates and clearer admin notices.
- Added a graceful error when the server is missing the PHP DOM/XML extension required by Visibility Score, preventing a fatal error.
- Added Blocked IP Addresses under AI Crawlers → Control — block specific IPs with PHP 403.
- Added taxonomy markdown archives for Markdown for Agents (categories, tags, and other public taxonomies).
- Added 5 built-in crawlers: Timesbot, TavilyBot, Andibot, Brightbot, and NovaAct (73 total).
- Expanded AI Visibility Score from 19 to 26 checks (llms-full.txt, Markdown delivery, noai/noimageai, Twitter cards, HTML lang, heading hierarchy, image alt text).
- Added robots.txt compliance monitor — top non-compliant crawlers with one-click PHP block.
- Added category bulk templates to disallow or allow entire crawler categories at once.
- Added WordPress admin dashboard widget (miniOrange AI Crawlers) with last-7-days activity summary.
- Added customizable PHP 403 block message under AI Crawlers → Control → PHP blocking.
- Blocked AI crawlers receive your saved plain-text response instead of a generic “Access denied” (sanitized, 500-character limit).
- Added Discoverability page with llms.txt generator, Markdown for Agents, and JSON-LD schema output.
- llms.txt supports post-type filters, max URLs, llms-full.txt, auto-regenerate on publish, and scheduled cron.
- llms.txt robots.txt integration — optional
LLMs:andLLMs-full:references in robots.txt. - llms.txt content controls — exclude by URL pattern, manual include/exclude posts, and post search in admin.
- Markdown endpoints serve
/{slug}.mdplus content negotiation for AI agents. - JSON-LD includes site identity, AI discovery pointers, and optional Article schema.
- Visibility Score — improved robots.txt detection (physical and virtual files, no HTTP loopback required).
- Visibility Score — improved XML sitemap detection for WordPress core, Yoast SEO, and Rank Math.
- Fixed Visibility recommendation link for robots.txt to open AI Crawlers → Control.
- Redesigned admin UI with modern sidebar navigation, dark mode, and refreshed dashboard layouts.
- Added AI Visibility Score tab with single-URL and site-wide scanning.
- Added 19 readiness checks, recommendations, scan history, diff, and PDF export.
- Added visibility scan database tables.
- Added crawler analytics dashboard with date ranges, filters, period comparison, and charts.
- Added AI crawler monitoring, custom crawlers, robots.txt rules, and PHP blocking (403).
- Added grouped side navigation for Analytics and AI Crawlers admin pages.
- Added miniOrange menu and navbar branding.
- Expanded built-in AI crawler list and crawler info tooltips.
- Initial release with MVC architecture and free/premium codebase structure.
- Added admin dashboard foundation.