VigIA - AI Crawler Activity Analytics & Control
| 开发者 |
fernandot
ayudawp |
|---|---|
| 更新时间 | 2026年2月17日 03:54 |
| PHP版本: | 7.4 及以上 |
| WordPress版本: | 6.9 |
| 版权: | GPLv2 or later |
| 版权网址: | 版权信息 |
详情介绍:
VigIA (Spanish for "lookout" or "watchman", incorporating "IA" - Spanish for "AI") is a comprehensive WordPress plugin that tracks, analyzes, and helps you control AI crawler activity on your website.
What does VigIA do?
- Tracks AI crawlers visiting your site (GPTBot, ClaudeBot, PerplexityBot, and 50+ others)
- Provides detailed analytics with charts, statistics, and exportable reports
- Blocks unwanted crawlers via PHP (403 response)
- Manages robots.txt rules for AI crawlers with compliance monitoring
- Sends email alerts about crawler activity (daily, weekly, or monthly)
- Generates llms.txt files to help AI systems understand your site
- Serves markdown endpoints for individual posts and pages (Markdown for Agents standard)
- Exposes abilities for AI agents and automation tools (WordPress 6.9+)
- Total visits, unique crawlers, and pages crawled statistics
- Timeline chart with daily breakdown
- Category distribution (AI Training, AI Search, AI Assistant, Data Scraper)
- Top crawlers and most crawled pages tables
- Recent activity log with filters
- Period comparison functionality
- CSV export Crawler Blocking
- Block crawlers via PHP with 403 Forbidden response
- Quick block dropdown in analytics dashboard
- Manage blocks from Extras page
- Works on any server (Apache, Nginx, LiteSpeed, etc.) Robots.txt Management
- Add Disallow rules for AI crawlers
- Visual preview of your robots.txt
- Compliance monitoring: see which crawlers ignore your rules
- One-click blocking for non-compliant crawlers
- Works with both physical and virtual robots.txt Email Alerts
- Daily, weekly, or monthly reports
- Three detail levels: Minimal, Normal, Complete
- Non-compliant crawler warnings
- Activity comparison with previous period Markdown for Agents
- Serve individual posts/pages as optimized markdown for AI agents
- Dedicated .md URL endpoints (e.g., /your-post.md)
- Accept: text/markdown content negotiation
- Discoverability via link headers and HTML alternate tags
- YAML frontmatter with title, date, author, categories, tags, and more
- Respects blocking rules (blocked crawlers get 403)
- Respects LLMs.txt exclusion filters
- Analytics integration: tracks markdown requests per crawler
- X-Markdown-Tokens response header
- Follows the Cloudflare Markdown for Agents standard LLMs.txt Generator
- Select content by post type with one click
- Filter by taxonomies (categories, tags, custom)
- Manual include/exclude with AJAX search
- Exclude by URL patterns (wildcards supported)
- SEO plugin integration (auto-exclude noindex content)
- Auto-regeneration (daily, weekly, monthly)
- Robots.txt integration (add llms.txt and llms-full.txt references)
- Generate llms.txt and llms-full.txt files
- Full content or excerpt mode
- Compatible with Yoast SEO, Rank Math, All in One SEO, SEOPress, and The SEO Framework
- OpenAI: GPTBot, OAI-SearchBot, ChatGPT-User
- Anthropic: ClaudeBot, Claude-Web, Claude-SearchBot
- Google: Google-Extended, GoogleOther, Gemini-Deep-Research
- Perplexity: PerplexityBot, Perplexity-User
- Meta: Meta-ExternalAgent, FacebookBot
- Microsoft: BingBot
- ByteDance: Bytespider
- Amazon: Amazonbot
- Apple: Applebot-Extended
- And many more...
安装:
- Upload the plugin files to
/wp-content/plugins/vigia/or install through the WordPress plugins screen - Activate the plugin through the 'Plugins' screen in WordPress
- Navigate to VigIA in the admin menu to view your AI crawler analytics
- Configure blocking rules and alerts in VigIA > Extras
屏幕截图:
常见问题:
Does this plugin slow down my site?
No. VigIA adds minimal overhead by checking the User-Agent string on each request. The check is very fast and only writes to the database when an AI crawler is detected.
What's the difference between robots.txt blocking and PHP blocking?
Robots.txt is advisory - crawlers should respect it but may choose to ignore it. PHP blocking returns a 403 Forbidden response, effectively preventing access regardless of whether the crawler respects robots.txt.
Will blocking crawlers affect my SEO?
Blocking AI training crawlers (like GPTBot or ClaudeBot) will not affect traditional search engine rankings. However, blocking AI search crawlers might affect how your content appears in AI-powered search results.
What is Markdown for Agents?
Markdown for Agents is a standard for serving web content as clean markdown to AI agents. Instead of processing full HTML, agents receive lightweight markdown with structured metadata. VigIA supports both dedicated .md URLs and Accept: text/markdown content negotiation. Enable it in VigIA > Extras > Markdown for Agents.
Can I add custom crawlers to monitor?
Yes! In the main analytics page, scroll down to "Custom crawlers" and add your own User-Agent patterns to track.
Where is the data stored?
All data is stored in your WordPress database in a custom table (wp_vigia_visits). No data leaves your server.
What is llms.txt?
The llms.txt file is a standard for helping AI systems understand your website's content and structure. It provides a machine-readable overview of your site that AI can use to better represent your content. Learn more at llmstxt.org.
Which SEO plugins are supported for noindex detection?
VigIA supports automatic noindex detection from: Yoast SEO, Rank Math, All in One SEO, SEOPress, and The SEO Framework.
What is the Abilities API?
The Abilities API is a new feature in WordPress 6.9 that allows plugins to expose their functionality in a standardized way. This enables AI agents, automation tools, and external systems to discover and use plugin features programmatically. VigIA implements 9 abilities for analytics, blocking, and robots.txt management.
更新日志:
1.5.0
- NEW: Markdown for Agents - serve individual posts as optimized markdown for AI agents
- NEW: Dedicated .md URL endpoints for each post and page
- NEW: Accept: text/markdown content negotiation support
- NEW: Discoverability via link rel="alternate" HTML tags and Link HTTP headers
- NEW: YAML frontmatter with full post metadata (title, date, author, categories, tags, image, language)
- NEW: X-Markdown-Tokens response header
- NEW: Respects LLMs.txt exclusion filters (noindex, URL patterns, manual excludes)
- NEW: Blocked crawlers receive 403 on markdown endpoints too
- NEW: Analytics integration tracks markdown requests from known crawlers
- NEW: Configurable post type selection for markdown endpoints
- NEW: New "Markdown for Agents" tab in Extras page
- Follows the Cloudflare Markdown for Agents standard
- Fixed: Abilities API category registration error on WordPress 6.9+
- Improved: Abilities API initialization method for better compatibility
- Improved: Dynamic promotional banners with random plugin and service rotation
- NEW: WordPress Abilities API integration (requires WordPress 6.9+)
- NEW: 9 abilities for AI agents and automation tools
- NEW: Analytics abilities: get-crawler-stats, get-top-crawlers, get-top-pages
- NEW: Blocking abilities: get-blocked-items, block-crawler, unblock-crawler
- NEW: Robots.txt abilities: get-robots-rules, add-robots-disallow, remove-robots-rule
- NEW: Custom ability category "AI Crawler Analytics" for better discoverability
- Improved: Graceful degradation for WordPress versions below 6.9
- NEW: Period indicator in dynamic section titles (timeline, categories, crawlers, pages)
- NEW: "Load more" pagination in Top crawlers and Top pages tables (10 items per load, up to 100)
- NEW: Showing counter display (e.g. "Showing 10 of 47")
- Improved: Plugin recommendation boxes now available on Extras page
- Fixed: Shortcodes from page builders and common plugins now display as clean text in llms.txt files
- Fixed: robots.txt Disallow rules now work correctly with physical robots.txt files (Yoast SEO and other plugins that create physical files)
- Improved: AI crawler rules are now synced to physical robots.txt automatically when adding or removing rules
- Fixed: UTF-8 encoding issues in llms.txt and llms-full.txt files (special characters like accents now display correctly in all browsers)
- Fixed: "Learn more" link is now translatable
- Fixed: LLMs.txt Generator now correctly adds robots.txt references when saving and generating
- Improved: Simplified LLMs.txt Generator to single "Save and generate" button for clearer workflow
- Updated: "Learn more" link now points to AyudaWP documentation
- Fixed: Responsive layout issues with charts and content boxes overflowing on smaller screens
- Improved: Better breakpoint handling for stats grid and charts adaptation
- Fixed: Renamed duplicate "VigIA" submenu to "Analytics" for better navigation clarity
- Fixed: Activation notice now properly dismisses when closed on the plugins page
- Improved: Added PHPCS ignore comments for intentional direct database queries (cache bypass)
- Improved: Added translator comments for all placeholder strings in LLMs generator
- Fix: LLMs.txt generator now correctly handles taxonomy filters for each post type
- Fix: Taxonomy filters (categories, tags) no longer incorrectly exclude pages or other post types
- Fix: Resolved object cache conflicts on hosts with Memcached/Redis (SiteGround, etc.)
- Fix: Settings now persist correctly after saving and regenerating
- Fix: Robots.txt references only added when llms files actually exist
- Fix: Robots.txt integration reads fresh data bypassing object cache
- Improved: Added "Save settings" button to save options without generating files
- Improved: Direct database queries for settings to avoid cache inconsistencies
- Improved: Better boolean normalization for checkbox values
- NEW: Redesigned LLMs.txt generator with improved content selection
- NEW: Select content by post type (posts, pages, custom post types)
- NEW: Filter content by taxonomies (categories, tags, custom taxonomies)
- NEW: Manual include with AJAX-powered search
- NEW: Manual exclude with AJAX-powered search
- NEW: Exclude content by URL patterns with wildcard support
- NEW: SEO plugin integration - auto-exclude noindex content
- NEW: Supported SEO plugins: Yoast, Rank Math, AIOSEO, SEOPress, The SEO Framework
- NEW: Auto-regeneration options (daily, weekly, monthly)
- NEW: Add llms.txt and llms-full.txt references to robots.txt
- Improved: Scalable architecture for sites with thousands of posts
- Improved: Better UI/UX for content selection
- Fix: Version dump to improve texts.
- NEW: PHP-based crawler blocking (403 Forbidden response)
- NEW: Robots.txt management for AI crawlers
- NEW: Compliance monitoring (detect crawlers ignoring robots.txt)
- NEW: Email alerts system (daily/weekly/monthly reports)
- NEW: LLMs.txt and llms-full.txt file generator
- NEW: Extras page with all new management tools
- NEW: Quick block dropdown in crawlers table
- Improved: Block actions directly from analytics dashboard
- Initial release
- AI crawler detection and tracking
- Analytics dashboard with charts
- CSV data export
- Custom crawler support
- Dashboard widget