ADG Markdown for AI Crawlers
| 开发者 |
andyodua
freemius |
|---|---|
| 更新时间 | 2026年9月19日 03:52 |
| PHP版本: | 7.4 及以上 |
| WordPress版本: | 7.1 |
| 版权: | GPLv2 or later |
| 版权网址: | 版权信息 |
详情介绍:
When a known AI crawler (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and others) requests one of your posts or pages, this plugin returns clean Markdown of the main content on the same URL instead of the full theme HTML — no header, footer, sidebar or scripts. Regular visitors keep seeing your normal site.
It also generates llms.txt and llms-full.txt — a sitemap-style index of your content for AI tools — served at
/llms.txt and /llms-full.txt. These are kept in sync as real files at your site root as a fallback for hosts that block the virtual route before WordPress runs, so the files stay available even there.
What it does
- Detects AI crawlers by User-Agent (the list is editable in the settings).
- Serves clean Markdown on the same URL with
Content-Type: text/markdown,Vary: User-AgentandX-Robots-Tag: noindex. - Works with any public post type — including WooCommerce products — plus a Markdown index for category, tag and custom-taxonomy archives.
- Configurable YAML front matter (title, url, date, author, tags, canonical) with per-field toggles.
- Include your own custom fields / ACF values in the front matter.
- Adds machine-readable Schema.org JSON-LD (title, dates, author, cover image, publisher).
- Generates
/llms.txtand/llms-full.txt, with an on-disk fallback file for hosts that block the virtual route. - Caches the converted Markdown in post meta for fast responses; the cache is cleared automatically when a post is edited.
- Plays nice with page caches: sets
DONOTCACHEPAGE(WP Rocket, W3 Total Cache, WP Super Cache, LiteSpeed) so bots are never served a stale HTML cache. - Exclude content with a per-post "Hide from AI bots" checkbox or by category/tag slug.
- "View Markdown" link in the editor and post list — preview exactly what crawlers receive.
- No required .htaccess edits and no Composer — a built-in, zero-dependency HTML→Markdown converter is included. An optional, one-click .htaccess fix is available if your host blocks
/llms.txt(see Settings → AI Bots & Markdown). - Does not interfere with SEO plugins, caches, or WooCommerce shop/cart/checkout pages.
- Private, password-protected and unpublished content is never served to bots — access rights are not bypassed.
安装:
- Upload the
adg-markdown-for-ai-crawlersfolder to/wp-content/plugins/, or install the plugin through the WordPress "Plugins" screen. - Activate the plugin.
- Go to Settings → AI Bots & Markdown to configure it.
- On Nginx, if
/llms.txtreturns a 404, make sure your config falls through toindex.php(the standardtry_files $uri $uri/ /index.php?$args;block covers this). On Apache it works out of the box.
屏幕截图:
常见问题:
Will private or draft pages be exposed to bots?
No. Markdown is served only for published, non-password-protected, non-private posts of the allowed types. Access rights are never bypassed.
Does it send my content to any external server?
No. Your content is never sent anywhere — detection is by User-Agent, and all conversion happens locally. The plugin does contact the Freemius API (freemius.com) for license validation, but only if you activate a Pro license key; this never involves your site's content.
Does it work with page builders like Elementor?
Yes. Content is taken through the_content, so shortcodes and builder output are rendered before conversion. Testing on your specific setup is recommended.
A bot still receives HTML instead of Markdown. Why?
Most often the request is blocked at the server/CDN level (HTTP 403) or served from a page cache before the plugin runs. Make sure your cache honors Vary: User-Agent or excludes bot user-agents.
My llms.txt file returns a 404 even though the plugin is active and enabled.
On Newfold-family hosting (Bluehost, HostGator, iPage and similar cPanel providers), a server-level block in .htaccess (# BEGIN NFD EPC) may prevent /llms.txt requests from reaching WordPress. This is not a plugin bug — the server intercepts .txt file requests before PHP. Since 2.2.0 the plugin also keeps a real llms.txt file at your site root as a fallback, which resolves this on many hosts automatically; if it's still blocked, Settings → AI Bots & Markdown offers a "Try automatic fix" button (Apache) or the exact manual line to add: open .htaccess, find the # BEGIN NFD EPC block, locate the line with RewriteCond %{REQUEST_URI} !(robots\.txt|...), and add llms\.txt|llms-full\.txt to the exception list: RewriteCond %{REQUEST_URI} !(robots\.txt|llms\.txt|llms-full\.txt|...). Save and re-test using Settings → AI Bots & Markdown → "Re-check now".
更新日志:
2.2.0
- New: /llms.txt and /llms-full.txt are now also written as real files at the site root, as a fallback for hosts that block the virtual route before WordPress runs — the site keeps serving them even where that block can't be fixed from within WordPress.
- New: "Try automatic fix" button applies a self-contained .htaccess exception ahead of a blocking host rule (Apache only), with an automatic backup of the previous file.
- Improved: self-test now recognizes WP Engine, Kinsta, Pantheon and SiteGround in addition to Newfold, with tailored guidance per host.
- Improved: regenerating llms.txt now also purges WP Rocket, LiteSpeed Cache and W3 Total Cache if active, so a page cache can't keep serving a stale copy.
- Fixed: the self-test no longer reports a false "blocked" when /llms.txt is being served correctly by the new on-disk fallback file.
- Improved: the /llms.txt self-test check now compares against /robots.txt to reliably distinguish a server-level block (Newfold/Bluehost-family hosting) from a plain rewrite-rule issue, with a "Copy fix" button for the exact .htaccess line.
- New: automatic detection of server-level blocking of /llms.txt (common on Bluehost/HostGator/iPage and other Newfold-family hosting via their ".htaccess NFD EPC" block) — an admin notice appears on Settings → AI Bots & Markdown with a "Re-check now" button and step-by-step manual fix instructions, including clearing the host's page cache.
- Single codebase with a Pro license (Freemius): one plugin, Pro features unlocked by a license key.
- Pro: training-vs-citation access policy, robots.txt management, bot IP verification (anti-spoofing), bot-visit analytics + CSV, AI referral tracking with a by-source chart, token-savings estimate, one-click self-test, AI Readiness Score, and a weekly email digest.
- New: Pro feature previews on the settings screen — the training-vs-citation access policy, robots.txt management and the AI analytics/citation dashboard are shown as locked teasers with a link to the Pro version.
- New: configurable YAML front matter (title, url, date, author, tags, canonical) with per-field toggles.
- New: include custom fields / ACF values in the Markdown front matter.
- New: Markdown index for category, tag and custom-taxonomy archives.
- New: WooCommerce products (and any public post type) can be served as Markdown and listed in llms.txt.
- New: exclude content by category/tag slug, in addition to the per-post checkbox.
- New: "View Markdown" preview link in the post editor and post list.
- Improved: page-cache compatibility — sets DONOTCACHEPAGE for WP Rocket, W3 Total Cache, WP Super Cache and LiteSpeed.
- Fixed: list item text (simple <li> entries) was dropped from the Markdown output.
- Initial release: content negotiation (Markdown for AI bots on the same URL), Schema.org/JSON-LD front matter, post-level Markdown cache, virtual llms.txt / llms-full.txt, and a "Hide from AI bots" metabox.