Site Answers – MCP Server, llms.txt & AI Crawler Log
| 开发者 | lalokupfer |
|---|---|
| 更新时间 | 2026年11月1日 00:28 |
| PHP版本: | 7.4 及以上 |
| WordPress版本: | 7.1 |
| 版权: | GPLv2 or later |
| 版权网址: | 版权信息 |
详情介绍:
search_content, get_post, list_recent and get_site_info. It cannot change
anything: there are no write tools at all, not even switched-off ones. Each token is
revocable and limited to 60 calls a minute, and the access log shows which token used which
tool on which page, and when. If your site runs WordPress's official MCP Adapter plugin, the
same four tools are registered as WordPress abilities too, so its agents get them with no
token at all.
It publishes an llms.txt file. llms.txt is an emerging convention — a plain,
readable Markdown index at the root of your site that tells AI assistants what you
have published and where to find it. Site Answers builds yours from your own posts and
pages: your site name, your tagline, then one line per URL with its title and a short
description. It is rebuilt automatically whenever you publish, edit or delete
something, so it never goes stale.
It publishes llms-full.txt too. Where llms.txt is the map, llms-full.txt is the
whole territory: the full text of every page llms.txt lists, one after another, in one file,
so an AI assistant can read your site in a single visit. Same pages, same exclusions, capped at
5 MB with a note saying where the rest is. You can switch it off in Settings.
It logs which AI crawlers visit. GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot,
CCBot and the others are recognised by name. For each visit you see which crawler
came, which page it asked for, and whether it got the page or an error. That answers a
plain question: is AI search actually reading my content, and which pages does it want?
It serves a clean Markdown copy of every page. Add .md to the end of any post or page
address and you get the article as plain Markdown — headings, paragraphs, lists, links and
tables — with the navigation, sidebars, share buttons and scripts left out. That is what an
automated reader wants, and it is far less likely to be misquoted than a page it had to dig out
of a theme. Each HTML page also advertises its Markdown twin in a Link header, so a crawler
can find it without guessing.
It tells you when your robots.txt is turning AI crawlers away. An empty crawler log often
has a simple cause: a line in robots.txt — added by another plugin, a host, or years ago by
hand — asking GPTBot or ClaudeBot to stay away. The Overview reads the robots.txt your site
actually serves, the way the crawlers read it, and names every AI crawler it blocks.
And it lets you choose, crawler by crawler. In Settings, each AI crawler has three
choices: No change, Allow or Block. Block ChatGPT's training crawler and keep the one that
fetches your page when someone asks ChatGPT about you, or the other way round. Your choices go
into WordPress's own robots.txt. Until you pick something, robots.txt stays exactly as it was.
How Site Answers is different
Site Answers publishes the file, serves each page as Markdown, and shows you which AI crawlers
came and what they asked for, on your own server. Publishing and measuring sit on one screen,
so you can see whether what you published is being read.
Its MCP tools are read-only: there is no write tool, and each token can be revoked in one
click. A token can read only published content of the types you ticked in Settings, never a
draft, a private or a password-protected post, and every tool returns the same clean Markdown
text the .md addresses serve, not raw page HTML for the agent to dig through.
Three further differences, each of which you can check in the code:
- It stands aside rather than fighting for the address. If another plugin, or a static file
in your web root, already serves
/llms.txt, Site Answers never registers its own route for it: theirs keeps working untouched, and your settings screen tells you which one is answering. Two plugins answering one address is how sites break, and it is always the newer plugin that gets blamed for it. - Its logs say what they keep, and keep it on your server. The crawler log keeps the crawler's name, the page path, the response status and the time: no IP address and no query string. The MCP access log keeps which token or which signed-in WordPress user called which tool, the page read or the words searched, whether the call was answered, and the time, for 90 days. Nothing leaves your server.
- It asks for nothing. No account, no API key, no external service, no outbound request of any kind, and nothing to upgrade to. Everything happens on your own server, so nothing about your site or your visitors leaves it.
? in an address is discarded before the record is written, because query
strings on real sites carry e-mail addresses, order references and reset tokens. Records
older than 90 days are deleted automatically, in both logs.
It changes nothing on its own. Your robots.txt is left exactly as it is until you choose
Allow or Block for a crawler, and no file is ever written to your server: if you have a
robots.txt file of your own, the plugin tells you so and leaves it alone.
Why this matters
AI assistants increasingly answer questions using content they read from websites, and being
readable to them is becoming its own discipline — answer engine optimization, or AEO, alongside
the generative engine optimization people now talk about next to ordinary SEO. Both come down
to the same two practical questions: can a large language model find and parse what you
published, and is it actually doing so? Site Answers gives you a straight answer to the second
one, and does the groundwork for the first.
Who it is for
Anyone who publishes writing and wants to know whether AI search engines are reading it.
Bloggers, documentation sites, shops with real content, agencies who need to show a
client what is happening.
Privacy
The crawler log keeps the crawler's name, the page path, the response status and the time:
no IP address and no query string.
The MCP access log keeps which token or which signed-in WordPress user called which tool,
the page read or the words searched, whether the call was answered, and the time, for 90
days. Nothing leaves your server.
On WordPress 6.9 and newer, any signed-in account that can read your site can call the
tools through WordPress's abilities, so on a site where visitors register, the MCP access log
can hold what a visitor's account read or searched. You can switch the abilities off in
Settings. Both logs live in your own WordPress database and are removed with the plugin
when you delete it.
Whether your site needs a consent notice is your decision; this plugin does not decide it for
you.
安装:
- Install and activate the plugin.
- Visit Site Answers → Overview. Your
llms.txtlink is at the top. - Optionally visit Site Answers → Settings to choose which content is listed and which AI crawlers may read your site.
- To let an AI agent read your site over MCP, visit Site Answers → AI Agents (MCP), make a token, and paste the address and the token into your agent. The screen shows a ready-made command and configuration to copy.
屏幕截图:
常见问题:
What is llms.txt?
It is a proposed convention — like robots.txt or sitemap.xml, but written for large
language models. A single Markdown file at /llms.txt that tells an AI assistant what a
site contains, in a form it can read without wading through navigation, adverts and
scripts. It is young and not every AI engine reads it yet, which is exactly why it costs
almost nothing to publish one now.
Where is my llms.txt?
At /llms.txt on your site, and the Overview screen links straight to it.
What is llms-full.txt?
The companion to llms.txt. Instead of a list of links, it holds the full text of every page
llms.txt lists, as Markdown, in one file at /llms-full.txt. Each page starts with its title
and a Source: line with its address. It leaves out exactly what llms.txt leaves out —
drafts, private and password-protected posts, and pages marked "noindex" — and stops at 5 MB.
On a large site the first visit may not include every page yet: pages are converted a few at a
time so the file never times out, and the file says so when that happens.
My llms.txt says "not found". Why?
Almost always because Settings → Reading → "Discourage search engines from indexing
this site" is switched on. Site Answers honours that: if you have asked search engines
to stay away, it does not publish an index of your content on your behalf. The Overview
screen tells you when this is the reason. Switch that setting off and the file appears
immediately.
If that setting is already off, something else is probably answering at that address: a
static llms.txt file in your site's root folder, or another plugin that publishes one.
Site Answers steps aside for both rather than fight over the address, and the Overview
screen names whichever it is.
Which AI crawlers are detected?
GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-User and anthropic-ai (Anthropic); PerplexityBot and Perplexity-User (Perplexity); Google-Extended and GoogleOther (Google); CCBot (Common Crawl); Bytespider (ByteDance); Amazonbot (Amazon); Applebot-Extended (Apple); Meta-ExternalAgent (Meta); cohere-ai (Cohere); DuckAssistBot (DuckDuckGo); YouBot (You.com); and Diffbot. Google-Extended and Applebot-Extended are robots.txt names only: Google and Apple fetch pages with their ordinary crawlers and use these names to let you opt out of AI training. They can be blocked in Settings, but they never appear in the visit log.
How do I get the Markdown version of a page?
Add .md to the end of its address. A post at https://example.com/hello-world/ is also
served at https://example.com/hello-world.md. The HTML page carries a
Link: <…>; rel="alternate"; type="text/markdown" header pointing at it, which is how an
automated reader is meant to discover it.
Drafts, private posts, password-protected posts and anything you have marked "noindex"
are not served this way, exactly as they are kept out of llms.txt. The Markdown copy
also carries X-Robots-Tag: noindex and a canonical link back to the real page, so it
cannot compete with your own article in search results.
What is MCP, and how do I connect an AI agent to my site?
MCP, the Model Context Protocol, is the open standard AI assistants use to connect to tools and data. Site Answers turns your site into a read-only MCP server:
- Go to Site Answers → AI Agents (MCP) and make a token. Give it a name you will recognise, such as "Claude on my laptop".
- Copy the token straight away: it is shown once, and only a fingerprint of it is kept.
- Give your agent the MCP server address shown on that screen and send the token as
Authorization: Bearer <token>. The screen has a ready-made Claude Code command and a JSON block for clients such as Cursor and VS Code that take an address and headers.
search_content (search your posts and pages),
get_post (read one in full, as Markdown, by id or address), list_recent (the newest
content) and get_site_info (your site's name, language, content types and llms.txt
address).
Can an AI agent change my site through MCP?
No. There are no write tools: the code to create, edit or delete anything does not exist in this plugin, so there is nothing a token, a bug or a setting could switch on. A token can only read, and only published content; revoking it stops it on its very next call.
Which content can an agent read?
The published content of the types ticked in Settings, with no password. While Leave
out pages marked "noindex" by my SEO plugin is ticked, as it is by default, those pages are
left out too. Drafts, private posts and password-protected posts are never reachable, even
through an administrator's agent. The siteanswers_llms_txt_exclude_post and
siteanswers_mcp_exclude_post filters withhold any other post in code.
That is the same kind of content llms.txt lists, but all of it: llms.txt lists only the
newest pages, up to the number set in Settings, while an agent can search and read every
page of those types, not only the newest ones. The MCP server also stays open when llms.txt
is not published: while "Discourage search engines" is switched on, or when another plugin or
a file answers /llms.txt. That is on purpose: a token is you personally giving your own
agent a key.
Does it work with the WordPress MCP Adapter?
Yes. On WordPress 6.9 and newer the four tools are also registered with WordPress's
Abilities API as site-answers/search-content, site-answers/get-post,
site-answers/list-recent and site-answers/get-site-info. If you use the official MCP
Adapter plugin, its agents find them with no further setup, signed in as a WordPress user
rather than with a Site Answers token, and their calls appear in the same access log. You
can switch this off in Settings.
My agent gets "401" or "unauthorized". What now?
Check that the token is sent as Authorization: Bearer <token> and has not been revoked.
If it is right and still refused, your host may be dropping the Authorization header before
WordPress sees it. This happens on some Apache servers, and adding
SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1 to your .htaccess fixes it. A security
plugin that blocks the WordPress REST API for logged-out visitors will also block the MCP
address; allow the site-answers/v1 REST route in its settings.
Is there a limit on how much an agent can read?
Each token may make 60 tool calls a minute, and so may each signed-in WordPress user calling the tools through WordPress's abilities (the MCP Adapter). That is plenty for an agent researching your site and stops a runaway one from loading your server. A page longer than 100 KB of text is cut there, and the agent is told so.
Does this plugin send anything to anyone?
No. It makes no outbound HTTP request at all. There is no account, no API key, no telemetry and no analytics of any kind. AI agents you connect over MCP send requests to your site; your site never contacts them. The Overview and AI Agents (MCP) screens carry one ordinary link, "Get early access", to a page about a possible paid add-on. Nothing is sent when a screen loads; the page opens in your browser only if you click, and your e-mail address reaches us only if you type it there.
Is there a paid version?
Not yet. Everything described here is free and stays free. We are considering an optional paid add-on that answers your visitors' questions from your own content; the "Get early access" link on the Overview and AI Agents (MCP) screens lets you ask to be told if it is built. It is never shown as a notice, and nothing in this plugin is locked behind it.
Does it record IP addresses?
No. The crawler log keeps the crawler's name, the page path, the response status and the time: no IP address and no query string. The MCP access log keeps which token or which signed-in WordPress user called which tool, the page read or the words searched, whether the call was answered, and the time, for 90 days. Nothing leaves your server.
My server log shows more AI crawler visits than this plugin does. Why?
If your site uses full-page caching — nginx FastCGI cache, WP Rocket, Cloudflare, a host level cache — then a cached page is served without WordPress running at all, so the plugin never sees that visit. This is true of every WordPress-level bot log, not just this one. Treat the numbers here as a floor rather than a total. On a very busy day there is a second reason: once the crawler log has stored 1,000 visits in a day, it keeps one visit in ten from then on, marks those rows as sampled, and the Overview says so.
Does it work with my SEO plugin?
Yes. If a page is marked "noindex" in Yoast SEO, Rank Math, SEOPress or The SEO
Framework, Site Answers leaves it out of llms.txt by default. You can turn that off in
Settings. For any other rule, the siteanswers_llms_txt_exclude_post filter lets you
exclude a page in code.
Will it list my drafts or private posts?
No. Only published content. Drafts, pending, private, trashed and password-protected posts are never listed, whatever you choose in Settings.
What happens when I delete the plugin?
Everything it created is removed: both of its database tables (the crawler log and the MCP access log), its MCP tokens, every option and cached value it wrote, and its scheduled cleanup job. Deactivating the plugin, by contrast, leaves your crawler history intact so you can switch it back on without losing anything.
Can I block AI crawlers like GPTBot?
Yes, if you choose to. Site Answers → Settings has a choice for every crawler it knows:
No change, Allow or Block. Block adds a Disallow: / rule for that crawler to your
robots.txt; Allow adds a rule inviting it in. Nothing changes until you choose, and each
crawler's row says what it is for — some collect training data, others open your page only
when someone asks an AI assistant about it, and blocking those means the assistant cannot read
you when asked.
robots.txt is a request that well-behaved crawlers follow, not a lock. If your site has a
robots.txt file of its own on the server, WordPress's robots.txt is never used, so the choices
cannot apply — the Settings screen tells you when that is the case.
Why does the Overview say my robots.txt blocks a crawler I never blocked?
Because something else did: another plugin, your host, or a line in a robots.txt file on your server. The Overview reads the robots.txt your visitors actually get and names every AI crawler it turns away, whoever wrote the rule. If you set a crawler to Allow and another rule still blocks it, the Overview says that too.
更新日志:
- Corrected: the Privacy section was wrong about the MCP access log, which keeps which token or which signed-in WordPress user called which tool, and the page read or the words searched. The readme and both screens now describe each log as the code keeps it, and the readme no longer says whether your site needs a consent notice: that is your decision.
- Corrected: the readme and the AI Agents (MCP) screen said a token reads what
llms.txtlists. It reads every published page of the types ticked in Settings, not only the newest onesllms.txtlists, and also whilellms.txtis not published. The words now say so; what a token can read is unchanged. - New: a "How Site Answers is different" card on the Overview. The readme no longer compares this plugin with others; it says what this one does.
- Signed-in users calling the tools through WordPress's abilities now have the same limit as a token: 60 calls a minute each.
- Through the abilities, a post that is missing or not readable now answers 404 instead of 500.
- The log queries name their tables through WordPress's identifier placeholder.
- Deleting the plugin no longer clears WordPress's stored rewrite rules, since the plugin adds none.
- What makes Site Answers different: it publishes your llms.txt, serves each page as Markdown, and shows you which AI crawlers came and what they asked for, on your own server.
- The AI Agents (MCP) screen now carries the same "Get early access" link as the Overview. It is a plain link — the plugin still makes no outbound request.
- New screenshots and banner showing the MCP server first.
- New: a read-only MCP server (Model Context Protocol), at the address shown on Site Answers → AI Agents (MCP), with four tools for AI agents:
search_content,get_post,list_recentandget_site_info. Pages come back as clean Markdown. There are no write tools. - New: Site Answers → AI Agents (MCP). Make and revoke a token for each agent (only a fingerprint is stored), 60 calls a minute per token, and an access log of which token used which tool on which page, and when. No IP addresses.
- New: on WordPress 6.9 and newer the same four tools are registered with the Abilities API, so the official MCP Adapter plugin offers them to signed-in agents. Can be switched off in Settings.
- The plugin's name is now "Site Answers – MCP Server, llms.txt & AI Crawler Log".
- The "llms.txt says not found" answer no longer suggests re-saving Permalinks, which cannot help: the plugin adds no rewrite rules.
- New: a "Get early access" link at the foot of the Overview screen, for a possible paid add-on. It is a plain link — the plugin still makes no outbound request.
- New:
/llms-full.txt— the full text of every page inllms.txt, in one file. Same pages and exclusions, 5 MB cap, can be switched off. - New: the Overview says when your robots.txt asks an AI crawler to stay away, whoever wrote the rule.
- New: allow or block each AI crawler in robots.txt from Settings. Nothing changes until you choose, and no file is written.
llms.txtnow points tollms-full.txtwhile it is published.- Tested on every PHP version from 7.4 to 8.4 with WordPress 6.5, 6.8, 7.0 and 7.1.
- First release.
- Publishes
/llms.txtfrom your published posts and pages, rebuilt automatically on publish, edit and delete. - Serves a clean Markdown copy of any post or page at the same address plus
.md, and advertises it in aLinkheader. - Logs visits from 19 named AI crawlers, with the page requested and the response status.
- Settings for content types, how many entries to list, and whether to honour "noindex".
- No external requests, no IP addresses, no telemetry.