RowanWeb Bot Traffic – AI Crawler & GEO Analytics

Referral traffic from AI assistants is hard to measure: apps and many browsers send no referrer, so it lands in "direct". Hand-checking AI answers is noisy too, because the same question gets different sources on different days. Every AI assistant still has to fetch your pages from your server. RowanWeb Bot Traffic records those requests and sorts them by what the bot is doing, so you can track your visibility in AI answers (GEO) from data your site already has. Bots sorted by intent Bots are grouped by why they visit, not just by name:

• Citation — an AI assistant reads the page right now to answer a person: ChatGPT-User, Claude-User, Perplexity-User, MistralAI-User, DuckAssistBot and others. This is the closest server-side signal of your site being shown to people in AI answers.
• AI search indexing — the page is added to an AI search index for future answers: OAI-SearchBot, Claude-SearchBot, PerplexityBot, YandexAdditional and others.
• Model training — content collected for training: GPTBot, ClaudeBot, CCBot, Bytespider, meta-externalagent and others.
• Classic search — Googlebot, Bingbot, YandexBot and others, for comparison.

The bundled list covers about 60 bots and was checked against the operators' own documentation wherever it exists. ChatGPT agent, which uses a regular browser user agent, is recognized by its signature header. What you see on the dashboard

• Requests by intent with the change against the previous period.
• Filters for the whole dashboard: today, 7, 14, 30, 60 or 90 days (as far as your retention setting allows) or your own dates, plus bots, intents and HTTP status codes. The CSV export follows the same filters.
• A daily chart by intent or by bot, with your own event markers ("updated llms.txt", "rewrote 40 articles"), so you can see what happened after a change.
• The pages AI bots read, with citations, indexing and training per page. The HTML and .md versions of the same post are counted together. You can also switch to raw paths to see requests for robots.txt, llms.txt and feeds.
• Requests by post type and category, and by HTTP status code.
• The most active bots with their intent, requests for .md files and llms.txt, and suspicious requests per bot.
• New bots that need a look. Unknown user agents are reviewed automatically once a day: uptime monitors, SEO tools, link previews, placeholder contacts and one-off visits are filtered out, so a new AI crawler is not lost among them. You can hide a bot you are not interested in, and an admin notice tells you when a new bot keeps coming back.
• Suspicious activity: requests for /.env, /.git/config, backup files and debug endpoints sent under AI bot names. They are kept out of the numbers above.

Privacy The plugin records only requests whose user agent looks like a bot. Visitors are never recorded, and IP addresses are stored only for bot requests. Query strings are dropped, so tracking parameters and tokens in URLs are not saved. Detailed data is kept for 90 days by default; you can choose from 30 days to a year in the settings. Everything is removed when you delete the plugin. Sites behind Cloudflare or a proxy If the site is behind Cloudflare or another proxy, set this up in Bot Traffic → Settings → Client IP address. Forwarded headers are trusted only from Cloudflare addresses, from a Cloudflare Tunnel running on the same server, or from the proxy addresses you list, so a bot cannot fake its IP.

0.1.5

• How long detailed data is kept is now a setting: 30, 60, 90, 180 or 365 days (90 by default). The dashboard offers the periods that fit.
• All database queries are built with prepared statements.

0.1.4

• Chart.js updated to 4.6.1.
• The free version shows no authenticity column; .md columns are hidden when a period has no requests to .md or llms.txt.
• CSV export: intents are written as on the dashboard (citation, ai_indexing, training, classic_search); response size is empty for the PHP logger, which cannot measure it.
• Bravebot added to the bot list.
• Pro: Claude-User requests from outside Anthropic's IP list are no longer marked as spoofed: Claude Code and Claude Desktop fetch pages from the user's own computer.

0.1.3

• Pro: Cloudflare as a data source. It also sees pages served from the Cloudflare cache and from page cache plugins. Bots verified by Cloudflare count as verified.
• A data source that fails (network, token) no longer overwrites the day with zeros; the error is shown in the settings.
• The dashboard shows one data source at a time, with a switch when there are several.

0.1.2

• Pro: fixed bot verification for requests recorded by the PHP logger (the IP was not read, so they stayed "not checked").
• Pro: event effect with a day-of-week adjusted baseline and a 95% confidence interval, or "within noise".

0.1.1

• Cloudflare mode now also works with Cloudflare Tunnel, where requests reach the site from the local cloudflared process.
• Pro: bot verification by the operators' official IP lists and reverse DNS. Spoofed bots are kept out of the numbers.

0.1.0

• First release: PHP logger for bot requests, about 60 bots sorted by intent, hourly processing, automatic review of unknown bots, suspicious activity filter, dashboard with filters and event markers, CSV export, trusted proxy settings.