SEOmarketeer Bot Flagger

Most analytics trust what a visitor says it is. Attack tools know that, so they call themselves Chrome, or Googlebot, or an AI crawler, and walk straight past. SM Bot Flagger checks what they actually do: a request for your cloud credentials is an attack, whatever name it gives. What it does Each front-end request is written to its own table with:

• date and time, in site time and UTC
• IP address, stored whole or masked
• user agent
• referrer and referring domain
• requested path and query string
• HTTP status code
• whether the request started a new visit, which makes it a landing page
• the category and client name the rules assigned
• how long WordPress took to build the page

Nothing is written to postmeta, and there is one INSERT per request. Categories Traffic is sorted into six buckets:

• Regular traffic — looks like a person in a browser
• Search bots — Googlebot, Bingbot, DuckDuckBot and friends
• AI bots — GPTBot, ClaudeBot, PerplexityBot, Bytespider, CCBot and around thirty others
• Malicious traffic — sqlmap, Nikto, WPScan, plus around 75 path signatures: config and env files, backup leftovers, SSH and private keys, AWS, Azure, Google Cloud, Oracle, OVH, Linode and Hetzner credentials, Docker and Kubernetes, Terraform state, Vite and Spring Actuator file disclosure, path traversal and SQL injection, and credential files for AI coding agents
• Other bots — SEO crawlers, uptime monitors, link previews, HTTP libraries
• Caching — your own cache plugins warming pages: WP Rocket preload and bot, WP Fastest Cache, LiteSpeed crawler, W3 Total Cache, WP Super Cache, FlyingPress, NitroPack, Cache Enabler, Swift Performance, Cloudflare Always Online
• Unclassified — empty or unrecognised user agents

Cache warming is the site requesting its own pages, so it is not a visit and it is not really a bot either. Keeping it separate stops preload runs from inflating the traffic numbers. Filter the log to Caching to check a preload actually ran. Settings has a "Do not log" row where you can tick categories to discard. Those requests are still classified, then dropped before anything is written, so they cost one pass over the rules and no database queries at all. This is worth doing for Caching on a busy site, where a preload run can easily be the largest source of requests. Nothing is stored, so skipped traffic cannot be recovered later and will not show up in the log, the charts or the API. A rule matches on one of three things: the user agent, the request path, or the IP address. User agent and path patterns match anywhere in the value, with * as a wildcard and an optional regular expression. Address patterns understand an exact address, a prefix such as 203.0.113, a wildcard such as 203.0.113.*, and CIDR ranges such as 203.0.113.0/24 or 2001:db8::/32. Prefixes only match on a boundary, so 192.168.1 will not match 192.168.10.4. This is how you handle a crawler that lies about its user agent, or a scraper running from one hosting range: match the address instead. It is also how you mark an internal monitoring path or a staging endpoint as something other than real traffic. Rules live under SM Bot Flagger → Classification. Each rule matches a substring or a regular expression against the user agent, the path or the IP. They are evaluated top to bottom and the first match wins, so put narrow rules above broad ones. You can edit, reorder, disable, delete and restore them. Performance Bot Flagger → Performance shows how long WordPress takes to build your pages, from the build time already recorded with every request:

• the typical page (median) and the slowest 5%, compared with the period before
• a chart per day, or per hour for the last 24 hours
• the slowest pages, each linking to its requests in the traffic log
• build times per visitor type, so you can see whether bots get slower pages

It starts with Regular traffic only, and you can add any other visitor type. The Dashboard has a small Performance card that follows its own filters. Pages served from a page cache never reach WordPress, so they are not measured. Privacy SM Bot Flagger processes personal data in the sense of the GDPR, because an IP address is personal data. The defaults are chosen to keep as little as possible:

• IP addresses are masked before storage, to /24 for IPv4 and /48 for IPv6. Full addresses are stored only if you switch that on.
• Visitors are counted through a hash of the masked address, the user agent and a secret that changes every day, so nobody can be followed from one day to the next.
• Values that look like secrets in URLs, such as password reset keys and tokens, are removed before a row is written.
• Administrators are not logged by default.
• Entries are deleted after 90 days by default.
• Nothing leaves your server. There is no tracking script, no external service, no call home. The plugin author never sees your data.

The plugin adds suggested text to your site's privacy policy guide under Settings > Privacy > Policy Guide. Adapt it to your configuration. If you store full addresses or keep entries longer, say so there, and check your legal basis: for most sites that is a legitimate interest in security, which should be recorded in your processing register. The optional MCP endpoint lets a tool you connect read the log. Whoever holds a token can read the same data you can, so treat tokens as passwords and revoke the ones you no longer use. The MCP endpoint Enable it under Settings, then create a token under MCP API. Endpoint: https://your-site.tld/wp-json/sm-bot-flagger/v1/mcp Transport: streamable HTTP, JSON-RPC 2.0 over POST Auth: Authorization: Bearer <token>, or X-MCP-Token: <token> if a proxy strips the Authorization header Tools:

• get_traffic_summary — totals and the category split for a period
• get_timeseries — hourly or daily counts per category
• get_top — ranks pages, landing pages, referrers, referring domains, user agents, clients, categories, status codes, IPs, rules or hour of day
• search_hits — individual requests, with filters
• classify_user_agent — test a user agent against the current rules
• list_classification_rules — the rules in evaluation order
• upsert_classification_rule — add or change a rule, write token only
• delete_classification_rule — remove a rule, write token only

Read tokens can only read. Give a token write access only if you want the client to change how traffic is categorised. If Authorization never arrives, add this to .htaccess on Apache: SetEnvIf Authorization "(.*)" HTTP_AUTHORIZATION=$1 Hooks Extension points for add-ons: sm_bot_flagger_categories (array) — add your own traffic categories. They appear in the rules editor, the charts, the filters and the MCP schemas automatically. sm_bot_flagger_classification (array) — override the category, label or rule for a request, after the rules have run. sm_bot_flagger_mcp_tools (array) — register MCP tools. Include a callback key and the free plugin will dispatch to it. sm_bot_flagger_stats_where (array) — append conditions to every stats query. sm_bot_flagger_admin_menu (action) — add a submenu page under SM Bot Flagger. sm_bot_flagger_dashboard_cards / sm_bot_flagger_dashboard_panels (actions) — add to the dashboard. sm_bot_flagger_log_columns (array) and sm_bot_flagger_log_column (action) — add columns to the log. sm_bot_flagger_settings_fields (action) and sm_bot_flagger_sanitize_settings (array) — add settings. SM_Bot_Flagger_Logger::current_classification() classifies the current request once and caches it, so an add-on can act on a crawler early rather than at shutdown. sm_bot_flagger_should_log (bool) — return false to skip logging a request. sm_bot_flagger_hit_data (array) — change the row before it is written, or return an empty array to drop it. Useful for adding a country lookup. sm_bot_flagger_mcp_tools (array) — add your own MCP tools. Give each one a handler named tool_<name> on a class of your choosing and hook tools/call accordingly. Notes on accuracy

• Logging happens on shutdown, so requests served entirely from a page cache or a CDN never reach PHP and are not counted. On a cached site, treat these numbers as a lower bound.
• Bot detection relies on self-reported user agents. Anything determined to look like a browser will be counted as regular traffic.
• Only pick a proxy header for the IP if a proxy actually sets it. A client can send X-Forwarded-For itself, which would let anyone forge their address.

Security notes

• Query string values for parameters like key, token, code, password and nonce are replaced with [redacted] before a row is written, so password reset links and OAuth callbacks do not end up sitting in the log table.
• CSV exports prefix any cell starting with =, +, - or @ with an apostrophe, so a crafted user agent cannot run as a formula when the file is opened.
• The MCP endpoint blocks an address for 15 minutes after 20 failed token attempts.
• Tokens are not accepted as a ?token= query parameter by default, because query strings leak into access logs and referrers. Enable it with the sm_bot_flagger_mcp_allow_query_token filter if a client cannot send headers.
• A regular expression that does not compile is switched off rather than being treated as literal text, so a typo cannot silently widen what a rule matches.
• Rule patterns are matched against at most the first 1000 characters of a header, which bounds the cost of a badly written expression.

A write-scoped MCP token can change classification rules, and those rules run on every front-end request. Treat a write token as trusted, and hand out read tokens for anything that only needs to look at the numbers. Behaviour rules Pattern rules judge a request on what it looks like. Behaviour rules judge a client on what it has been doing. They run at log time, because they need the response status and a short memory, and they need a stored IP address, so they do nothing when IP storage is off. Confirmed search and AI crawlers are never demoted by them. Random paths. A 404 on a single path segment of 4 to 12 alphanumeric characters with no vowels and mixed case, such as /6bXX29bt, is treated as scanning. Real slugs contain vowels; generated tokens do not. The 404 is required, so a short consonant heavy page of your own is never caught. Repeated 404s. Flagged on both a count and a share: by default 8 misses within 10 minutes, and at least half of that client's requests in the window missing. The share is what makes this safe. A busy legitimate client can collect a handful of 404s among hundreds of good requests, and counting alone would flag it. Scanners miss on almost everything they ask for. Login attempts. A cumulative count of requests to wp-login.php and xmlrpc.php, 10 per hour by default. Deliberately not a rate: credential stuffing is usually slow and spread over many addresses, so a per-second limit never sees it. Request rate, off by default. On real traffic the fastest clients are usually legitimate, while scanners and credential stuffing run slowly enough to pass any threshold worth setting. Counting 404s separates the two far better than counting requests. Measure your own traffic before switching this on. Once a client trips a rule it stays flagged for an hour by default, and what it already did in that window is recategorised too. Path precedence A user agent is a claim the client makes about itself. Anything can send "OAI-SearchBot" or "Googlebot", and plenty of attack tooling does, because it gets past naive filtering. Settings has "Trust the path", on by default, which tests path and address rules before user agent rules. A request for /root/.aws/credentials is then recorded as an attack whatever it claims to be, and your genuine crawler rules stay correct for genuine crawlers. Turn it off if you would rather rules were evaluated strictly in the order shown on the Classification screen. Filtering Dashboard tiles toggle categories, and several can be combined. Rows in the top lists are links: click a page, landing page, status code, client or referring domain to open the traffic log filtered to it. The log itself filters on category, status family (2xx to 5xx), an exact status code, a page path, landing pages only, and a free text search across path, user agent, referrer, IP and client. Combining a path with "landing pages only" gives you the entry traffic for one page. Filters survive paging and are applied to CSV exports. Correcting a classification Open the Traffic log and click the client name on any row. You get the user agent as an editable pattern, a name, and a category. Shorten the pattern to the distinctive part before saving, for example MyScraper/2. rather than the whole string, so the rule catches other versions too. Saving puts the rule first in the evaluation order, so it beats the shipped defaults. Tick the box to also rewrite requests that are already in the log, which is matched on the user agent containing your pattern. The chart above the table follows the filters, and can be grouped by category, by client or by the raw user agent. The eight busiest values get their own band and the rest are folded into "Everything else". "Show full user agent" prints the whole string under each client name.

1.2.0

• New: a Performance page showing how long WordPress takes to build your pages, as the typical page and the slowest 5%, compared with the period before.
• New: a build time chart per day or hour, a Slowest pages table and a By visitor type table.
• New: a Performance card on the Dashboard, replacing the Avg render tile.
• New: sort the traffic log slowest first, with a Build time column.

1.1.0

• Fixed: Ignore IPs now works with an address copied from the traffic log.
• New: the settings screen shows your own address, whether it is ignored, and an Add my address button.
• New: a warning when your site seems to sit behind a proxy or CDN.
• New: remove requests logged before an address was ignored.
• Faster address lookups on large logs.

1.0.0

• First public release.