Coywolf Robots.txt Manager

Coywolf Robots.txt Manager turns robots.txt into a managed list of rules you build from the WordPress admin — no FTP, no hand-editing. It ships the full catalog of bots from the Cloudflare Bot Directory, grouped by category (AI Crawler, Search Engine Crawler, Advertising, Security, and more), so you can disallow whole categories of unwanted bots with a click.

• Rules table — every directive as a row with a name and plain-English description, with bulk delete and pagination. Identical directives are consolidated.
• Direct editor — a Robots.txt page shows the current file in an editable box; saving re-parses and optimizes your hand edits into managed rules. A copy button puts the whole robots.txt on your clipboard.
• Guided rule editor — choose a Rule Type (Disallow or Allow) and a Rule Path (folder, prefix, file type, query string, allow-exception, and more); the editor builds the correct directive, with auto-filled name and description, plus a URL testing tool.
• Conflict checking — every add or edit is checked against existing rules; if it duplicates or contradicts one, the editor explains the clash and won't save until it's resolved.
• Bot catalog — every bot from the Cloudflare Bot Directory, plus a curated set of well-known AI crawlers it doesn't list yet (ClaudeBot, CCBot, Google-Extended, Applebot-Extended, and more), grouped by category, with select-all per category. The catalog ships with the plugin and is refreshed through plugin updates.
• Curated robots.txt tokens — the exact User-agent token each bot obeys is curated for all 600+ bots (the directory's own match data is often a URL, operator name, or generic word), so rules block the right token and imported robots.txt files match the right bot.
• Commented output — each rule is written as its own section prefixed with a comment carrying its name and description; a rule's bots are stacked under User-agent: lines with the directive written once.
• Physical or virtual — write a real robots.txt in your site root, or let WordPress serve it virtually. On install, an existing physical file is detected, managed, and imported automatically.
• Backup & restore — your original robots.txt is captured before the plugin reads, parses, or rewrites anything; on deactivation it asks whether to restore that original or keep the rules it built (markers removed).
• Import & export — on the Import / Export Rules page, download all rules and settings preferences as one JSON file, then import it to restore them or copy the setup to another site. Import replaces the current rules; sitemap URLs aren't included (they're site-specific).
• Optional comments — the per-rule description comments can be excluded from robots.txt under Settings → Rule comments, and added back any time.
• Role-based access — under Settings → Access, choose which user roles can see and use the plugin; Administrator is always allowed and can't be locked out. Only administrators can change the access list and the serving mode.
• Cleans up on import — fixes misspelled directives, missing colons, and missing leading slashes; consolidates duplicate user-agent groups and resolves Allow/Disallow conflicts; strips deprecated/unsupported directives (Crawl-delay, Noindex, Nofollow, Host, Request-rate, Visit-time); removes render-blocking /.css and /.js blocks and the overly broad /wp-content/; fixes malformed wildcards; and makes Sitemap links absolute and de-duplicated, dropping any that point at a different domain (also handles a UTF-8 BOM and Google's 500 KiB limit).
• XML sitemaps — add sitemap URLs under Settings → XML sitemaps (written at the end of robots.txt), or exclude sitemap links entirely.
• Owns the virtual robots.txt — in virtual mode the plugin serves the whole file. WordPress's own default rules (Disallow: /wp-admin/) are imported and reconciled into your managed rules when you activate or switch to virtual (WordPress's rules win, so its "*" rule absorbs a matching per-bot rule), and WordPress stops adding a duplicate block on top. Other SEO plugins' additions (e.g. Yoast) are excluded too. Physical mode is served by WordPress directly.

1.0.55

• Render plugin screenshots in the GitHub readme (#58).

1.0.54

• Move Import / Export to its own page, exclude sitemap URLs from the export, rename Rules to All Rules, and pad the copy button (#57).

1.0.53

• Add JSON import/export for rules and a copy button on the Robots.txt page (#56).

1.0.52

• Refuse a new rule already covered by a broader Disallow (#55).

1.0.51

• Update Settings screenshot and caption to show the new Access controls (#54).

1.0.50

• Keep readme.md in the WordPress.org build so the Documentation page works there (#53).

1.0.49

• Add role-based access control (Settings → Access) and clear all Plugin Check warnings (#52).

1.0.48

• Fix WordPress.org review: get_home_path() for robots.txt path, prefix class names (#51).

1.0.47

• Fix WordPress.org review issues: dynamic admin-ajax path, no remote calls, matching text domain (#50).

1.0.46

• CI: upgrade actions to Node 24-native versions (#49).

1.0.45

• CI: run bundled JS actions on Node 24 (#48).

1.0.44

• Set distinct Plugin URI and Author URI (#47).

1.0.43

• Add WordPress.org screenshots + captions (#46).

1.0.42

• Fix readme Contributors/Tags header line (#45).

1.0.41

• Set readme Contributors to jonhenshaw (#44).

1.0.40

• Fix all WordPress.org Plugin Check errors (#43).

1.0.39

• Add WordPress.org variant build + gated SVN deploy (#42).

1.0.38

• Migrate existing rules and robots.txt when a bot token is renamed (#41).

1.0.37

• Replace 38 guessed bot tokens with the documented user-provided tokens (#40).

1.0.36

• Exclude bots with no robots.txt token (Link CLI); auto-exclude in the maintenance flow (#39).

1.0.35

• Update the Cloudflare bot list (+4 bots) with curated tokens (#38).

1.0.34

• Security and performance audit — all checks passed (no functional changes) (#37).

1.0.33

• Add a Documentation page; document conflict resolution + bot-token curation (#36).

1.0.32

• Own the virtual robots.txt: import & reconcile WordPress's base rules (#35).

1.0.31

• Use bedrockbot for all Amazon Bedrock AgentCore Browser variants (#34).

1.0.30

• Set ThousandEyes token to TE/1.0 (its documented identifier) (#33).

1.0.29

• Curate robots.txt tokens for all 603 bots + add a bot-list update process (#32).

1.0.28

• Override Yoast SEO's robots.txt output when this plugin is active (#31).

1.0.27

• Add a curated AI-crawler supplement so imported AI bots match the catalog (#30).

1.0.26

• Make the Edit Rule Delete control a bordered red button, aligned with the row (#29).

1.0.25

• Tint robots.txt directive chips by type (light red Disallow, light green Allow) (#28).

1.0.24

• Move rule Edit/Delete into row-actions; add Delete to the edit screen (#27).

1.0.23

• Space out the physical/virtual resolver buttons (#26).

1.0.22

• Match imported User-agents to catalog bots instead of adding them as custom (#25).

1.0.21

• Pad the Cancel button on the deactivation modal (#24).

1.0.20

• Security + performance audit: sanitize bot catalog on ingest; cache hot paths (#23).

1.0.19

• Add an editable Robots.txt page that re-parses rules on save (#22).

1.0.18

• Back up the original robots.txt before any parse/import; add a Settings option to exclude per-rule comments (#21).

1.0.17

• Importer drops cross-domain sitemaps; Rules table bulk delete + pagination; reactivation cleans orphaned comments (#20).

1.0.16

• Back up robots.txt on activation; offer restore-or-keep on deactivation (#19).

1.0.15

• Harden robots.txt import: repair syntax, resolve conflicts, drop deprecated/SEO-bad, fix patterns & sitemaps (#18).

1.0.14

• Add Rule: show a Cancel button when no Rule Path is selected (#17).

1.0.13

• Rework Auto-update settings to match the Link Checker auto-run UX (#16).

1.0.12

• Split editor into Rule Type + Rule Path; move XML sitemaps to Settings; unwrap physical file on deactivate (#15).

1.0.11

• Per-rule comments in robots.txt; remove import button; block conflicting saves (#14).

1.0.10

• Fix stuck readme.txt Stable tag (1.0.0) and make the release bump robust (#13).

1.0.9

• Sitemap URL editor; default "All robots" off + lock for whole-site rules; manage existing robots.txt on activation (#12).

1.0.8

• Fix duplicated rule on physical→virtual switch; delete file on that switch (#11).

1.0.7

• Updater: Atom-feed fallback + visible failure when GitHub API is rate-limited (#10).

1.0.6

• Sitemap rule type, auto-switch to virtual, Link-Checker-style auto-update with email (#8).

1.0.5

• Fix stray Disallow: / under User-agent: * when a rule mixes * with specific bots (#7).

1.0.4

• Sitemap at end of robots.txt; resolver button labels + spacing; scrollable preview (#6).

1.0.3

• Rules UX + round 2: consolidation, conflicts, Yoast override, sitemap-exclusion, physical resolver (#5).

1.0.3

• "Import from robots.txt" appears only when a physical file exists; in virtual mode existing rules are imported automatically on first use. On install, an existing physical robots.txt is managed by default and its rules imported.
• The current robots.txt is always shown on the Rules page (white background, scrolls on overflow); heading renamed to "Current robots.txt".
• robots.txt output consolidates user-agents that share the same rule under stacked User-agent: lines.
• Rule descriptions say "selected" where they refer to the chosen crawlers.
• After a bot-list update, newly added bots are checked in a rule only if its whole category was selected.
• Bot category lists are indented with the checkbox in the list-marker position so wrapped lines hang-indent.
• New rules are checked for conflicts before saving, with a dialog to fix or save anyway.
• Settings: option to exclude XML sitemap link(s) from robots.txt (also suppresses the one WordPress adds).
• If Yoast SEO is active, this plugin overrides it and controls robots.txt.
• When in virtual mode while a physical robots.txt exists, the Rules page warns and offers to switch to physical (import the file's rules), switch to physical (overwrite the file with the current rules), or delete the physical file.

1.0.0

• Initial release.