Crawlune – AI Crawler Controls

Crawlune audits how easily your published content can be extracted, controls AI crawler access, and publishes machine-readable summaries. These features run locally and do not require an account or license. It does not measure or promise citations by answer engines. Hosted rewriting is available with a Crawlune license from WP Shelf. Verify your account email to start a one-time trial of 25 items on one site; no credit card is required. One rewritten opening, one heading rewrite or one grounded FAQ set uses one item. Additional one-time packs are available in your account. Local audits, llms.txt and crawler controls remain free without an account. Open WP Shelf → Crawlune → Settings for the trial and sign-in links, then paste your Crawlune license key into the License card. Start the 25-item trial Plans and item rules Implemented:

• Per-agent AI crawler controls for GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and Google-Extended, distinguishing training crawlers from retrieval crawlers
• Three named positions to start from — Open, "Answers yes, training no", Closed — each setting every crawler and the matching content signals in one go
• robots.txt directives generated from those rules
• Optional refusal (403) for a crawler you blocked that requests pages anyway — off by default, and the requests keep being counted
• A count of people arriving from ChatGPT, Claude, Perplexity, Gemini and Copilot — click-throughs, which is a floor under citations rather than a measure of them
• Content signals (search, ai-input, ai-train) in robots.txt, which state what a page may be used for rather than who may fetch it — off until you set them, and a declaration rather than an enforcement
• The same refusal of AI training repeated as a noai response header, a robots meta tag and a W3C TDM reservation at /.well-known/tdmrep.json
• Markdown for agents: every page also available at its own address with .md on the end, and on request via Accept: text/markdown — off by default, and honouring the same exclusions as llms.txt
• IndexNow submission when you publish or update a page — off by default, and honouring the same exclusions as llms.txt
• An extractability audit scoring content on liftable answer blocks, question-shaped headings, external citations, quotable statistics, modification dates and thin content
• A site-wide visibility report aggregating that across all published content, worst pages first
• /llms.txt and /llms-full.txt, generated from your real content and refreshed when you publish
• Admin screens for the report, a per-page work-list behind every number, crawler controls, llms.txt and licensing, plus REST routes for the same
• Export the searched and sorted loaded Pages list across table pages as a private CSV, with saved measurements, manual steps and explicit freshness and scope limits. Search wording is included only when you request this download; the separate filter-wide export remains available without JavaScript.
• Background analysis of your content, so none of the above is computed while you wait for a page to load
• Structured data for your pages when no SEO plugin is already providing it
• AI rewriting of a page's opening into a liftable answer block, reviewed as a word-level diff before anything is applied
• AI rewriting of section headings toward the question they answer, keeping each heading's level and its existing link address
• Generated FAQ data for pages that answer questions they never ask, where every proposed answer has to be text already on the page — anything else is discarded before you see it
• Applying a rewrite without disturbing surrounding block markup, and reverting a run from a journal
• License activation against the WP Shelf licensing service

Reviewing or applying an existing result does not generate another item. A service-review pause preserves unused items and existing results. Contact WP Shelf support, then explicitly retry after the review is complete. Not included:

• Citation tracking — whether a given engine actually quoted you. Counting people who click through from an assistant is implemented and is a different thing: it is a floor under citations, not a measure of them

Crawlune does not claim it will get your site cited by ChatGPT, Perplexity, Claude or Google AI Overviews. That is not measurable from inside WordPress and will not be claimed. What it reports is what it measured about your content.

0.72.0

• Compare recorded retrieval page requests across two adjacent completed-day periods, with exact site-calendar dates and count differences. Incomplete retained history and unavailable reads are explained; no new tracking, settings change or AI request occurs.

0.71.0

• Confirm saved llms.txt settings before reporting success; refused or partial writes provide explicit recovery without claiming rollback.
• Keep form entries mounted when a save fails or its preview becomes empty, and hide unconfirmed saved output until an explicit retry.

0.70.0

• Export all search matches from the loaded Pages list in their browser sort order, across table pages, without another server request or analysis.
• Name loaded-row limits, retrieval and export times, saved-measurement freshness and private search/editor-link contents in the CSV. Unavailable or empty lists cannot export stale rows; browser preparation failures permit an explicit retry.

0.69.0

• Keep the Pages finding filter, search, sort and page on its browser-history entry for Back, Forward and reload. Search wording stays out of the URL and exported CSV; choosing Pages in the sidebar starts fresh. Changed work-lists clamp the restored page and failed reads retain recovery context.

0.68.0

• Failed saved-analysis report reads show an explicit retry instead of zero counts or an empty priority list. Retry keeps analysis progress and settings; independent crawler controls remain available.

0.67.0

• Analyse the current saved published page from its editor without submitting unsaved edits, spending AI items or restarting the site scan.
• Explain published-only eligibility and show failed score reads separately from unanalysed content.

0.66.0

• Inspect the saved measurements, analysis time and limits behind each page score without running another analysis.

0.65.12

• Show unavailable page work-lists as an error instead of an empty healthy result after database failures.
• Retry or refresh the repair list while keeping its filter, search, sort and page.
• Keep history and rewrite review reads explicit on failure, preserving a draft while stale actions and counts stay unavailable.
• Show saved approvals as read-only, keep a differing unsaved draft separately for copying, and offer only decisions the current item can accept.
• Reload saved suggestions after a failed run read before re-enabling decisions, keeping the open wording intact.

0.65.11

• Pause incomplete site analysis on database read or write failure and retain the last confirmed page.
• Retry paused analysis without restarting completed work; keep saved results visibly separate from a complete scan.

0.65.10

• Fix: keep review wording owned by its pending decision and ignore receipts from a previous screen or run.
• Fix: keep Activity filters after undo, ignore receipts after navigation, and explain unconfirmed undo requests.

0.65.9

• Export the selected measured repair list as a private CSV with page/editor links, saved scores and analysis times.
• Include export counts, scan freshness and snapshot limits without generating content or changing pages.

0.65.8

• Respect AIOSEO per-post and inherited indexing preferences in free summaries and Markdown eligibility.
• Recheck AIOSEO indexing preferences on every origin summary request, including after direct model changes, and withhold pages when their preferences cannot be read.

0.65.7

• Keep interactive analysis and editor screens available on supported older WordPress versions.
• Turn measured page findings into free manual repair checklists with links to the relevant editor and site settings.
• Explain which changes need a fresh analysis and keep optional paid rewriting separate from free repairs, without promising citations.

0.65.6

• Fix: keep applying recoverable when the saved-suggestion count cannot be read.
• Fix: applying approved suggestions keeps unreviewed saved suggestions available for a later apply. Earlier writes remain explicitly undoable after review or cancellation.
• Paused runs with saved suggestions can stop remaining generation and enter review without a refill; approval, apply and undo remain explicit.
• Quota pauses link to the product account allowance without promising a period reset; saved suggestions remain visible, and service safety pauses never prompt a purchase.
• Find the free trial directly from Rewrite, then return to activate its key.
• Start with five suggested items on a first run and continue from an activated license to rewriting or product-specific item packs.
• Go to free analysis before rewriting when analysis is missing or outdated. Previews and draft creation are clearly separate from starting generation.

0.65.5

• Preserve heading anchors and adjacent inline text when reviewing, applying and undoing a heading rewrite.
• Exclude hidden heading markup and refuse ambiguous source ranges before changing an article.
• Clear stale run controls on navigation and refresh saved review statuses when generation completes.

0.65.4

• Add product-specific trial signup and sign-in links beside the license field, keeping the selected plugin through account setup and email verification.
• Explain the 25-item one-time trial, one-site limit, email verification and key connection steps. Local features remain free without an account.

0.65.3

• Make Markdown and IndexNow URLs work immediately after settings changes, including plain permalinks, and retire stale key routes safely.
• Align public discovery with content exclusions, honor Markdown Accept preferences, prevent negotiated responses from entering shared caches, and refresh summaries after relevant metadata changes.
• Explain physical-file overrides when saving visibility settings.

0.65.2

• Require a saved recovery snapshot before applying answer-block, heading or FAQ rewrites, and refuse unsafe retries after manual edits, newer rewrites or undo.
• Resume interrupted writes from their existing snapshots, verify saved content and FAQ data, accept equivalent FAQ saves and recover missing item receipts and applied counts.
• Serialize apply and undo, stop pending writes before restoring a run, keep older applying runs moving when newer drafts exist, and explain blocked undo attempts consistently across screens.

0.65.1

• Fixed: on the Pages screen, the Extractability column showed "Not scanned" for every page after the first, even for pages the audit had scored. Every page now shows its own score, read in one query for the whole list.
• Fixed: clicking the Extractability column header a second time flipped the arrow but not the rows. The list now sorts highest-first when the arrow says so, and pages the audit has not reached stay at the bottom either way.

0.65.0

• Fixed: applying an answer-block rewrite to a page whose content starts with a code block, a picture or a similar element before the first paragraph could replace that element instead of the paragraph. Only the first real paragraph is now replaced, the element before it is left exactly as it was, and block-editor markup around it is kept. Rewrites queued by an earlier version are applied the same safe way.
• Fixed: the audit measured a code block or picture before the first paragraph as the page's opening answer, and counted an image-map area link as a citation. Scores on pages like that can change after updating.
• Fixed: unticking one crawler on the Visibility screen while a preset was selected saved the preset again, so the crawler stayed allowed. The change you make is now the one saved, and the screen shows your rules as custom.
• New: if your license key stops being accepted — for example because we replaced it at your request — Settings → License says so and shows where to enter the replacement, and administrators see one notice on Crawlune's other screens pointing there. Before, rewrite runs failed without saying why. The audit, llms.txt and crawler controls keep working with no key.
• Changed: saving on a Crawlune settings screen now shows that it is working, then moves keyboard focus to the result, so you — and a screen reader — know at once whether the save worked. A failed save says so in a message you can dismiss, next to the form, instead of a paragraph above it.
• Fixed: on a Crawlune screen opened directly — from a bookmark or a reload rather than through the menu — its interactive parts (confirmations, inline actions) did not start until something redrew the screen. They now work from the first load.
• Changed: on a site running several WP Shelf plugins, the newest version of the shared admin styles is used, whichever plugin was installed or updated last. Before, updating an older WP Shelf plugin could make every WP Shelf screen fall back to its older styles.

0.64.2

• No visible change: on a site with no licence key, the plugin no longer reads the licence server's address on every admin screen that checks licence state. It was one database query per screen whose result could never be used. The shared WP Shelf code now checks for a key first.
• Fixed: the action links in a WP Shelf table row (for example a link next to a button) sat a few pixels apart vertically. They now share one line. This comes from the shared WP Shelf screens, and Crawlune's copy had not yet taken it.

0.64.1

• Fail a rewrite whose result is not text, instead of offering a fragment of program output for review as a heading or an answer.
• Say plainly that such a failure is ours to fix rather than showing a raw code.

0.64.0

• Save the crawler settings on the Visibility screen without reloading the page: the form posts to the plugin's own REST route, reports the result in place, and redraws the screen from what was actually stored.
• Keep the plain form post working unchanged for browsers with JavaScript turned off, through the same save.

0.63.0

• Show, under a finished rewrite run, how the score of each page it changed has moved: the median score before the run, measured on the copy Crawlune kept of each page, against the score from the page's latest analysis.
• Name pages that have not been analysed since the run instead of showing them as zero, and count pages put back or edited since separately.
• Say why a FAQ data run leaves page scores as they were.

0.62.6

• Recover interrupted dispatches and partial result delivery without issuing duplicate paid work.
• Preserve saved results during a service-review pause and resume only on an explicit request.
• Continue recovering older multi-batch runs without requiring an open admin screen.
• Restore the complete rewrite history for each run, including large runs, while protecting newer rewrites.
• Preserve quotes and backslashes in saved article text and generated FAQ data.
• Report partial undo results accurately and keep unsuccessful restores available to retry.
• Show one-time allowances without a reset date, and display resets only when the service confirms renewal.

0.62.5

• Escape assembled admin HTML at output while retaining form controls and nonces.
• Enforce non-HTML content types and nosniff for text and Markdown endpoints.
• Simplify JSON encoding flags while retaining JSON-LD script-injection protection.

0.62.4

• Guide installations without a local license key to settings before creating a hosted rewrite run; existing runs and free tools remain available.
• Fixed: malformed service acknowledgements now use bounded retries instead of leaving a rewrite run waiting indefinitely.
• Fixed: unavailable results stop retrying after six hours even when the service reports the job complete; valid late results still arrive normally.
• Improved: failed batch messages explain the next step without implying that allowance was refunded.

0.62.3

• Renamed the plugin to Crawlune – AI Crawler Controls, including its translation domain.
• Secured JSON-LD output against script-element breakout in page metadata and FAQ content.
• Added the submitting WordPress.org account to the contributor credits.

0.62.2

• Fixed hosted generation and result polling on fresh installs with no saved server URL.
• Improved request-input sanitization and documented temporary CSV stream handling.
• Included shared JavaScript sources and standalone build instructions in the plugin ZIP.
• Clarified external-service requests and privacy disclosures.
• Verified clean installation and removal on WordPress 7.1.

0.62.0

• Added JavaScript search, sorting and pagination for admin data tables.
• Navigate between plugin subpages asynchronously, with Back/Forward support. Earlier release notes are included in CHANGELOG.md.