Linux 软件免费装
Banner图

CrawlerToll

开发者 charthouse
更新时间 2026年10月26日 20:28
PHP版本: 7.4 及以上
WordPress版本: 7.1
版权: Apache-2.0 OR GPL-2.0-or-later
版权网址: 版权信息

标签

paywall content monetization pay per post ai crawlers x402

下载

2.1.0 2.1.1

详情介绍:

Encrypt your content. CrawlerToll is a paywall where the content itself is the lock. Mark a post as premium and place the cut. Everything below the cut is encrypted (AES-256-GCM) before it leaves your server, so visitors, AI crawlers, RSS feeds, the REST API and page caches all get the same thing until someone pays: your preview and ciphertext. After payment the key is released and the reader's browser decrypts the post in place. Everything above is in the free plugin. CrawlerToll Pro adds access passes, section bundles, metered free articles, per-path pricing, per-crawler rail routing, a revenue dashboard, bot-request logs, email alerts, signed unlock webhooks and "Revoke access" for refunds. For developers: what happens on each request
  1. Recognise. The user-agent is matched against a bundled catalogue of 184 declared crawler user-agents: GPTBot, ChatGPT-User, ClaudeBot, Amazonbot, GrokBot, Google-Extended, PerplexityBot, Bytespider, CCBot and more. Recognition is the User-Agent string only. It is not every AI visit, and community lists are larger and still growing. The default rules name seven training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended, Meta-ExternalAgent, Bytespider). PerplexityBot and the rest are recognised and allowed until you add a rule.
  2. Apply policy. Your robots.txt policy, written with RSL-style lines (License:, Permits:, Prohibits:, Compensation:), is applied to the request path.
  3. Answer. HTTP 402 with Crawler-Price, Crawler-Price-Rail and Link headers plus a structured JSON payment offer when policy says so and your site can be paid; 403 "not licensed" when it cannot be paid yet or policy disallows without compensation; otherwise the request passes through with X-CrawlerToll-Action, X-CrawlerToll-Operator and X-CrawlerToll-Bot-Name headers for downstream logging.
Two discovery endpoints for crawlers: /robots.txt (your RSL-style policy, added through WordPress's standard robots_txt filter) and /.well-known/context-license.json (machine-readable licence metadata built from your settings). What this plugin does NOT do CrawlerToll Pro The free plugin is a complete paywall. CrawlerToll Pro adds the revenue tooling: $4.90/month or $32.90/year per site, with a 14-day trial and no card required. See crawlertoll.com/pro. External services Crawler recognition, RSL policy and plain 402 responses run entirely on your server. The sealed paywall and the payment rails use these services. None of them ever receives your article text in the clear. Why CrawlerToll exists AI answers read articles without paying for them, paywalls that only hide content are bypassed, and bot blocking relies on crawlers telling the truth about who they are. CrawlerToll builds on open standards instead of a closed platform: HTTP 402, RSL-style licence lines (rslstandard.org) and x402.

安装:

  1. Upload the crawlertoll folder to /wp-content/plugins/, or install via the Plugins → Add New menu.
  2. Activate through the 'Plugins' menu in WordPress.
  3. Go to Settings → CrawlerToll.
  4. Review the default policy. It names the 7 most common AI training crawlers with a per-crawl price (Compensation: 5000 micros, i.e. $0.005). The bundled catalogue recognises 184 declared user-agents, including PerplexityBot; only those seven are priced, and the rest are allowed until you add a rule. Matching is the User-Agent string only — not every AI visit, and not a ceiling. Crawlers named in the policy are shown that price once your site can be paid (Stripe keys, a USDC address or a payment URL); until then they are told the content is not licensed. Adjust price, currency, rail and policy to fit your site.
  5. Save.
To sell content to people, set your Article price for readers (default $1.00), enter your Stripe keys and/or USDC address, then switch on Premium in the Paywall panel of the post editor and place the cut where the free preview ends. The post is encrypted as it publishes. The plugin starts enforcing immediately. Test with: curl -sI -H 'user-agent: GPTBot/1.2' https://your-site.example/ Once your site can be paid you should see a 402 Payment Required response with the Crawler-Price header; before that, a 403 "not licensed" answer.

屏幕截图:

  • Settings → CrawlerToll: live status, the Get started checklist with links to each fix, and the https warning where cards need it.
  • Revenue dashboard (Pro): realised revenue next to priced 402s, and Traffic: browser requests, declared AI crawlers, search engines and undeclared automation, with the sealed-content funnel.
  • Section pricing (Pro): per-path prices, free metered reads, access tiers and bundles.
  • Bot-request logs (Pro): every AI-crawler request with the action taken, price and content hash, filterable and exportable.
  • What a declared AI crawler gets: a priced HTTP 402 with the offer headers and a link to your machine-readable policy.

升级注意事项:

2.1.1 Recognises 184 declared crawler user-agents, up from 30. Default charge rules are still the same seven training crawlers. PerplexityBot stays allowed until you add a rule. Matching is still the User-Agent string only. 2.1.0 First WordPress.org release. If you used an earlier 0.x version, your pricing, rail and policy carry over; open Settings → CrawlerToll and follow the Get started checklist.

常见问题:

Does CrawlerToll take a cut of my sales?

No. Card payments run on your own Stripe account and USDC is paid to your own wallet. CrawlerToll holds no Stripe secret, no platform account, no facilitator key and no wallet key. Our only revenue is the optional Pro subscription.

Can a paying reader share the article?

Yes. Encryption controls who can read a post, not what a paying reader does afterwards, and every buyer of a post receives the same key. This is true of any paywall. What sealing removes is the free leaks: the article sitting in the page, the RSS feed, the REST API or a cache.

What happens if the unlock service is down?

No payment can start while the unlock service is down: the wall shows that unlocking is unavailable. If a card payment goes through and the article doesn't open, the reader's browser remembers the payment and finishes it on the next try in the same browser, without a second charge. The rest of your site keeps working normally. There is no SLA at launch. See https://crawlertoll.com/docs/unlock-service for what the service stores.

Is there a Pro version?

Yes. CrawlerToll Pro adds access tiers and bundles, read packs, summaries sold separately, metered free articles, per-path pricing, per-crawler rail routing, a revenue dashboard, filterable bot-request logs with CSV/JSON export, email alerts, content provenance, unlock webhooks and access revocation. The free plugin's crawler recognition, 402 responses and sealed paywall with both payment rails remain fully functional without it. See crawlertoll.com/pro.

Can I charge a few cents per article?

Yes. Readers paying with a USDC wallet can pay from $0.001 per article, and the money goes straight to your wallet. Card payments have a $0.50 minimum, so for card readers sell a read pack (Pro), for example 10 articles for $5, paid once on your own Stripe account.

Can AI agents pay a different price from readers?

Yes. Set "Article price for AI agents" in Settings. Declared AI crawlers such as ClaudeBot and GPTBot are quoted and charged that price, in USDC. It can be higher than the reader price, not lower. A bot that hides who it is pays the reader price, and the article stays encrypted either way.

Do I have to invoice anyone?

No — payment settles before access. When a crawler or reader pays, the money lands directly in your own Stripe account or wallet, and only then is the content unlocked. There is no credit, no monthly billing cycle, and nothing to chase. Every paid unlock leaves a receipt you can read back in Settings → CrawlerToll (recent unlocks; the Pro revenue dashboard totals them by rail). On the Stripe rail the payment is in your own Stripe account; readers can ask for Stripe's emailed receipt on the card step. Stripe does not add tax or issue invoices for these payments unless you set that up: you are the seller, so VAT or sales tax on what you sell is yours to handle (ask your accountant). On the x402 rail, the on-chain transaction hash is the receipt, linked in the unlock record to the exact article and time.

Do EU and UK readers have a right to cancel?

Yes, normally 14 days, but for content that unlocks immediately it ends once the reader has agreed to immediate access and acknowledged losing it. With Settings → CrawlerToll → "EU/UK right of withdrawal" on (automatic for sites that look European), the paid options stay locked until the reader ticks that box. The consent time is stored on your Stripe payment and on the unlock receipt, and the reader can save a receipt that states it. You are the seller; this helps you meet the rule and is not legal advice.

Will this block Google or Bing from crawling my site?

No. The default policy targets specifically AI crawlers — GPTBot, ClaudeBot, Google-Extended (Google's training-data crawler, not Googlebot), Applebot-Extended (Apple's training crawler, not Applebot), etc. Search-engine crawlers continue to index your site normally unless you explicitly add them to the policy. PerplexityBot is recognised and allowed until you add a rule.

Does recognising a crawler mean it is priced?

Only if your rules say so. The plugin ships with seven default rules and a catalogue of 184 declared user-agents. PerplexityBot and the other declared agents are recognised and allowed until you add a rule for them. Recognition is the User-Agent string only — it is not every AI visit, and community lists are larger and still growing.

Can I use it alongside Cloudflare Pay Per Crawl?

Yes. They run at different layers and there is no integration between them. Pay Per Crawl runs at Cloudflare's edge; CrawlerToll runs inside WordPress on any host and adds the sealed paywall, the card rail and the human-reader unlock. You can use both, or just one.

Does it work with another hosted paywall?

If you already use a hosted paywall or licensing service, choose the custom payment URL rail and the 402 response will point crawlers at it via a Link: <…>; rel="payment" header. Sealing and the built-in rails are not required.

What if my site is on a multi-site network?

Activate it on each site. Every site has its own settings. Network activation has not been tested.

How do I customise the policy?

Edit the policy textarea in Settings → CrawlerToll. The policy uses standard robots.txt syntax extended with RSL-style directives: License:, Permits:, Prohibits: and Compensation:. See rslstandard.org for the vocabulary they are modelled on.

Does it slow down my site?

The crawler check is a user-agent match and a policy lookup on each front-end request, with no external calls. A sealed post is encrypted the first time it is viewed after it changes; later views reuse the stored ciphertext.

How do I uninstall?

Plugins → Deactivate → Delete. Your /robots.txt returns to its previous form. Settings are kept unless you tick "Remove all data on uninstall" in Settings → CrawlerToll first; then deleting the plugin also removes every option (including your Stripe keys), its tables, per-post data and scheduled tasks.

Where do I get help?

Post in the plugin's support forum on WordPress.org, or email hello@crawlertoll.com.

My site is behind Cloudflare or a reverse proxy. Do the payment limits still work per reader?

Yes. CrawlerToll limits payment attempts per reader, and reads the reader's address from the proxy's headers when the request comes from Cloudflare or from your own server or private network. If your proxy uses other public addresses, add them with the crawlertoll_trusted_proxies filter (a list of IPs or CIDR ranges). Headers from any other source are ignored, so a visitor cannot fake their address.

更新日志:

2.1.1 2.1.0 Also in this release (first published version): Pre-release history (never published on WordPress.org)