How Cloudflare Pay Per Crawl Charges AI Bots Per Page

What is Cloudflare Pay Per Crawl?

Introduced by Cloudflare on 1 July 2025, Cloudflare Pay Per Crawl lets a site owner charge an AI crawler a set price for every page it fetches, instead of only choosing between blocking the crawler entirely or letting it in for free. Cloudflare's own announcement, checked 9 September 2026, frames the feature as a middle option between those two extremes, aimed at content owners who want compensation for the AI crawl traffic hitting their sites. At announcement, Cloudflare stated the feature was in private beta.

How does the 402 response work?

A crawler that requests a page under a Charge rule receives an HTTP 402 Payment Required response carrying a crawler-price header that states the cost, according to Cloudflare's own announcement. The mechanism repurposes a status code that has existed in the HTTP specification for decades but saw almost no real deployment, giving crawlers a machine-readable way to learn a price without a human reading a page first. A crawler that wants the content retries the same request with payment authorization attached, and Cloudflare's edge decides whether to let the retried request through.

How does per-crawl pricing work?

A site owner sets a flat, per-request price across the entire site rather than pricing individual pages, per Cloudflare's own announcement. That single number becomes the amount named in the crawler-price header on every 402 response the site sends to a crawler under a Charge rule. The site owner can still exempt a specific crawler from that price entirely, which Cloudflare's announcement describes as a route for a publisher and an AI company to agree on separate terms outside the platform.

Who does Cloudflare Pay Per Crawl apply to?

Cloudflare's own announcement names news organizations, publishers and large content platforms as the site owners the feature targets, and any AI crawler operator willing to pay for access as the other side of the exchange. The mechanism only changes behavior for a crawler request that arrives through Cloudflare's network and that the site owner has placed under a Charge rule; every other request continues to receive whatever response the site would otherwise return. A site with no AI crawler traffic worth pricing gets no benefit from turning the feature on.

What does a site owner have to configure?

A site owner sets payment details and a per-request price inside their Cloudflare account, then chooses one of three settings for each named crawler: Allow, Charge or Block, according to Cloudflare's own announcement. Allow keeps that crawler's access free, Charge attaches the 402 and crawler-price mechanism to its requests, and Block denies it outright regardless of any price. Set the per-crawler choice deliberately, since a crawler defaulted to Allow receives every page for free the same as it did before the feature existed.

How does a crawler prove it paid?

A crawler proves its request under Cloudflare Pay Per Crawl using Web Bot Auth, a scheme built on an Ed25519 key pair the crawler registers with Cloudflare in advance, per Cloudflare's own announcement. Each request then carries an HTTP Message Signature generated with that key, which lets Cloudflare's edge verify both the crawler's identity and that the request was not altered in transit before deciding to admit it. A crawler operator has to complete that registration once, then sign every subsequent request the same way, rather than negotiating payment terms on each individual fetch.

Read the plain robots.txt side of the same problem in the robots.txt guide, since a Block decision under Cloudflare Pay Per Crawl sits next to, not instead of, a standard Disallow rule naming the same crawler. CCBot and ChatGPT-User are two of the named crawlers a site owner decides to allow, charge or block under this system, each with its own separate identity to configure.

Is Cloudflare Pay Per Crawl available to every site?

Cloudflare's own announcement describes the feature as a private beta at launch on 1 July 2025, which means access ran through Cloudflare rather than being open to every customer by default at that point. A site owner outside the beta still controls the same crawlers through a standard robots.txt Disallow rule, the option covered for Amazonbot, just without the priced, machine-readable 402 exchange Cloudflare built on top of it. Whichever route a site takes, what an AI system does with the content it did get to fetch is a separate question from access itself, and the public threads it reads are covered by social media monitoring.

Frequently Asked Questions

When did Cloudflare announce Pay Per Crawl?

Cloudflare announced Cloudflare Pay Per Crawl on 1 July 2025, describing it at launch as a private beta. The announcement is Cloudflare's own primary source for the feature's mechanism and stated availability at that time.

What does the crawler-price header contain?

It carries the price Cloudflare's edge is charging for the requested page, sent alongside an HTTP 402 Payment Required response under a Charge rule. A crawler reads that header to learn the cost before deciding whether to retry with payment.

Can a site owner price different pages differently?

No. Cloudflare's own announcement describes a flat, per-request price set across the entire site, not a price that varies page by page. A site owner can still exempt one specific crawler from the price entirely.

What happens if a crawler ignores the 402 response?

Nothing further is served for that request; the crawler receives the 402 status and the crawler-price header instead of the page content. The crawler only receives the page if it retries with a signed, paid request under Cloudflare's Web Bot Auth scheme.

Does Cloudflare Pay Per Crawl replace robots.txt?

No. Cloudflare's own announcement places Allow, Charge and Block alongside the crawler's existing behavior rather than replacing robots.txt, and a site owner can still write a standard Disallow rule for any crawler it wants blocked outright.

What does a crawler need before it can pay under this system?

An Ed25519 key pair registered with Cloudflare in advance, then an HTTP Message Signature attached to each request using that key, per Cloudflare's own announcement. That registration is what lets Cloudflare's edge verify the crawler's identity before admitting a paid retry.

Who is Cloudflare Pay Per Crawl designed for?

Cloudflare's own announcement names news organizations, publishers and large content platforms as the site owners the feature targets, with any AI crawler operator willing to pay for access on the other side. The mechanism only does something for a site owner who actually gets AI crawler traffic worth pricing. A site with none of that traffic has nothing to switch on.

Can a publisher and an AI company negotiate a separate deal outside Pay Per Crawl?

A site owner can exempt one specific crawler from the flat per-request price entirely, per Cloudflare's own announcement. Cloudflare frames that exemption as the route for a publisher and an AI company to settle their own terms outside the platform. Every other crawler under a Charge rule still pays the single price set for the whole site.

Does Cloudflare Pay Per Crawl change every request a site receives?

The feature only changes behavior for a crawler request that arrives through Cloudflare's network and sits under a Charge rule the site owner set. Every other request gets whatever response the site would have sent before the feature existed. A site owner with no AI crawler traffic worth pricing gets no benefit from turning it on.

Was Pay Per Crawl open to every Cloudflare customer at launch?

Cloudflare described Pay Per Crawl as a private beta at its 1 July 2025 launch, so access ran through Cloudflare rather than sitting open to every customer by default. A site owner left outside the beta could still manage the same crawlers with a standard robots.txt Disallow rule. That route just skips the priced, machine-readable 402 exchange Cloudflare built alongside it.