AI agent traffic is already a publisher cost problem: an agent can fetch a thousand pages and send the business to one, while ads still depend on a human arrival. On Cloudflare’s third-quarter call, CEO Matthew Prince said some Cloudflare customers consider AI agents “a bane to their business and want to keep them away,” and the company has predicted agent activity 1,000 times greater than clicks from humans. Newsrooms have to decide what to block, what to charge, and what licensing will not cover.
Updated November 6, 2026
Key Takeaways
- Prince’s 1,000-restaurant example: 999 incur a cost and get no benefit.
- Cloudflare expects a 1,000:1 bot-to-human ratio any year now.
- More than 50 publisher-AI deals are nearly all bespoke and bilateral.
- Pay Per Use, in beta, pays for reported use, not the crawl.
What does AI agent traffic cost publishers?
Prince’s illustration, as reported from the earnings call, is the model. An agent searches 1,000 restaurant websites and menus; the user chooses one. The other 999 incur costs and get no benefit.
On Cloudflare’s network, daily requests from AI agents grew more than 1,700% over the past year, the company wrote in September 2026. In that post it also said that, for the first time, more than half of Internet traffic was not human. On the earnings call, Prince cited an expected bot-to-human ratio of 1,000:1 any year now.
In Retail, Computer Software, IT & Services, and Financial Services, human traffic fell as much as 40% in less than a year. Publishers should not treat that category decline as their own traffic audit.
How do block, Pay Per Use, and licensing compare?
Since July, Search, Agent, and Training are separate controls on every plan, including Free. Fewer than 1% of sites on Cloudflare block search crawlers; 17% block training. An agent that books a table or buys a dataset can be a customer, and turning it away may turn away the person who sent it.
More than 50 publisher-AI deals since 2023 are nearly all bespoke and bilateral, between large publishers and large AI companies. They do not reach most of the web or most buyers, and they rarely show what was used. Anthropic’s licensing deal with AP is that lane.
| Response | When money moves | Who it reaches | Limit stated in sources |
|---|---|---|---|
| Block agents | No payment; the request stops | Sites using Cloudflare controls, including Free | May also turn away the person behind the agent |
| Pay Per Use | On a reported use, not the crawl | Publishers that accept a buyer’s offer; beta | Buyer sets price and use; reports are self-reported |
| Monetization Gateway | Per request, via x402 | Eligible U.S. Cloudflare customers; closed beta | For APIs and tools, where each request is the use |
| Bilateral licensing | A negotiated contract | Nearly all: large publishers and large AI firms | More than 50 deals since 2023; not most of the web |
What should ad-supported publishers charge or block?
On ad pages, Cloudflare’s recommendation is to disallow training and block agents, because an ad pays only when a person sees it. Pay Per Use, now in beta, is the priced alternative. A verified buyer sets a use and a price; the publisher accepts or leaves; Cloudflare bills reported uses and pays monthly.
Pay Per Crawl, launched in 2025, charges for access. Pay Per Use pays for what happens next.
These sources publish no prices, so a use fee cannot yet be compared with a lost impression. The limits that are stated:
- The buyer defines the paid use and sets the price.
- Usage is self-reported, then checked against enrolled publishers.
- Context such as keywords or topic is planned, not yet reported.
OpenAI, Google, and AWS sign requests under Web Bot Auth, and Cloudflare sees more than 500 billion verified bot requests a week. Since July 1, approximately five percent of the web had opted into Cloudflare’s content protection products.
Training is a separate switch: by stated purpose, AI training was 22% of crawler requests in spring 2025 and 52% by June 2026. Disallow AI Training, shipped September 15, keeps search indexing; Apple, Google, and Microsoft have committed to honor it. A shared training opt-out does not price the agent that cites and never clicks.
Frequently asked questions
Why are AI agents costly for website operators?
Matthew Prince said an agent can search 1,000 restaurant websites and menus and the user may choose one, so the other 999 incur costs and get no benefit. He said some Cloudflare customers consider agents a bane and want to keep them away.
How does Pay Per Use differ from bilateral licensing?
Pay Per Use, in beta, pays when a verified buyer reports a defined use, and a publisher can opt out. More than 50 publisher-AI deals since 2023 are nearly all bespoke and bilateral, and they do not reach most sites.
Should ad-supported publishers block AI agents?
Cloudflare advises ad-supported sites to disallow training and block agents on pages that carry ads, because an ad pays only when a person sees it. Blocking an agent can also block the person who sent it.
What shift in AI agent traffic has Cloudflare reported?
Daily AI-agent requests on its network grew more than 1,700% over the past year, and Cloudflare said more than half of Internet traffic was not human. It expects a 1,000:1 bot-to-human ratio any year now.
