After Cloudflare switched on pay-per-crawl and default blocking of unpaid AI crawlers on July 1, 2025, a growing share of content sites, retailers and ticketing platforms have made bot verification, signed identity and paid access the default entry requirement, repeatedly stopping autonomous browsing agents from OpenAI, Perplexity and Google mid-task. The shift narrows the reachable surface of agentic commerce and turns internet access from a free default into a negotiated resource. The core
The sequence unfolded in three stages. First came blocking. On July 1, 2025, Cloudflare made pay-per-crawl its default rule, returning block responses to AI crawlers that had not negotiated payment across a network it says carries roughly 20 percent of global web traffic, affecting millions of domains. Second came identity. In September 2025, Cloudflare, together with OpenAI and Anthropic, introduced Web Bot Auth, using HTTP message signatures to give legitimate agents a cryptographically verifiable identity instead of self-declared User-Agent strings. Third came settlement. That same month OpenAI and Stripe published the Agentic Commerce Protocol, while Google launched AP2 with more than 60 payment and financial institutions as initial partners. In October, OpenAI released the Atlas browser to Plus and Pro subscribers, and Perplexity moved Comet from a $200-per-month Max exclusive to
Three preconditions matured around 2025. The economic one: training-data licensing had already produced real prices, with Reddit and News Corp among those signing deals, so publishers saw revenue in content that had previously been free to scrape, turning blocking from sentiment into a revenue decision. The technical one: reverse proxies such as Cloudflare sit at every entry point to a site, so enforcing something stronger than robots.txt requires a single configuration push, and robots.txt has never carried legal force. The competitive one: advertising and subscription revenue driven by search referrals is being eroded by answer engines, and if an agent completes comparison and checkout on
Liu Gong's take: within the next 12 months, holding a site-issued verifiable identity will matter more to an agent's usefulness than benchmark scores. General-purpose agents that have not completed signature and payment negotiations will most likely fall below a 50 percent success rate in high-value scenarios such as retail checkout and ticketing, while the handful of players that sign licensing deals will secure de facto exclusive entry points.
From a technical standpoint, Web Bot Auth turns "who am I" into a cryptographically verifiable claim using HTTP message signatures, which is structurally the same idea as attaching a traceable marker to every part in industrial quality inspection. In a system without an identity layer, the only available discriminator is behavioural detection, and behavioural detection always costs more at the margin than signature verification. The pattern Liu Gong observes in AI visual inspection is that once one side can observe enough session features, the cost of impersonation rises exponentially. By comparing request-header consistency, DOM event timing and cursor traces, a site can separate an agent from a human within a few sessions. For an agent to look human it must deliberately slow down, which contradicts the efficiency promise of the agent paradigm itself. At the same $20-per-month tier, a sidebar assistant and a full browser agent feel very different to users, yet in front of a blocking site their reachability flattens to the same level.
On the business side, the blockers do not want the door permanently shut. Cloudflare's pay-per-crawl, the Agentic Commerce Protocol from OpenAI and Stripe, and Google's AP2 all convert bot traffic from a cost line into a revenue line. What actually changes is bargaining structure: sites with exclusive content can charge a premium, while retailers with transparent product data may open up voluntarily, because agent-driven orders generate commission anyway. Liu Gong predicts that in the first half of 2027 the first catalog interfaces with dedicated agent pricing will appear, billed per query or as a share of completed transactions.
For the industry, the consequence is a layered open web: a licensed zone requiring signatures and payment, a public zone still freely crawlable, and a large middle of small and mid-sized sites holding the line with CAPTCHAs. Those sites gain visible licensing revenue in the short term, but risk handing distribution leverage to a few gateways and a few agent platforms over time.
- Cloudflare says its network carries roughly 20 percent of global web traffic, and since July 1, 2025 it has blocked unpaid AI crawlers by default, affecting millions of domains.
- Web Bot Auth, introduced in September 2025 with OpenAI and Anthropic among supporters, replaces User-Agent self-declaration with HTTP message signatures; Google's AP2 launched the same month with more than 60 payment and financial institutions as initial partners.
- OpenAI's Atlas browser shipped in October 2025 for macOS Plus ($20 per month) and Pro users, while Perplexity's Comet moved from a $200-per-month Max exclusive to free access in the same period, sharply expanding the agent traffic base.
- Amazon sued
For the industry, the short term (three months) will see blocking spread from training crawls to live task execution, forcing gateway vendors and model companies into signature interoperability and billing integration; agents without a signed identity will show measurably lower task completion. Over the medium term (one year), content licensing and agent commissions will become separately accounted revenue lines, raising the bargaining power of sites with exclusive data or inventory and squeezing pure aggregators.
For ordinary users, the near-term effect is a dumber agent: the same instruction completes on some sites and stalls at a CAPTCHA or login wall on others, requiring manual takeover
Questions worth exploring:
- The identity layer: whether Web Bot Auth becomes the TLS of the agent era, and who controls root trust and revocation.
- Pricing mechanics: pay-per-crawl, per-query billing and outcome-based commissions, and which produces a stable equilibrium between publishers and retailers.
- The small-site dilemma: accept a gateway's uniform rules in exchange for licensing revenue, or build its own blocking to keep distribution leverage.
Further reading: 2026 Frontier LLM Comparison: A Practical Guide — a durable, evergreen reference. Bookmark it.
This item comes from The Verge AI (published 2026-10-17 13:00:00). Wenfeng Radar provides a Chinese/English summary and commentary on overseas AI developments; the original content is owned by its author.
Daily aggregation of overseas AI developments with in-depth analysis. Bookmark us to never miss an important signal; back to homepage.