The default reaches new domains, new sites from existing customers and free-tier accounts. Cloudflare fronts about a fifth of the web, TechCrunch reported when the change was announced in July.
Cloudflare’s reasoning is that an ad is proof a page was built for a person to see. An agent that fetches the page on a shopper’s behalf reads the price and the reviews and leaves the ad unseen. The company said fewer than 1% of its sites block search bots, while 17% restrict AI training. That gap is why it split its single “Block AI Bots” switch into three controls.
Crawler traffic explains the timing. Cloudflare’s July bot report found 52% of crawler requests were for AI training as of June, up from 22% in spring 2025. Mixed-use crawlers that blend search, agent use and training made up more than 36% of activity. Pure search crawling is a small and shrinking share.
Apple, Google and Microsoft earned an “Accountable” label by agreeing to honor no-training preferences without penalizing search rankings, according to the announcement.
Retailers Draw Their Own Lines
The default applies only to sites that earn money from ads. A retailer’s product page is a different case. Cloudflare said a merchant may prefer fewer visitors who are more likely to buy, citing figures that AI referrals convert at 3 to 5 times the rate of traditional search.
We’d love to be your preferred source for news.
Please add us to your preferred sources list so our news, data and interviews show up in your feed. Thanks!
The block falls hardest on the review sites, comparison guides and publisher content that shopping agents read before they reach a checkout.
Retailers have tested their own defenses. Amazon sued Perplexity in November to stop its Comet browser agent from shopping inside customer accounts, accusing the startup of hiding when a bot was acting for a real person, Bloomberg reported. The case is still working through the courts. Walmart and Target have taken the opposite route, testing ways to work with AI shopping platforms while keeping their own role in the transaction.
Cloudflare sells to both sides of that fight. In August it launched Kitesurf, a cloud-hosted browser built for AI agents rather than people, TechCrunch reported.
Discovery Moves Faster Than Checkout
Nearly 132 million U.S. adults have bought a retail product with AI’s help, and 59% of AI-assisted purchases still end at Amazon, PYMNTS Intelligence found. Consumers use AI to narrow their options, then buy where they already shop. Agents that can’t read the pages where that narrowing happens lose their reason to exist.
Merchants can barely see the traffic. Only 23% can identify both AI-driven traffic and the purchases it produces, PYMNTS Intelligence reported. Another 21% recognize agent traffic but can’t connect it to a sale. A separate 41% of merchants draw a line at agents redirecting customers to competitors.
Cloudflare has set a target of driving mixed-use crawler traffic on its network to zero within a year. It also aims to let site owners control how much of their content appears in AI summaries by mid-2027, set once on Cloudflare rather than with each operator separately.
For all PYMNTS AI coverage, subscribe to the daily AI Newsletter.
