Cloudflare's September 15 Deadline: What 'Content Independence Day' Means for Scrapers and AI Agents
From Sept 15, 2026, newly onboarded ad-supported domains on Cloudflare get presets that block AI Agent crawlers on pages with ads and disallow AI training. What changed, what it means for Googlebot, and what to check.
Cloudflare just turned a slow-burning conflict of interest into a hard deadline. On July 1, 2026, it published what it's calling its second "Content Independence Day" — a new way to classify AI crawler traffic, and a switch that flips the default access rules for new domains on September 15, 2026 (Cloudflare).
If you run a site behind Cloudflare, build anything that fetches pages at scale, or just want to understand why your analytics might look different in October, the next two months are worth paying attention to.
Update (October 2026): On September 15, Cloudflare changed the multi-purpose crawler rule described below before it bit. It added a Disallow AI Training setting that publishes a no-training preference in
robots.txt, designated Googlebot, Applebot and Bingbot as "Accountable" mixed-use crawlers that stay allowed for search under that setting, and migrated existing Training "Block" choices to Disallow AI Training. Choosing Block now stops those crawlers entirely, search included (Cloudflare). On September 30, Pay Per Use entered beta (Cloudflare).
- Jul 1, 2026Content Independence Day
Cloudflare classifies AI crawlers as Search, Agent or Training, and every plan can set each category.
Cloudflare - Jul 2026Pay Per Use outlined
Buyers pay for a specific use of content, such as an excerpt shown in an answer, rather than per fetch.
Cloudflare - Sep 15, 2026New defaults switch on
Presets for new ad-supported domains block Agent crawlers on pages with ads and set Training to Disallow AI Training, which lets Googlebot, Applebot and Bingbot keep crawling for search.
Cloudflare - Sep 30, 2026Pay Per Use enters betaCloudflare
What Actually Changed on July 1
Cloudflare introduced three categories for AI-related crawler traffic:
| Category | What it means | Preset for new ad-supported domains (as shipped Sept 15) |
|---|---|---|
| Search | Crawls to power search results and answer engines | Allowed |
| Agent | Crawls triggered live by an AI agent acting on a user's request | Blocked on pages with ads |
| Training | Crawls to build or fine-tune model training data | Disallow AI Training (robots.txt preference) |
The classification applies per crawler, and Cloudflare is doing the categorization — not the site owner guessing from a user-agent string. Every customer, including the Free tier, can set each category independently — block everywhere, block only on pages with ads, or allow (Cloudflare changelog).
The headline change, as announced on July 1: starting September 15, Agent and Training crawlers get blocked by default on pages that display ads, for all new domains onboarding to Cloudflare. Search crawlers keep flowing (Cloudflare). What shipped on September 15 is softer for Training: the recommended presets for ad-supported sites block Agent on pages with ads but set Training to Disallow AI Training, a robots.txt preference (Cloudflare). Existing domains don't get the new defaults, but the multi-purpose crawler change below was set to reach any site that already blocks Training — unless the owner opted out in Security settings before September 15.
The Part That Actually Surprised People: Multi-Purpose Crawlers
Here's the mechanic buried in the announcement that's generating the most confusion.
Some crawlers aren't single-purpose. Googlebot, Applebot, and Bingbot all serve both Search and Training functions — Search crawls feed the search index; the same infrastructure (or a related one) also feeds AI training pipelines at the same company. Cloudflare's rule for these multi-purpose crawlers: they get evaluated under all of their behaviors, with the most restrictive applicable rule winning (Cloudflare).
That means if a site blocks Training crawlers, a multi-purpose crawler gets blocked too — even on the same page where Search crawlers are explicitly allowed.
In plain terms: a site owner trying to keep their content out of AI training sets, without realizing Googlebot is one of the crawlers caught by that same switch, could end up degrading their own Google Search visibility as a side effect. That behavior — one lever pulling on two outcomes — is the single most important thing to actually understand before September 15, not after.
- A Training block stops Googlebot from crawling for search too
- Mixed-use crawlers are judged under every purpose
- The most restrictive rule wins
- Googlebot, Applebot and Bingbot keep crawling for search
- Choosing a full Block still stops them, search included
- Training blocks move to Disallow AI Training, a robots.txt preference
Verdict: Pick Disallow AI Training to stay in search; pick Block only if you mean to leave it.
We Called This One in March
This blog wrote about Cloudflare's entry into crawling infrastructure back in March 2026, and flagged the structural position Cloudflare was building toward: sitting on both sides of the AI content market, protecting publishers from scraping and selling the infrastructure to do it.
The prediction then was specific: "If Cloudflare builds a publisher-facing access control layer — essentially a 'verified crawlers' list that gets through Bot Management — they control the key chokepoint in the AI web content market. Publishers flip a switch to allow specific AI crawlers (for a licensing fee), and Cloudflare enforces it at the infrastructure level."
Content Independence Day is that layer, shipped. Cloudflare's Pay Per Crawl feature, launched in 2025, charges AI companies for access. Cloudflare outlined Pay Per Use in July: buyers pay for a specific use of content — an excerpt shown in an answer, a review that shapes a recommendation — rather than for each fetch (Cloudflare). Combined with the new default-block rules, the mechanism is now live: block by default, then sell access back.
Whether that's good for the open web is a separate question from whether it's a smart business move. It's an extremely smart business move.
What This Means If You're a Publisher
Audit your Cloudflare AI Crawl Control and AI traffic settings before September 15, specifically:
- 01Decide what you actually want blocked. "Block all AI crawlers" is not the same decision as "block Training but keep Search" — and if you pick the former without realizing the multi-purpose crawler mechanic, you may be trading away search visibility you didn't mean to give up.
- 02Check whether your ad-supported pages are the ones you meant to protect. The new defaults only apply on pages that display ads — ad-free pages aren't covered by this specific default flip.
- 03Watch your referral traffic in October, not just your crawl logs. If Googlebot gets caught by a Training block you set, the symptom shows up as a drop in organic search traffic weeks later, not as an obvious crawl error today.
Cloudflare's settings sit at the edge, but your robots.txt is the part every AI crawler reads first. Paste yours to see which of them it lets in, and which line decides:
Paste any site's file from https://its-domain/robots.txt. It's checked in your browser; nothing is fetched or sent.
- BlockedGPTBotOpenAITrainingLine 8: Disallow: / (its own group)
Crawls content that may be used to train OpenAI's models. Docs
- AllowedOAI-SearchBotOpenAISearchLine 12: Allow: / (the * group)
Decides whether your pages can appear in ChatGPT search. Docs
- AllowedChatGPT-UserOpenAIUser fetchLine 12: Allow: / (the * group)
Fetches pages when a ChatGPT user asks; OpenAI says robots.txt rules may not apply. Docs
- BlockedClaudeBotAnthropicTrainingLine 8: Disallow: / (its own group)
Collects content that may be used for model training. Docs
- AllowedClaude-SearchBotAnthropicSearchLine 12: Allow: / (the * group)
Indexes content for Claude's search results. Docs
- AllowedClaude-UserAnthropicUser fetchLine 12: Allow: / (the * group)
Fetches pages when a Claude user asks. Docs
- BlockedGoogle-ExtendedGoogleTrainingLine 8: Disallow: / (its own group)
Controls use for Gemini training and grounding; doesn't affect Google Search. Docs
- BlockedApplebot-ExtendedAppleTrainingLine 8: Disallow: / (its own group)
Controls use of Applebot's crawl for Apple's generative AI training; it doesn't crawl. Docs
- AllowedPerplexityBotPerplexitySearchLine 12: Allow: / (the * group)
Surfaces and links sites in Perplexity's search results. Docs
- AllowedPerplexity-UserPerplexityUser fetchLine 12: Allow: / (the * group)
Fetches pages a user asks for; Perplexity says it generally ignores robots.txt. Docs
- BlockedMeta-ExternalAgentMetaTrainingLine 8: Disallow: / (its own group)
Crawls for uses such as training AI models or indexing content. Docs
- AllowedMeta-ExternalFetcherMetaUser fetchLine 12: Allow: / (the * group)
Fetches links a user asks for; Meta says it may bypass robots.txt. Docs
- BlockedCCBotCommon CrawlTrainingLine 8: Disallow: / (its own group)
Builds Common Crawl's open web corpus, widely used to train models. Docs
What This Means If You Scrape for a Living
If your crawling looks like an AI lab's — broad, unauthenticated, server-side, high-volume — expect more Cloudflare-protected sites to default to blocking you starting mid-September, on ad-supported pages specifically. Cloudflare says more than 20% of the web sits behind its network, so this isn't a rounding error.
If your crawling looks nothing like that — a single person, in their own authenticated session, pulling data from pages they can already see in their own browser — you were never really the target of this policy, and the traffic pattern reflects that. Cloudflare's classification system exists to sort automated, server-side, at-scale crawler traffic into buckets. It has no mechanism (and no reason) to flag a single browser session behaving like a single browser session.
What This Means If You Use CrawlPilot
This is worth being precise about rather than turning into a pitch: CrawlPilot is a browser extension. It runs the extraction inside your browser, using your session, your cookies, your IP — it's the same traffic pattern as you clicking around the page yourself, because that's mechanically what's happening. There's no separate server-side crawler fleet hitting Cloudflare's edge on your behalf.
That doesn't make it invisible to every anti-bot system on earth — nothing is — but it does mean Content Independence Day's Search/Agent/Training classification, which is built to categorize automated crawler infrastructure, isn't really evaluating the same thing a local, human-driven browser session is doing. It's a genuinely different traffic shape.
The Honest Caveats
- This is Cloudflare-only. Roughly a fifth of the web sits behind it, which is significant, but four-fifths doesn't. Sites on other CDNs or none at all aren't affected by this specific change.
- Defaults are overridable, in both directions. A site can choose to allow Agent and Training crawlers if it wants the AI visibility; a site can also block Search crawlers if it wants to disappear from AI answer engines entirely. September 15 sets the default, not a mandate.
- This doesn't resolve the underlying legal ambiguity around what crawling is lawful in the first place. A federal court ruled in December 2025 that
robots.txtisn't a DMCA "technological protection measure" at all (opinion) — meaning ignoring it isn't automatically the legal violation a lot of people assume it is. Cloudflare's rules are a private company's access-control defaults enforced at the infrastructure layer, which is a separate question from what the law actually requires. See our legal guide to web scraping for the broader framework, though it predates this specific ruling. - The classification itself is Cloudflare's call. There's no independent audit of which crawlers land in which bucket — you're trusting Cloudflare's categorization the same way you're trusting its bot-detection scores today.
What to Actually Check Before September 15
If you do one thing after reading this: log into your Cloudflare dashboard, open your AI traffic settings, and look at what you currently allow or block for Training versus Search on your domain. Decide on purpose whether you want the September 15 defaults, rather than finding out in October from a traffic graph.
The bigger story here isn't really about scrapers at all. It's the first real, live instance of a infrastructure company sitting at the chokepoint of the AI content economy and actually pulling the lever — not speculating about it. Whoever else builds the next version of this (and someone will) is going to look a lot like this one.
