Cloudflare's September 15 Deadline: What 'Content Independence Day' Means for Scrapers and AI Agents

From Sept 15, 2026, newly onboarded ad-supported domains on Cloudflare get presets that block AI Agent crawlers on pages with ads and disallow AI training. What changed, what it means for Googlebot, and what to check.

Rahul Bisht

Founder, CrawlPilot

·
Jul 14, 2026
·
Verified Oct 25, 2026
·Industry & Ethics·
11 min read
·
Cloudflare's September 15 Deadline: What 'Content Independence Day' Means for Scrapers and AI Agents
Key takeaway
From September 15, 2026, ad-supported domains newly onboarding to Cloudflare get presets that block Agent crawlers on pages with ads, set Training to "Disallow AI Training" and allow Search. July's announcement had Training blocked outright, which would have caught Googlebot; what shipped is a robots.txt preference that keeps "Accountable" search crawlers allowed (Cloudflare).

Cloudflare just turned a slow-burning conflict of interest into a hard deadline. On July 1, 2026, it published what it's calling its second "Content Independence Day" — a new way to classify AI crawler traffic, and a switch that flips the default access rules for new domains on September 15, 2026 (Cloudflare).

If you run a site behind Cloudflare, build anything that fetches pages at scale, or just want to understand why your analytics might look different in October, the next two months are worth paying attention to.

Update (October 2026): On September 15, Cloudflare changed the multi-purpose crawler rule described below before it bit. It added a Disallow AI Training setting that publishes a no-training preference in robots.txt, designated Googlebot, Applebot and Bingbot as "Accountable" mixed-use crawlers that stay allowed for search under that setting, and migrated existing Training "Block" choices to Disallow AI Training. Choosing Block now stops those crawlers entirely, search included (Cloudflare). On September 30, Pay Per Use entered beta (Cloudflare).

  1. Jul 1, 2026
    Content Independence Day

    Cloudflare classifies AI crawlers as Search, Agent or Training, and every plan can set each category.

    Cloudflare
  2. Jul 2026
    Pay Per Use outlined

    Buyers pay for a specific use of content, such as an excerpt shown in an answer, rather than per fetch.

    Cloudflare
  3. Sep 15, 2026
    New defaults switch on

    Presets for new ad-supported domains block Agent crawlers on pages with ads and set Training to Disallow AI Training, which lets Googlebot, Applebot and Bingbot keep crawling for search.

    Cloudflare
  4. Sep 30, 2026
    Pay Per Use enters beta
    Cloudflare
From announcement to the new defaults, as of October 2026

What Actually Changed on July 1

Cloudflare introduced three categories for AI-related crawler traffic:

CategoryWhat it meansPreset for new ad-supported domains (as shipped Sept 15)
SearchCrawls to power search results and answer enginesAllowed
AgentCrawls triggered live by an AI agent acting on a user's requestBlocked on pages with ads
TrainingCrawls to build or fine-tune model training dataDisallow AI Training (robots.txt preference)

The classification applies per crawler, and Cloudflare is doing the categorization — not the site owner guessing from a user-agent string. Every customer, including the Free tier, can set each category independently — block everywhere, block only on pages with ads, or allow (Cloudflare changelog).

The headline change, as announced on July 1: starting September 15, Agent and Training crawlers get blocked by default on pages that display ads, for all new domains onboarding to Cloudflare. Search crawlers keep flowing (Cloudflare). What shipped on September 15 is softer for Training: the recommended presets for ad-supported sites block Agent on pages with ads but set Training to Disallow AI Training, a robots.txt preference (Cloudflare). Existing domains don't get the new defaults, but the multi-purpose crawler change below was set to reach any site that already blocks Training — unless the owner opted out in Security settings before September 15.


The Part That Actually Surprised People: Multi-Purpose Crawlers

Here's the mechanic buried in the announcement that's generating the most confusion.

Some crawlers aren't single-purpose. Googlebot, Applebot, and Bingbot all serve both Search and Training functions — Search crawls feed the search index; the same infrastructure (or a related one) also feeds AI training pipelines at the same company. Cloudflare's rule for these multi-purpose crawlers: they get evaluated under all of their behaviors, with the most restrictive applicable rule winning (Cloudflare).

That means if a site blocks Training crawlers, a multi-purpose crawler gets blocked too — even on the same page where Search crawlers are explicitly allowed.

In plain terms: a site owner trying to keep their content out of AI training sets, without realizing Googlebot is one of the crawlers caught by that same switch, could end up degrading their own Google Search visibility as a side effect. That behavior — one lever pulling on two outcomes — is the single most important thing to actually understand before September 15, not after.

As announced on July 1
  • A Training block stops Googlebot from crawling for search too
  • Mixed-use crawlers are judged under every purpose
  • The most restrictive rule wins
As shipped on September 15
  • Googlebot, Applebot and Bingbot keep crawling for search
  • Choosing a full Block still stops them, search included
  • Training blocks move to Disallow AI Training, a robots.txt preference

Verdict: Pick Disallow AI Training to stay in search; pick Block only if you mean to leave it.

A site that blocks AI training, and what happens to Googlebot

We Called This One in March

This blog wrote about Cloudflare's entry into crawling infrastructure back in March 2026, and flagged the structural position Cloudflare was building toward: sitting on both sides of the AI content market, protecting publishers from scraping and selling the infrastructure to do it.

The prediction then was specific: "If Cloudflare builds a publisher-facing access control layer — essentially a 'verified crawlers' list that gets through Bot Management — they control the key chokepoint in the AI web content market. Publishers flip a switch to allow specific AI crawlers (for a licensing fee), and Cloudflare enforces it at the infrastructure level."

Content Independence Day is that layer, shipped. Cloudflare's Pay Per Crawl feature, launched in 2025, charges AI companies for access. Cloudflare outlined Pay Per Use in July: buyers pay for a specific use of content — an excerpt shown in an answer, a review that shapes a recommendation — rather than for each fetch (Cloudflare). Combined with the new default-block rules, the mechanism is now live: block by default, then sell access back.

Whether that's good for the open web is a separate question from whether it's a smart business move. It's an extremely smart business move.


What This Means If You're a Publisher

Audit your Cloudflare AI Crawl Control and AI traffic settings before September 15, specifically:

  1. 01
    Decide what you actually want blocked. "Block all AI crawlers" is not the same decision as "block Training but keep Search" — and if you pick the former without realizing the multi-purpose crawler mechanic, you may be trading away search visibility you didn't mean to give up.
  2. 02
    Check whether your ad-supported pages are the ones you meant to protect. The new defaults only apply on pages that display ads — ad-free pages aren't covered by this specific default flip.
  3. 03
    Watch your referral traffic in October, not just your crawl logs. If Googlebot gets caught by a Training block you set, the symptom shows up as a drop in organic search traffic weeks later, not as an obvious crawl error today.

Cloudflare's settings sit at the edge, but your robots.txt is the part every AI crawler reads first. Paste yours to see which of them it lets in, and which line decides:

Paste any site's file from https://its-domain/robots.txt. It's checked in your browser; nothing is fetched or sent.

Training
6 of 6 blocked
Search
0 of 3 blocked
User fetch
0 of 4 blocked
  • BlockedGPTBotOpenAITraining
    Line 8: Disallow: / (its own group)

    Crawls content that may be used to train OpenAI's models. Docs

  • AllowedOAI-SearchBotOpenAISearch
    Line 12: Allow: / (the * group)

    Decides whether your pages can appear in ChatGPT search. Docs

  • AllowedChatGPT-UserOpenAIUser fetch
    Line 12: Allow: / (the * group)

    Fetches pages when a ChatGPT user asks; OpenAI says robots.txt rules may not apply. Docs

  • BlockedClaudeBotAnthropicTraining
    Line 8: Disallow: / (its own group)

    Collects content that may be used for model training. Docs

  • AllowedClaude-SearchBotAnthropicSearch
    Line 12: Allow: / (the * group)

    Indexes content for Claude's search results. Docs

  • AllowedClaude-UserAnthropicUser fetch
    Line 12: Allow: / (the * group)

    Fetches pages when a Claude user asks. Docs

  • BlockedGoogle-ExtendedGoogleTraining
    Line 8: Disallow: / (its own group)

    Controls use for Gemini training and grounding; doesn't affect Google Search. Docs

  • BlockedApplebot-ExtendedAppleTraining
    Line 8: Disallow: / (its own group)

    Controls use of Applebot's crawl for Apple's generative AI training; it doesn't crawl. Docs

  • AllowedPerplexityBotPerplexitySearch
    Line 12: Allow: / (the * group)

    Surfaces and links sites in Perplexity's search results. Docs

  • AllowedPerplexity-UserPerplexityUser fetch
    Line 12: Allow: / (the * group)

    Fetches pages a user asks for; Perplexity says it generally ignores robots.txt. Docs

  • BlockedMeta-ExternalAgentMetaTraining
    Line 8: Disallow: / (its own group)

    Crawls for uses such as training AI models or indexing content. Docs

  • AllowedMeta-ExternalFetcherMetaUser fetch
    Line 12: Allow: / (the * group)

    Fetches links a user asks for; Meta says it may bypass robots.txt. Docs

  • BlockedCCBotCommon CrawlTraining
    Line 8: Disallow: / (its own group)

    Builds Common Crawl's open web corpus, widely used to train models. Docs

Which AI crawlers your robots.txt allows, by the rules in RFC 9309. The sample blocks training and allows search.

What This Means If You Scrape for a Living

If your crawling looks like an AI lab's — broad, unauthenticated, server-side, high-volume — expect more Cloudflare-protected sites to default to blocking you starting mid-September, on ad-supported pages specifically. Cloudflare says more than 20% of the web sits behind its network, so this isn't a rounding error.

If your crawling looks nothing like that — a single person, in their own authenticated session, pulling data from pages they can already see in their own browser — you were never really the target of this policy, and the traffic pattern reflects that. Cloudflare's classification system exists to sort automated, server-side, at-scale crawler traffic into buckets. It has no mechanism (and no reason) to flag a single browser session behaving like a single browser session.

What This Means If You Use CrawlPilot

This is worth being precise about rather than turning into a pitch: CrawlPilot is a browser extension. It runs the extraction inside your browser, using your session, your cookies, your IP — it's the same traffic pattern as you clicking around the page yourself, because that's mechanically what's happening. There's no separate server-side crawler fleet hitting Cloudflare's edge on your behalf.

That doesn't make it invisible to every anti-bot system on earth — nothing is — but it does mean Content Independence Day's Search/Agent/Training classification, which is built to categorize automated crawler infrastructure, isn't really evaluating the same thing a local, human-driven browser session is doing. It's a genuinely different traffic shape.


The Honest Caveats

  • This is Cloudflare-only. Roughly a fifth of the web sits behind it, which is significant, but four-fifths doesn't. Sites on other CDNs or none at all aren't affected by this specific change.
  • Defaults are overridable, in both directions. A site can choose to allow Agent and Training crawlers if it wants the AI visibility; a site can also block Search crawlers if it wants to disappear from AI answer engines entirely. September 15 sets the default, not a mandate.
  • This doesn't resolve the underlying legal ambiguity around what crawling is lawful in the first place. A federal court ruled in December 2025 that robots.txt isn't a DMCA "technological protection measure" at all (opinion) — meaning ignoring it isn't automatically the legal violation a lot of people assume it is. Cloudflare's rules are a private company's access-control defaults enforced at the infrastructure layer, which is a separate question from what the law actually requires. See our legal guide to web scraping for the broader framework, though it predates this specific ruling.
  • The classification itself is Cloudflare's call. There's no independent audit of which crawlers land in which bucket — you're trusting Cloudflare's categorization the same way you're trusting its bot-detection scores today.

What to Actually Check Before September 15

If you do one thing after reading this: log into your Cloudflare dashboard, open your AI traffic settings, and look at what you currently allow or block for Training versus Search on your domain. Decide on purpose whether you want the September 15 defaults, rather than finding out in October from a traffic graph.

The bigger story here isn't really about scrapers at all. It's the first real, live instance of a infrastructure company sitting at the chokepoint of the AI content economy and actually pulling the lever — not speculating about it. Whoever else builds the next version of this (and someone will) is going to look a lot like this one.

Sources