Robots.txt Doesn't Mean What You Think: What the Ziff Davis v. OpenAI Ruling Actually Says
A federal court ruled robots.txt isn't a DMCA 'technological protection measure.' Here's what that actually changes — and what it very much doesn't.
Most developers have absorbed one piece of scraping folklore as settled fact: ignore robots.txt and you're breaking the law. A federal court just said that's not really how it works — at least not the way most people assume.
In one of the copyright suits against OpenAI, Judge Sidney H. Stein of the Southern District of New York ruled on a motion to dismiss claims brought by digital publisher Ziff Davis. Buried in that ruling is a holding worth understanding precisely, because it's already getting oversimplified into "robots.txt is legally meaningless" — which isn't what the court said either.
What the Case Actually Argued
Ziff Davis, like several publishers now litigating against AI labs, argued that OpenAI's scraping violated the DMCA's anti-circumvention provision — Section 1201, the part of copyright law that makes it illegal to bypass a "technological protection measure" that controls access to a copyrighted work. This is the same legal machinery that makes it illegal to crack DRM on a DVD or bypass a paywall's authentication.
The theory: OpenAI's crawlers accessed Ziff Davis content despite robots.txt directives telling them not to. If robots.txt counts as a technological protection measure, ignoring it could be circumvention — a DMCA violation independent of whether the underlying use of the content was infringing.
The court partially granted OpenAI's motion to dismiss on this theory. Its reasoning: a robots.txt file is not a technological measure that "effectively controls access" under Section 1201. It's a plain-text file that makes a request. Nothing about it technologically prevents a crawler from fetching the page anyway — compliance is entirely voluntary. Courts have historically required an actual technological barrier (encryption, authentication, a paywall) for Section 1201 to apply, not a signal a visitor can simply disregard.
Why This Distinction Actually Matters
The legal logic here isn't arbitrary — it maps onto a real, meaningful line between two very different kinds of "no."
A robots.txt file is closer to a "no soliciting" sign than a locked door. It communicates intent clearly, and reputable crawlers (search engines, most legitimate bots) respect it by convention — but nothing stops a crawler that chooses not to. A login wall, CAPTCHA gate, or IP-based blocking system is a locked door: it takes active, technical effort to get past, which is exactly the kind of "circumvention" Section 1201 was written to punish.
Treating the sign and the lock as legally equivalent was always a stretch. This ruling is a court saying so explicitly, in a live, high-profile case.
What This Doesn't Mean
This is the part worth being precise about, because "robots.txt doesn't matter legally" is the wrong takeaway, and it's already circulating.
The underlying copyright infringement claims are unaffected. The DMCA anti-circumvention theory was one specific legal argument among several in the suit. Whether OpenAI's actual use of the scraped content infringes copyright — training a model on it, reproducing it, competing with it — is a completely separate question the ruling doesn't resolve.
Breach of contract and terms-of-service claims are untouched. If a site's terms of service explicitly prohibit automated access and a crawler ignores both the terms and robots.txt, a breach-of-contract theory doesn't depend on robots.txt being a "technological protection measure" at all.
CFAA "unauthorized access" arguments still exist, separately. Courts weighing whether access was "unauthorized" under the Computer Fraud and Abuse Act have historically looked at signals including robots.txt, even if it isn't dispositive on its own. Ignoring a clearly stated crawling preference can still be evidence of intent in a case built on other legal theories — this ruling narrows one specific weapon, not the whole arsenal.
It's one federal district court, on a motion to dismiss. Motions to dismiss test whether a claim is legally viable at all, not whether it's true — and this is one judge's reasoning in one case, not binding precedent nationally. Expect this exact question to keep getting litigated, possibly with different outcomes in different circuits, for years.
The Interesting Asymmetry This Creates
Here's where this connects to something else moving right now. Cloudflare's new AI crawler defaults, rolling out September 15, 2026, block Agent and Training crawlers at the infrastructure layer — not by asking nicely in a text file, but by actually intercepting and rejecting the request before it reaches the origin server.
Put the two developments side by side and an asymmetry appears: the voluntary, honor-system layer (robots.txt) just had its legal teeth narrowed, while the enforced, technological layer (Bot Management-style blocking) is exactly the kind of "effectively controls access" barrier that would plausibly qualify under Section 1201 if someone built a circumvention tool specifically to defeat it. The practical message for site owners who actually want legal leverage, not just a documented preference, is unambiguous: a polite request in a text file was never going to carry the legal weight people assumed. An actual technological barrier might.
That's not a coincidence of timing — it's the market and the case law arriving at the same conclusion from different directions at once: intent alone isn't enforcement, and enforcement is what actually has legal consequences attached to it.
What to Actually Do With This
If you run a site: keep your robots.txt accurate and current — it's still useful documentation of intent, still respected by well-behaved crawlers, and still potentially relevant evidence in other legal theories. Just don't mistake it for a technological control. If you need actual enforcement against traffic that ignores your preferences, that has to happen at the infrastructure layer.
If you build crawlers or scrapers: this ruling narrows one specific liability theory in one specific type of case (an AI lab being sued for how it used scraped content at scale for model training). It is not a blanket legal green light for scraping generally — copyright, contract, and CFAA exposure haven't gone anywhere, and the specific facts of the Ziff Davis case (a well-funded publisher suing a well-funded AI lab over training data) look very different from, say, a small business pulling public product listings for competitive research. See our legal guide to web scraping for the fuller framework — this ruling is a data point inside that framework, not a replacement for it.
The headline "robots.txt doesn't legally protect you the way you thought" is accurate. The headline "scraping is now legal" is not. Those are different claims, and the gap between them is exactly where the real risk still lives.
