The EDPB's Draft Web Scraping Guidelines: What They Ask of AI Scrapers, and How to Comment by October 30

Picture a content-sharing platform that blocks scrapers with robots.txt and a CAPTCHA, and says on its site that its users' data may not be used to build AI models. In the European Data Protection Board's new draft guidelines, that platform is Example 5, and the Board's verdict takes one line: its users "cannot reasonably expect third parties to scrape that data for that purpose" (Guidelines 03/2026, ¶64).

A US court compared robots.txt last December to a keep-off-the-grass sign that doesn't control access (opinion; our write-up). Nothing in the EDPB draft depends on the file being a lock. Its point is about evidence: under the GDPR, the file says something about what the people on that site could expect, so ignoring it can count against the scraper.

What follows is a reading of the draft itself, with paragraph numbers so you can check each point. It isn't legal advice.

Guidance in Draft, Not a New Law

The EDPB's own description of its work separates the binding decisions it addresses to national authorities in cross-border disputes from "general guidance (including guidelines, recommendations and best practices) to clarify and promote a common understanding" of the law (EDPB). Guidelines 03/2026 are the second kind. They create no new obligations; they say how the Board reads the GDPR's existing ones when the processing is scraping for generative AI. They're also version 1.0, open for comment, so the wording below may change.

The scope is narrow, and it's worth reading before anything else (¶1–6):

• Covered: an organisation that scrapes to train or fine-tune generative AI, itself or through a contractor, and one that reuses a dataset someone else scraped.
• Not covered: data brokers who scrape and hold datasets "whose intention is not to train AI models themselves" (¶4), an organisation's own data (¶5) and public bodies (¶6).
• Still the law anyway: ¶5 adds that leaving a situation out "does not mean that the excluded situations are not subject to data protection legislations."

The GDPR applies when scraping processes personal data, whether it identifies someone directly (names, pictures, contact details) or indirectly (¶12). A "mixed" dataset is still covered for its personal part (¶13).

Six Places the Text Touches a Scraping Pipeline

The draft describes scraping as four steps: defining collection criteria, extraction, cleaning, and structuring and storing (¶11). Its obligations land on each of them, then carry on through training and into what you owe the people in the data. Step through it:

Step 1 does most of the work. The draft wants the assessment made "before the scraping begins, so as not to collect personal data that is not necessary" (¶36). It also says data minimisation "does not prohibit training an AI model or AI system with large volumes of data." The rule is about which data you take, not how much.

On transparency, the escape hatch in Article 14(5)(b), where notifying each person would take disproportionate effort, is available but "should not be routinely relied upon" outside archiving, research and statistics (¶27). The draft's two examples set the line. A 20-year scrape of forums with no direct identifiers qualifies for a public notice instead (Example 2). A recent scrape of 5,000 named people in a closed group with email addresses does not, and those people should be told one by one (Example 3). Listing domains and URLs in a searchable format, with collection dates, is called good practice (¶31).

Consent Doesn't Work, So Legitimate Interest Has to Be Earned

The draft rules out the basis many people assume first. Consent "would most probably not be an applicable legal basis", because a scraper has no direct relationship with the people in the data and can't ask each of them beforehand (¶44). Publishing something on an open web page isn't consent to having it scraped, and "the absence or non-applicability of a robots.txt file on a web site does not amount to consent" (¶45).

The three conditions come from the Board's 2024 guidelines on legitimate interest and are cumulative (¶47): a legitimate interest, necessity, and a balancing test that the person's rights don't override. The balancing test splits into several factors in the draft (¶53–64), and the safeguards in ¶66 are what can tip it back. Turn any of them off and see where the assessment stops:

The draft also gives the passing case. Example 7 is a text model trained only on freely accessible data that people manifestly made public, excluding copyrighted content unless it's public domain or its rights holders haven't opted out of text and data mining, with anti-memorisation safeguards, rights handling and sources named in a public privacy policy. For that one, "the balancing test may generally be considered as met." It covers most of the switches above: data people made public themselves, rights handling, anti-memorisation safeguards and published sources.

robots.txt, CAPTCHAs and Login Walls as Privacy Evidence

This is the section scrapers will quote most, so it's worth being exact about what the text says. Signals show up in two places.

• ¶37, data minimisation: before collecting, exclude "websites which clearly oppose the scraping of their content, through the use of technical measures", naming authentication to view content, robots.txt or ai.txt files, and CAPTCHAs.
• ¶64, reasonable expectations: among the things a controller "should consider" are "the restrictions imposed by the scraped website", with the same measures as examples. Then: "If data subjects are aware that a website implements these measures and a website is nevertheless scraped, it is less likely that they can expect the processing of their personal data by a scraping entity."

Two details are easy to miss. The ¶64 sentence is conditional on people being aware of the measures, and expectations are only one factor in the balancing test (¶53, ¶62). Paragraph 63 adds that publishing a privacy notice doesn't, by itself, make processing expected. Reed Smith's commentary reads this more broadly: "Ignoring robots.txt or circumventing access controls now carries direct GDPR consequences for the balancing test" (Reed Smith). That's a fair summary of where the text points, though the text itself stays at "less likely".

Combine the signals yourself. Each switch shows the paragraph behind it:

Logged-in scraping gets its own line. Among the mitigating measures, ¶66(c) lists excluding personal data "only accessible when logged into the site", and data on freely accessible sites where people can't be considered aware of it: for instance where a social network returns more to the scraper's user agent (timestamps, access statistics) than to a normal user.

Since robots.txt is now part of the evidence, it helps to know what yours, or your target's, actually says. Here is a sample for a community forum that wants search engines but no training crawlers; edit it or try other paths:

Filtering Before You Fetch, and the Special-Category Problem

Paragraph 37 is the closest thing in the draft to a build spec. Before collection: consider synthetic data, define precise criteria, map and inventory the data, filter out unneeded categories such as bank transactions or geolocation, skip site categories that structurally hold sensitive data (the example is sites "used mainly by minors"), and skip sites that oppose scraping. During and after (¶38): syntax-based filters such as regular expressions for things like social security and phone numbers, synthetic replacement where feasible, and anonymisation or pseudonymisation. Accuracy adds three more (¶42): prefer reliable, maintained sources over "secondary or outdated aggregators", timestamp what you scrape, and validate before training.

Special categories (health, ethnic origin, political opinions, sexual orientation and the rest of Article 9) are prohibited without a derogation in Article 9(2) on top of an Article 6 basis (¶67). The draft is candid that a scraper usually can't know whether it collected any until after the fact (¶68). Its answer borrows from the CJEU's GC & Others ruling on search engines, C-136/17, where the prohibition applied "within the framework of [the operator's] responsibilities, powers and capabilities" (¶70). The Board says that reasoning "can be relevant" for incidental and residual collection when scraping to train AI, and "should not be seen as a general exemption" (¶72). It holds only if you take measures at four stages:

The controller also has to be able to show that the conditions apply and that its measures work (¶75), and keep checking them against the state of the art (¶76–77).

Who Is the Controller When You Scrape for Someone Else

If you run scraping as a service, ¶17 is the line to remember: "The organisation performing the scraping is not necessarily the controller under the GDPR." The draft gives three arrangements.

The roles are decided "on a case-by-case basis" from the facts, not from what the contract calls you (¶16). If you're the processor, the instructions you receive are where the minimisation work above should be written down. Where a developer reuses a dataset scraped by someone else, the draft recommends its notice say how to contact the original scraper, ideally with a direct link (¶32).

If You Scrape Prices, Not Training Data

Most scraping isn't for training models, and this draft says nothing directly about it. The scope covers training and fine-tuning generative AI (¶1–3). Even data brokers that scrape without training are left out (¶4).

Two things follow from the text. Scraping that touches no personal data, like product prices, stock levels or catalogue fields, isn't GDPR processing at all, because the regulation applies when scraping "includes processing operations... of personal data" (¶12). And scraping that does touch personal data, such as seller names, reviewer profiles or contact details, stays under the GDPR whatever its purpose, since excluded situations remain "subject to data protection legislations" (¶5).

What the draft doesn't say is how much of its reasoning carries over. My read, as opinion: the minimisation steps in ¶37–38 and the expectation factors in ¶64 apply general GDPR principles and the Board's 2024 legitimate-interest guidelines rather than anything specific to AI, so a supervisory authority looking at a non-AI scraper could reason along similar lines. The parts tied to models, like memorisation, output filters and unlearning, have no counterpart for a price tracker. For the wider legal picture beyond the GDPR (CFAA, contracts, copyright), see our legal guide to web scraping.

Commenting Before 30 October

The feedback period runs from 8 July to 30 October 2026, 23:59 CET, through the form on the consultation page. Before you send anything, read what the page says about publication:

1. 01
   Submitting means you "acknowledge that your comments will be published on the EDPB website."
2. 02
   EDPB staff screen replies only to block unauthorised submissions such as spam, then publish them; attached files "are not altered in any way".
3. 03
   Your contribution may be subject to an access-to-documents request under Regulation 1049/2001.

So write it as a public document: no client names, no internal URLs, nothing you wouldn't put on your own site.

Sources

• Guidelines 03/2026 on web scraping in the context of generative AI, version 1.0 (PDF), European Data Protection Board
• Public consultation: Guidelines 03/2026, European Data Protection Board
• Guidelines at a Glance: web scraping for generative AI, how and when to use it (PDF), European Data Protection Board
• Tasks and duties, European Data Protection Board
• Guidelines 1/2024 on processing of personal data based on Article 6(1)(f) GDPR, European Data Protection Board
• Opinion 28/2024 on certain data protection aspects related to the processing of personal data in the context of AI models, European Data Protection Board
• GDPR Article 99, Entry into force and application (gdpr-info.eu)
• EDPB web scraping guidelines for AI: Making the impossible possible?, Reed Smith (commentary)