Privacy and Ethics in Web Scraping: A Developer's Actual Legal Guide

Most "ethics in web scraping" articles say the same things: respect robots.txt, don't overload servers, check the terms of service. This is the equivalent of telling a driver to "be careful" — technically correct, almost entirely useless for making actual decisions.

This post covers the legal terrain that matters: the cases that set precedent, the specific GDPR provisions that apply, and the enforcement reality (which looks very different from the legal theory). It is not legal advice — consult an attorney for your specific situation. It is an attempt to help developers understand the landscape accurately.

The Four Legal Frameworks That Actually Matter

Web scraping sits at the intersection of four distinct legal frameworks. Understanding which applies to your situation determines your actual risk profile.

1. The Computer Fraud and Abuse Act (CFAA) — USA

The CFAA criminalizes "unauthorized access" to computer systems. For years it was interpreted broadly — companies used it to argue that violating a website's terms of service constituted "unauthorized access."

The key case: Van Buren v. United States (2021)

The Supreme Court narrowed the CFAA significantly. In Van Buren, the Court held that the "exceeds authorized access" provision only applies to circumventing access restrictions on systems you're authorized to use — not to using authorized access for unauthorized purposes.

What this means for scrapers: accessing a publicly accessible website — one with no login wall, no technical barrier — is almost certainly not a CFAA violation, even if the site's ToS prohibits scraping. You are not "circumventing" anything by accessing data the server freely serves to anyone who asks.

This does not eliminate all risk. The CFAA still applies if you:

• Bypass a login wall to access non-public data
• Circumvent technical access controls (rate limiting, IP blocks, CAPTCHAs)
• Access systems with explicit restricted-access notices

The 9th Circuit: hiQ Labs v. LinkedIn (2022)

hiQ scraped LinkedIn's public profile pages. LinkedIn sent a cease and desist citing the CFAA. The 9th Circuit (on remand from the Supreme Court) upheld a preliminary injunction in hiQ's favor, finding that accessing publicly available data did not constitute a CFAA violation.

The case was eventually settled without a Supreme Court ruling, which means the 9th Circuit's reasoning stands as significant precedent but is not Supreme Court law. Other circuits may reach different conclusions.

Practical implication: In the US, scraping publicly accessible data from websites with no login walls is unlikely to constitute a CFAA violation under Van Buren + hiQ. Scraping behind login walls, bypassing CAPTCHAs, or defeating rate limiting occupies significantly riskier legal territory.

2. Terms of Service Violations

Almost every major website's ToS prohibits automated access. ToS violations are not crimes — they're a basis for civil liability, usually breach of contract.

What you're actually risking with ToS violations:

• Account termination
• IP banning
• Cease and desist letters
• Civil lawsuit for breach of contract

The actual damages argument is difficult to sustain without proof of specific harm. LinkedIn's lawsuit against hiQ mentioned "enormous expense" to combat scraping, but quantifying that as concrete damages attributable to a single scraper is legally complex.

More practically: ToS lawsuits against individual developers for scraping publicly available data are rare. They're expensive to prosecute, hard to win on damages, and generate negative publicity. Large-scale commercial operations are a different story — particularly if there's a competing product built on scraped data.

The risk profile for ToS violations scales with:

• Volume — scraping 100 pages vs. 100 million pages
• Commercial impact — personal research vs. a competing product
• Visibility — operating quietly vs. announcing "we use their data"
• Target aggression — LinkedIn actively litigates; most sites don't

3. GDPR — European Union

GDPR applies when you process personal data of EU residents. This is where many developers make a critical error: they assume GDPR only applies if they're a European company. It doesn't.

GDPR Article 3 explicitly states it applies to any organization, regardless of location, that processes personal data of EU residents in the context of offering goods or services to them, or monitoring their behavior.

What counts as personal data in scraped content?

• Names, email addresses, phone numbers: obviously personal data
• Profile photos, usernames linked to identifiable persons: personal data
• Professional history, locations: personal data when linked to an identified person
• Publicly posted social media content: personal data (even if publicly accessible)
• Job listings with named hiring managers: personal data for the named individual

The key question: what's your lawful basis?

GDPR requires a lawful basis (Article 6) for processing personal data. The most relevant bases for scraped data:

Legitimate interests (Article 6(1)(f)): You may process personal data if you have a legitimate interest that isn't overridden by the individual's interests or rights. This requires a three-part test: (1) identify a legitimate interest, (2) show the processing is necessary, (3) show the interest isn't overridden by the individual's interests.

For scraped data: market research, price monitoring of business listings, academic research can potentially qualify — but only if the data subjects' interests don't override yours, and you implement appropriate safeguards (data minimization, access controls, retention limits).

Consent (Article 6(1)(a)): Scraping by definition doesn't get consent from every person whose data you collect. This basis is effectively unavailable for web scraping.

The publicly accessible data misconception: GDPR Recital 47 says legitimate interests may include processing "for direct marketing purposes." GDPR Recital 111 mentions that personal data "already made public" can be processed under specific conditions. But neither creates a blanket exception for scraping public profiles. The EU's General Court has ruled that data being publicly accessible doesn't strip it of GDPR protection.

Practical GDPR risk assessment:

• Scraping pricing data with no personal information: very low GDPR risk
• Scraping public LinkedIn profiles to build a sales contact database: high GDPR risk
• Scraping news articles (no personal data): negligible GDPR risk
• Scraping property listing sites that include names of real estate agents: moderate GDPR risk for agent data

4. California CCPA / Other State Privacy Laws

CCPA applies to businesses that collect personal information of California residents and meet certain revenue or data thresholds. The technical applicability to web scraping is similar to GDPR but with different specific requirements.

The CCPA does not create individual causes of action for data collection (except for data breaches). Enforcement is by the California Privacy Protection Agency. As of 2025, there have been no significant CCPA enforcement actions specifically targeting web scraping.

Other states (Virginia, Colorado, Connecticut, Texas) have passed similar laws. The substance is similar to CCPA.

The robots.txt Question

robots.txt is a convention, not law. There is no legal obligation to honor it.

A website that serves data to all HTTP requests, with a robots.txt disallowing crawlers, has set a preference — not a legal restriction. In the hiQ v. LinkedIn litigation, LinkedIn argued that their robots.txt file constituted a technical access barrier. The court was not persuaded that robots.txt alone created the kind of technical restriction the CFAA contemplates.

This doesn't mean ignoring robots.txt is consequence-free:

• It's evidence that you knew the site didn't want you there (relevant in ToS litigation)
• It makes your intentions look worse if you're sued
• In the GDPR context, it's one factor in the "legitimate interests" balancing test

Honoring robots.txt is good practice not because it's legally required, but because violating it weakens your legal position across every other framework.

What Actually Gets Enforced

The gap between legal theory and enforcement reality is significant.

What gets prosecuted under the CFAA: Large-scale attacks on computer systems, insider threats, and sophisticated cybercrime. Not individual developers scraping publicly accessible websites. The DOJ has not prosecuted anyone solely for scraping publicly available data following Van Buren.

What generates civil lawsuits: Companies with competitive interests in the data and the resources to litigate. LinkedIn has sued multiple scraping companies (hiQ, several others). Twitter/X has sued several data providers. These are companies that had significant commercial operations built on scraped data — not individuals running price monitors.

What generates GDPR enforcement: Large-scale personal data collection from social platforms and public databases, particularly when used for profiling or direct marketing. The ICO (UK) and CNIL (France) have investigated and fined companies for scraping personal data without a lawful basis. Notably, the enforcement targets have been companies selling the resulting data, not developers doing one-off research.

The practical enforcement profile: A solo developer scraping product pricing from e-commerce sites, with no EU personal data in the dataset, doing small to moderate volume, is essentially zero enforcement risk in any jurisdiction. A company scraping LinkedIn profiles to build a B2B contact database and selling it commercially is in a genuinely high-risk category in multiple jurisdictions.

A Five-Question Pre-Scraping Checklist

Before starting a scraping project, answer these:

1. Does the data contain personal information of identifiable individuals? If yes, EU users trigger GDPR. Apply the legitimate interests test seriously, or find a data source that doesn't contain personal information.

2. Is the data behind a login wall or technical access barrier? If yes, the CFAA risk is substantially higher. The publicly-accessible data precedents (hiQ, Van Buren) don't protect you here.

3. Am I circumventing any technical protection measures? Bypassing CAPTCHAs, defeating rate limiting through proxy rotation, defeating IP blocks — these are all on a legal spectrum that becomes riskier as the technical sophistication of the countermeasure increases.

4. Could this data be used to directly harm the target site's business? Building a competing product using their scraped data, or publishing a database that substitutes for their product, significantly increases civil litigation risk.

5. At what scale am I operating, and is it commercial? Personal research, academic projects, and small-scale monitoring are treated very differently (both legally and practically) from commercial operations at large scale.

The Ethical Dimension That Isn't Covered by Law

The law answers "what can you get away with." Ethics asks something different.

Some scraping harms no one: collecting publicly available pricing data to build a comparison tool genuinely benefits consumers and doesn't damage anyone. The site didn't sell that price data; you making it more accessible doesn't take anything from them.

Other scraping imposes real costs: hitting a small independent website with hundreds of requests per minute to collect its entire product catalog causes infrastructure costs that the site owner bears. It's not illegal. It's not ethical.

The test that's more useful than "is it public?": would the site owner, informed of exactly what you're doing and why, consider it reasonable? Not would they be happy about it (they won't) — but would a reasonable person in their position consider it within the range of acceptable behavior?

By this test: scraping Amazon's pricing is reasonable (they have the resources, it's competitive intelligence, no one's harmed). Scraping a small blog's entire article archive to train a commercial LLM without compensation is not reasonable — the creator gets nothing while a commercial entity profits.

The fact that something is legally permissible does not mean it's without consequence for the people affected. That's the ethical question that law doesn't answer for you.