The Web Was Not Built for Machines: Why Structured Data Access Needs to Change

Somewhere between 4 and 5 billion web pages exist on the public internet today. They contain the pricing decisions of millions of businesses, the output of millions of writers and researchers, the opinions and reviews and arguments of billions of people. The sum of accumulated human economic and intellectual activity, much of it public, most of it findable.

Almost none of it is usable as data.

Not because accessing it is illegal. Not because people don't want it to be accessible. Because the format it's stored in — HTML — was never designed to be queried by machines. It was designed to be read by humans.

That gap — between the world's largest collection of information and the ability to actually use it — is what Crawl Pilot exists to close. This post is about why we think that's worth doing, the counterarguments we take seriously, and what we believe about how it should be done.

The Query Layer That Doesn't Exist

Every mature information system has a query layer. A relational database has SQL. A document store has filter APIs. A file system has search. These interfaces are how you go from "there's data in here somewhere" to "here is the specific data I need."

The web has no query layer. There's Google, which surfaces documents, not data — it can tell you which page contains the price of a Sony WH-1000XM5, but it cannot tell you the price itself in a form you can compute with. There are official APIs from companies willing to expose them, covering perhaps 1–2% of the information on the web. And then there's everything else: billions of pages of structured information locked inside display markup, accessible only to eyes and not to code.

This means that right now, in 2025, extracting information from the web at any scale requires either:

1. 02
   Writing and maintaining brittle custom scrapers that break every time a site redesigns
2. 04
   Paying data providers who've already done the scraping (and are charging substantial margins for it)
3. 06
   Doing it manually — humans reading pages and copying data into spreadsheets

None of these are adequate. Option 1 is expensive to maintain. Option 2 only covers sites large enough to have commercial data markets. Option 3 doesn't scale.

The cost of this inadequacy is enormous and mostly invisible. Researchers duplicate each other's data collection efforts. Businesses pay for data products they could have computed themselves. Small operators can't afford the commercial data providers and so make decisions without market intelligence. The web's information exists but remains effectively inaccessible to the majority of people who could use it.

Why This Has Been Hard to Fix

The obvious response: why haven't companies just built a universal web query API?

Because the economics work against it.

The companies that own the most valuable web data — Amazon, LinkedIn, Meta, Zillow, Google — have enormous incentives to keep that data proprietary. Amazon's product and pricing data drives a competitive intelligence industry worth billions. LinkedIn's professional graph is their primary asset. Their data is valuable precisely because it's not freely accessible as a dataset. Opening it would destroy the business model built on top of it.

So the web's most valuable data is locked behind walls. The rest — the long tail of content from smaller sites — is technically accessible but practically unreachable at any useful scale without investment in custom extraction infrastructure.

The result: a web that's simultaneously the most information-dense place in human history and one of the most difficult information sources to actually use systematically.

The Counterargument We Take Seriously

There's a legitimate version of the opposing view, and it's worth stating it clearly.

The argument goes: information has value because someone created it. That creator — a journalist, a retailer, a researcher, a developer — did the work of collecting, structuring, and presenting it. If you extract that information and use it freely, you benefit from their work without compensating them. At sufficient scale, you undermine the economic basis for creating the information in the first place.

This isn't hypothetical. It's happening. News publishers are in financial collapse partly because aggregators use their content without payment. AI companies trained on the web's text without compensating the writers whose work they used. The web's information commons is being harvested at scale by companies that return nothing to the ecosystem that generated it.

We don't think this argument says "web data should be inaccessible." We think it says who benefits and who doesn't matters, and the current system handles it badly.

Right now the people who benefit from web data at scale are the ones who can afford to build scraping infrastructure or pay commercial data providers. Small businesses, independent researchers, individual analysts — they have worse data access than large enterprises, even for information that's technically public. That's not a fair distribution of access.

And the people who create the information — journalists, content sites, small retailers — often get nothing regardless of whether it's a scraper or a data company or an AI company consuming their content. The current system doesn't protect them well either.

A better system would: make information broadly accessible to people who need it, and create mechanisms for people who create valuable information to receive something in return when it's used. Both of those things, simultaneously. They're not actually in conflict.

What We Think About How This Gets Done

The technical problem is largely solved. Browser automation is mature. LLMs can interpret unstructured HTML and extract structured data. The infrastructure to run headless browsers at global scale exists. You can build a system today that queries any public web page like a database.

The unsolved part is not technical. It's about norms, access, and economics.

On access: extraction tools should make web information accessible to individuals, not just large enterprises. The ability to extract data from a public web page should not require a $50K/year data provider contract or an engineering team. It should be a capability that a researcher, analyst, or business owner can use with a browser extension.

On responsibility: unrestricted extraction is not costless. Hitting a small site with thousands of requests per hour causes real infrastructure harm. Extracting personal information without a lawful basis violates people's reasonable privacy expectations. Using another company's entire dataset to build a direct competitor is parasitic. The tools that make extraction easy also make these harms easy, and the tools' designers have a responsibility to think about that.

On the long-term: we believe the web is moving toward a model where structured data access is metered and monetized at the infrastructure level. Cloudflare, Google, and others are building the technology layers that will let publishers control who accesses their content programmatically and at what price. This is probably the right direction for the information economy, even if the specific implementations are still being worked out. It's better than the current state where large companies scrape freely and small publishers have no recourse.

We're building extraction tools in this context — aware that the access we're making easier will eventually be structured differently than it is today, and that the most defensible position for a tool like Crawl Pilot is making the access legitimate (session-based, browser-native, not server-farm traffic) and accessible to individuals rather than only to large infrastructure operators.

The Specific Thing We Believe

The web should be queryable. Not in some abstract future sense, but now, with the tools that exist.

A small business owner should be able to check competitor prices without hiring a developer. A market researcher should be able to collect a week of data from a public site without a six-figure data contract. A journalist should be able to verify publicly available information at scale without writing Python.

We think the gap between "this information is publicly visible" and "I can use this information systematically" is a genuine injustice in how the current web works. Not a catastrophic injustice — but a real one, with real economic consequences for the people on the short side of it.

That's what we're trying to fix. Not because "data wants to be free" — it doesn't, and the people who create it deserve compensation for it. But because right now, the people who have access to the web's information systematically are large enterprises and well-funded data companies, and the people who don't are everyone else. That distribution is wrong, and it's worth building tools that make it less wrong.

Whether we succeed at that while also being a viable business is a question we're working on in real time. But it's the right goal.