Design a Web Crawler in 2026: Frontier, Politeness and a Signed Bot Identity

Every page a crawler downloads gets paid for twice. You pay in bandwidth and compute; the site owner pays in server time they never agreed to spend. The first bill is yours to scale. The second sets the rules: a crawler that hammers a small shop gets its IPs blocked, its user agent written into robots.txt, and in 2026 its traffic sorted into a category the CDN blocks by default. So the interview answer that starts from "how many machines do I need" is starting from the wrong bill. Start from how hard you're allowed to hit any one host, and the rest of the design falls out of that.

Whose Servers Set the Speed Limit

Take an illustrative target of one billion HTML pages a day. Every number below is an assumption chosen to make the arithmetic visible, not a measurement of any real crawler:

• 1,000,000,000 ÷ 86,400 s ≈ 11,600 fetches a second on average; at a 2× peak, about 23,000.
• At an assumed 100 KB of HTML per page, the peak is about 18.5 Gbit/s of download, and a day is 100 TB raw. Compressed to a quarter, about 25 TB a day to keep.
• Now the politeness constraint. If each host gets at most one request every 5 seconds, then 23,000 fetches a second only stays polite when about 116,000 different hosts are due for a fetch at the same moment.

That last line is the one that matters. Bandwidth and storage are a purchasing problem. Keeping a hundred thousand hosts in rotation, each on its own clock, is a scheduling problem, and it lives in one component: the URL frontier. Move the sliders and watch which number grows fastest:

The Crawler Everyone Draws First

The whiteboard version has five boxes and one loop. Seed URLs go into a queue, fetchers pull from it, a parser extracts links, and the links go back into the same queue.

Nothing in it is wrong. It just has no idea that the URLs in the queue belong to different people. A product page links to forty other product pages on the same site, so the parser enqueues forty same-host URLs in a row, and the next forty idle fetchers all hit that one shop together. That's the failure, and it's worth seeing rather than taking on faith.

Put the Naive Frontier Under Load

The simulator below crawls 60 made-up URLs on five hosts, in the order a parser would discover them: in same-host bursts. Each host allows one request start every D seconds. Start on Global FIFO, press Replay, and watch the red blocks: each is a request that started sooner than the host allows. Then try the other two designs. Arrow keys on the timeline step one simulated second.

Three things to notice, all from the simulator's own illustrative numbers:

1. 01
   Global FIFO is fast because it's rude. With 4 fetchers and a 2-second delay it finishes at about 5.5 pages a second, and almost every request breaks the limit. Fetchers land on the same host in the same tenth of a second.
2. 02
   Adding a wait to the single queue fixes politeness and kills throughput. When the head of the queue is a host that isn't due, every fetcher waits behind it: head-of-line blocking. Throughput drops to about 0.6 pages a second, and adding fetchers changes nothing.
3. 03
   Per-host queues stay polite and roughly double that. Fetchers skip to whichever host is due soonest. The run now ends when the biggest host drains: shop.example's 24 URLs at one every 2 seconds. With only five hosts, polite throughput can never exceed five requests per delay window. A real crawler escapes that ceiling the only way available: more hosts in rotation, which is the 116,000 from the sizing above.

Two Queues Deep: Priority in Front, Politeness Behind

The fix that survives scale was described in Mercator, Heydon and Najork's crawler from 1999: the frontier is a set of FIFO subqueues, and the subqueue a new URL goes into "is determined by the URL's canonical host name", one subqueue per worker thread, so at most one thread downloads from any given server. Modern write-ups split that into two layers. Front queues decide what is worth fetching next. Back queues, one per host, decide when it is polite to fetch it. Follow one link through the whole path:

Two details in step ⑥ deserve a word. Mercator found that DNS, not HTTP, was its bottleneck: blocking lookups took 87% of each worker thread's elapsed time until the team wrote a multi-threaded resolver, which cut that to 25% (Heydon and Najork, §3.7). So keep a DNS cache per frontier shard and resolve asynchronously. And the politeness delay shouldn't be a constant: when a host answers 429 or 503, back that host off, and honour Retry-After when it's sent (RFC 6585 §4). Google tells site owners to return 500, 503 or 429 precisely to slow Googlebot down (Google), so a crawler that ignores those codes is ignoring the one signal sites have been told to use.

The front layer is where the ordering choice lives, and it's a real fork:

Asking Permission: robots.txt Under RFC 9309

Before the first GET to a host, the fetcher needs that host's robots.txt. Since September 2022 the format is a proposed standard, RFC 9309, and the edge cases are the parts interviewers like:

• Caching. You may cache it, but "crawlers SHOULD NOT use the cached version for more than 24 hours, unless the robots.txt file is unreachable" (§2.4). Store it next to the host's queue so the check is a memory lookup.
• Errors split two ways. A 4xx means unavailable: the crawler may access anything. A 5xx or network failure means unreachable: assume complete disallow until it comes back (§2.3.1.3–2.3.1.4). Getting these backwards means hammering a struggling site the moment it fails.
• Limits. Follow at least five redirects, and parse at least 500 KiB (§2.3.1.2, §2.5).
• Matching. A bot uses its own user-agent group over *, and the longest matching rule wins (§2.2.2).
• Crawl-delay isn't in the RFC. Google says it doesn't support the field (Google); Cloudflare's crawl endpoint honours it (Cloudflare). Treat it as a hint that can only make you slower.

In 2026 most of a robots.txt file's new lines name AI crawlers. Here is a shop that wants search traffic but not model training. Try /products/blue-kettle, then /cart/, and see which bots each rule catches:

The design point: your crawler's user-agent token is its contract with the site. If it does several jobs (search indexing and training data), sites can't express a different answer for each unless you split the jobs across separate tokens. That split is the whole story of Cloudflare's 2026 changes, below. And remember what robots.txt is legally: a request, not a lock. A federal judge held in December 2025 that ignoring it isn't DMCA circumvention, which removes one legal theory and leaves the others (our write-up).

Seen It Already: URLs, Bytes and Near-Copies

Deduplication happens twice, on different keys. Before fetching, on the URL: have we queued this address already? After fetching, on the content: have we stored these bytes, or nearly these bytes, under a different address? Step through both:

The URL side is a memory question. An exact set of ten billion URLs at roughly 100 bytes each is about a terabyte; a Bloom filter at a 1% false-positive rate needs about 9.6 bits per URL, so roughly 12 GB for the same ten billion (standard Bloom filter sizing; the URL count and length are illustrative). The catch is which way the error points. In a fraud blocklist a false positive costs an extra lookup. In a crawler a false positive means a URL the filter wrongly thinks it has seen, so that page is never fetched and nothing reports it missing. Choose the rate as "how many pages am I willing to lose silently", and that's why step ② confirms the important ones.

The content side is older than it sounds. Mercator kept a 64-bit fingerprint per document for exact copies and found that 8.5% of successful downloads in an eight-day crawl were duplicates (Heydon and Najork, §6). Exact hashes miss pages that differ only in an ad or a date, so researchers at Google showed that Charikar's SimHash works for near-duplicates at multi-billion-page scale, with a fast way to find stored fingerprints that differ in at most k bits (Manku, Jain and Das Sarma, WWW 2007). For URL normalisation itself, start from the syntax-based rules in RFC 3986 §6 and add per-site rules (which query parameters are noise) learned from content hashes that keep colliding.

Coming Back: The Re-Crawl Loop

A page fetched once is a snapshot. Keeping an index fresh means fetching it again, and every re-fetch competes with new URLs for the same politeness budget. So re-crawls go through the same frontier, with a priority set by how often that page actually changes.

Conditional requests are cheap to support and save the site the most: a 304 Not Modified carries no body (RFC 9110 §15.4.5). Sitemaps help too, with care: Google says it uses <lastmod> only when it's "consistently and verifiably" accurate, and ignores <priority> and <changefreq> (Google). Your scheduler should take the same stance: trust a site's hints in proportion to how often they've matched what you observed.

Splitting the Work by Host

One machine can't hold 116,000 host clocks, a 12 GB filter and 23,000 open connections, so the frontier is sharded. The key is the host (or registered domain), for the same reason a rate limiter shards by the key it limits: every politeness decision about shop.example then happens on one shard, with no cross-machine coordination. A link to another host found on any shard is forwarded to the owning shard. Assembled:

Three consequences of host sharding. Hot hosts are a skew problem: a site with millions of URLs pins one shard's queue but not its fetchers, since that host only gets one request per delay anyway. Rebalancing moves clocks: when a shard dies, its hosts move with their next-allowed times and robots.txt cache, or the new owner will treat every host as fresh and burst them. Storage splits by access pattern: raw pages go to object storage in large compressed batch files, while URL state (last fetch, ETag, hashes, next visit) lives in a key-value store keyed by URL and partitioned by the same host hash.

Traps, Timeouts and Hostile Pages

The web contains infinite URL spaces, most of them accidental. A calendar with a "next month" link, faceted search that permutes filters, session IDs in paths, relative links that nest /a/b/a/b/…. None of them break the crawler loudly. They fill a host's queue with junk and spend its politeness budget on pages nobody wants.

The per-host budget does double duty: it's the trap guard and the fairness guarantee. A host can only consume what the frontier assigns to it, however many URLs it generates.

Showing Your Papers in 2026

Everything above is the classic answer. Three things changed it in 2026, and all three are about identity rather than throughput.

Sites now sort crawlers by purpose. Cloudflare classifies AI crawler traffic as Search, Agent or Training (Cloudflare). From September 15, 2026, a domain newly onboarding is offered a preset; for ad-supported sites it allows Search, sets Training to "Disallow AI Training" (a no-training preference published in robots.txt), and blocks Agent crawlers on pages with ads. Googlebot, Applebot and Bingbot are designated "Accountable" mixed-use crawlers that stay allowed for search under that setting (Cloudflare; our analysis of the deadline). For a crawler designer the lesson is structural: one token doing several jobs gets the strictest answer any of those jobs earns. Give each purpose its own user-agent token and its own robots.txt group.

Honest self-identification is the norm, even for the infrastructure vendors. Cloudflare's own crawl endpoint, launched March 10, 2026, "self-identifies as a bot", honours robots.txt including crawl-delay, and "cannot bypass Cloudflare bot detection or captchas" (Cloudflare changelog). If the company that runs the bot wall ships a crawler that announces itself and accepts being stopped, a design that rotates browser user agents to look human is choosing a fight, not a feature.

Identity is becoming cryptographic. A user-agent string can be copied by anyone. The IETF's Web Bot Auth working group, chartered to cryptographically authenticate automated clients from search crawlers to AI agents (charter), adopted draft-ietf-webbotauth-httpsig-protocol as its working draft: version -00, published September 1, 2026, by authors from Cloudflare and Google. It's an Internet-Draft, not a standard, and it can still change. The mechanism builds on HTTP Message Signatures (RFC 9421): each request carries Signature, Signature-Input and Signature-Agent headers, the signature must cover the target authority plus created and expires, uses the tag web-bot-auth, and the Signature-Agent URL points at a key directory served from /.well-known/http-message-signatures-directory. Cloudflare already accepts these signatures for verified bots, with Ed25519 keys (Cloudflare docs). One signed fetch:

What this adds to the design is small and concrete. A signing service holds the private keys (in a KMS, never on fetcher disks) and signs on the fetch path with short expiry. The key directory is a tiny public endpoint you must keep up with key rotation. And the frontier gains a per-host identity policy: sign everywhere, keep separate tokens per purpose, and record per host which identity was refused so the scheduler stops spending budget on hosts that said no.

Keep This Page

Sources

• Heydon and Najork, Mercator: A Scalable, Extensible Web Crawler (per-host FIFO subqueues, DNS bottleneck, 8.5% duplicates)
• Manku, Jain and Das Sarma, Detecting Near-Duplicates for Web Crawling, WWW 2007
• IETF, RFC 9309: Robots Exclusion Protocol
• IETF, RFC 6585 §4: 429 Too Many Requests
• IETF, RFC 9110 §15.4.5: 304 Not Modified
• IETF, RFC 3986 §6: URI normalisation and comparison
• IETF Web Bot Auth working group, charter and draft-ietf-webbotauth-httpsig-protocol-00
• Cloudflare, Web Bot Auth verification
• Cloudflare, Crawl endpoint changelog, March 10, 2026
• Cloudflare, Content Independence Day: AI options and Accountable mixed-use AI crawlers
• Google, How Google interprets robots.txt, Reduce the Googlebot crawl rate and Build a sitemap