The CrawlPilot blog
Interactive system designs, scraping guides and the news that changes how the web can be read. 63 posts and counting.
The European Data Protection Board's draft Guidelines 03/2026 set out how the GDPR applies to scraping for generative AI: legitimate interest, robots.txt and CAPTCHAs as evidence of expectations, filtering, special-category data and contractor roles. Comments close October 30, 2026.
Turn a Walmart search into a spreadsheet of titles, prices, was-prices, ratings, review counts and product links with CrawlPilot: two clicks, real export columns, and how to clean what comes out.
Instant Data Scraper and CrawlPilot are both free Chrome extensions that turn a page into a spreadsheet. Where each is the better pick: one-click tables, many pages, each item's page, re-running, and exports.
A cloud zone losing power should cost you a third of your servers, not your service. How a zone loss turns into a regional outage, and the designs that stop it: static stability, zone-affine routing, cells, shuffle sharding and retry budgets, with a simulator to try them.
What Temporal is, how its event history and replay let a workflow survive a crashed worker, where it fits and where it doesn't, and a checkout saga in TypeScript, recorded from real runs: a flaky gateway, a declined card and a killed worker.
Meta's revised Commercial Terms let it crawl sites 'associated with' any business account. It dropped the word 'train' after an outcry and moved the date to November 20. What the clause says, what changed, and what robots.txt can and can't do about it.
A web crawler is limited by other people's servers, not yours. Build the frontier, robots.txt handling, dedup and re-crawl loop, then add 2026's AI-crawler rules and signed bot identity.
On July 31, 2026 a New York federal judge let Reddit's DMCA anti-circumvention claims against SerpApi and Perplexity go forward, treating Google's SearchGuard as an access control. What survived, what didn't, and why Google's own suit over the same lock went the other way.
A system design walkthrough for one API in front of several model providers and many tenants: token-aware rate limits, weighted fair share, circuit breakers, failover before and after the first token, caching, hard budgets and keys, with a simulator you can break.
