Design a Price Monitoring System: Spending a Freshness Budget, Not Crawling Everything

A system design walkthrough for tracking competitor prices across millions of product pages: why it isn't a web crawler, adaptive re-check intervals that spend fetches where prices move, a cheapest-first fetch ladder, structured-data extraction with sanity checks, product matching, change-point storage and alerting, with a proxy-cost calculator.

Rahul Bisht

Founder, CrawlPilot

·
Dec 16, 2026
·
Verified Nov 15, 2026
·System Design·
15 min read
·
Design a Price Monitoring System: Spending a Freshness Budget, Not Crawling Everything
Key takeaway
A price tracker looks like a crawler and has the opposite goal. A crawler wants to discover as many pages as possible; a price tracker already knows its pages and wants to know, as soon as possible, when the number on each one changes. So the core of the design is a scheduler that spends a fixed fetch budget where prices actually move, a fetch ladder that uses the cheapest method that works, and validation strict enough that a wrong number never becomes an alert.

If you've read the web crawler design, a price monitoring system will look familiar: a frontier of URLs, politeness per domain, fetchers, parsers, storage. Most of the boxes are the same. The objective is not. A crawler measures success in coverage: how much of the web did you find? A price tracker's URL list is mostly fixed (the products you care about), and success is freshness: how long between a retailer changing a price and you knowing about it?

That single difference flips most of the design decisions. A crawler spends its budget on new URLs. A price tracker spends its budget re-fetching the same URLs, and the whole game is deciding which ones, how often, and how cheaply.

The Job, in Numbers

Competitive price intelligence for a retailer or brand. Working requirements (illustrative numbers):

  • 20 million product URLs across about 500 competitor and marketplace sites.
  • Freshness by tier: the top 1% of products (best-sellers, price-sensitive categories) within 15 minutes; the next 20% within a few hours; the long tail daily.
  • Accurate: a wrong price is worse than a late one, because someone may reprice against it.
  • History: every observed price change kept for analytics and audit.
  • Polite and lawful: respect robots.txt and per-site rate limits, and only collect public product data. The web data and the law posts cover the legal side; this one covers the system.

The Naive Design: Check Everything Every Hour

The first version is a cron job that loops over every URL every hour. Do the arithmetic: 20 million fetches an hour is about 5,600 a second, around the clock. Most of them return the same price as last time, because most products don't change price most hours. You're paying proxies, compute and the retailer's patience to learn nothing, and you're still an hour late on the products that matter.

Worse, a fixed schedule is easy to spot. Five hundred requests a minute to one retailer, every minute, from the same few networks, is exactly the pattern anti-bot systems are built to catch. The anti-bot detection post goes into how.

The fix is to stop thinking in schedules and start thinking in a budget.

The Freshness Budget

You can afford some number of fetches a day, set by proxy spend, compute and per-site politeness limits. Call it the budget. The scheduler's job is to spend it where it buys the most freshness.

Each URL gets a next_check time computed from three things:

  1. 01
    How often its price has actually changed. Track it per URL. When a fetch finds a change, shorten the interval (halve it, say). When it finds no change, lengthen it (multiply by 1.5). Keep it between a floor and a ceiling per tier.
  2. 02
    How much it matters. A business priority multiplier: best-sellers and items you're actively repricing check more often.
  3. 03
    What's happening. On known sale events (a retailer's big sale day, a holiday weekend), temporarily lower every interval for that retailer.

Then the per-domain budget caps the total. If a retailer's politeness limit is 2 requests a second, that's about 170,000 fetches a day for that domain, no matter how many of its URLs are due. When demand exceeds the cap, the most overdue, highest-priority URLs go first.

How one URL's re-check interval adapts
1yes2no3next_check4next_check4Fetch + extractprice observedChanged?vs last priceInterval ÷ 2floor: 15 minInterval × 1.5ceiling: 24 hDomain budgete.g. 2 req/s per site
  1. A fetch extracts the current price and compares it with the last one stored for this URL.
  2. If it changed, the interval halves, down to the tier's floor. A product in an active price war ends up checked every 15 minutes.
  3. If it didn't change, the interval grows by half, up to the ceiling. A product whose price hasn't moved in a month drifts to daily checks.
  4. Either way the URL goes back in the queue with a new next_check. The per-domain budget decides what actually gets fetched when more URLs are due than the site's limit allows.

4 steps. Press Play to walk through them, or click any part of the diagram.

The interval shrinks when a change is seen, grows when it isn't, stays within tier bounds, and drops for sale events; the domain budget decides what actually gets fetched.

The effect is large. Most catalogues have a small share of products whose prices move often and a long tail that rarely moves. An adaptive scheduler concentrates fetches on the first group automatically, without anyone hand-tuning tiers. The scheduling machinery itself (leases, shards, due-time polling) is the same as in the job scheduler design; here every URL is a schedule whose interval rewrites itself.

Insight
Add a little randomness to every next_check. It costs nothing in freshness, and it stops your requests to each site from arriving on a beat that is easy to fingerprint.

The Fetch Ladder: Cheapest Method That Works

Fetching a product page can cost a thousand times more one way than another. A plain HTTP request through a datacenter proxy is cheap. A full headless browser through a residential proxy is much more expensive in bandwidth and in CPU. So each site gets a ladder, and fetchers start at the bottom rung that's known to work for it:

RungMethodWhen it's enough
1Retailer feed, sitemap with lastmod, or public APIThe site publishes product data you're allowed to use
2Plain HTTP GET, conditional where possibleServer-rendered pages; If-None-Match or If-Modified-Since can skip unchanged pages entirely (RFC 9110)
3HTTP GET of the site's own JSON endpointThe page loads its price from an API call you can see
4Headless browserThe price only exists after JavaScript runs
5Headless browser on residential or mobile routesThe site blocks datacenter traffic

Each site's rung is learned, not configured once: if rung 2 starts returning challenge pages, the site moves up a rung for a while, and periodically the system tries stepping back down. The proxy industry post explains why the upper rungs cost what they do.

Proxy bandwidth is usually the biggest line item, and it scales with fetches, page size and success rate. Try your own numbers:

4,000,000
350 KB
85%
$5.00
Fetches a day
4,705,882
Bandwidth a day
1,647 GB
Proxy cost a day
$8,235
Proxy cost a month
$247,059/mo

At this spend, a data aggregator, an official API (Amazon SP-API, $0 for approved use), or a smaller high-value subset of pages usually beats raw proxy bandwidth.

Proxy bandwidth only: not compute, vendor minimums, or the extra parsing retries cost. Billed in decimal GB (1 GB = 1,000,000 KB), the way vendors meter it. Per-GB prices vary by vendor and are illustrative, not a quote: roughly $0.50–2 datacenter, $3–15 residential, $2–8 ISP, $15–50 mobile.

Proxy bandwidth for a price monitoring fleet. Defaults are illustrative (4 million successful fetches a day after adaptive scheduling), not a vendor quote.

Notice what moves the total most. Halving the number of fetches (better scheduling) halves the bill. Raising the success rate cuts retries. Fetching a 20 KB JSON endpoint instead of a 350 KB HTML page is a 17-fold saving on that site.

Extracting the Right Number

Getting a page is half the problem. A product page has many numbers on it: the price, the crossed-out list price, the per-unit price, a bundle price, the price of a recommended product, a shipping cost. Extracting the wrong one is the commonest source of bad data.

Extract in order of reliability:

  1. 01
    Structured data. Many retailers embed schema.org Product and Offer markup as JSON-LD, with price, priceCurrency and availability, because Google uses it for product rich results. When it's present, it's the retailer's own statement of the price, in a fixed format.
  2. 02
    Per-site selectors. For sites without structured data, maintained CSS or XPath selectors. They break on redesigns; monitor extraction failure rates per site.
  3. 03
    An LLM extractor for the long tail of small sites, given the page and a schema. It costs more per page and needs the same validation. The selectors versus LLMs post works through when that trade pays off.

Then validate every extracted price before it's stored:

  • Currency and unit match what's expected for that site and product.
  • Plausibility against history: a change of more than a set percentage (50% is a reasonable start) is held, not published.
  • Confirmation refetch: a held price is refetched, ideally via a different route, a few minutes later. Two independent observations of a big drop are a real drop. One is often a glitch, a test page or a wrong selector.
  • Availability is a separate field. "Out of stock" isn't a price of zero.

Matching Their Product to Yours

A competitor's price is useless until you know which of your products it corresponds to. Matching is its own subsystem, and it runs when a URL is added and periodically after, not on every fetch.

Matching a competitor listing to your catalog
identifiers on page?1exact hit1no IDs23high confidence4uncertain4Competitor listingtitle, brand, specs,Exact IDsGTIN, UPC, MPNCandidate searchtext + image embeddingsMatch scoreattributes must agreeHuman reviewuncertain middleAccepted matchcompetitor URL → your
  1. If the page exposes a GTIN, UPC, EAN or manufacturer part number that matches one of yours, the match is exact. Prefer this whenever it's available.
  2. Otherwise, search your catalog for candidates using text embeddings of the normalised title and image embeddings of the main photo.
  3. Score each candidate on brand, title similarity, image similarity and key attributes. Attributes act as hard checks: a 500 ml bottle is not a 1 l bottle, and a 3-pack is not a single, however similar the titles.
  4. Auto-accept only high scores. Send the uncertain middle to human review, and feed reviewers' decisions back as training data for the scorer.

4 steps. Press Play to walk through them, or click any part of the diagram.

Try exact identifiers first; otherwise score candidates on title, brand, attributes and image similarity; auto-accept high confidence, review the middle, reject the rest.

The embedding and vector search part is the same machinery as the vector database post. The part people underestimate is the attribute checks. Most bad matches I'd expect to see are near-identical products that differ in size or pack count, and those produce price comparisons that look like huge undercuts.

Storing Prices as Change Points

Store every observation, briefly; store history as change points.

  • Observations (URL, time, extracted price, availability, fetch method, raw page reference) go to cheap append-only storage with a short retention. They're for debugging extraction.
  • Price history stores a row only when something changes: (product, competitor, price, currency, availability, valid_from, valid_to). A product checked every 15 minutes for a year whose price changed 12 times is 12 rows, not 35,000.
  • Current price per (product, competitor) lives in a fast key-value store for dashboards and repricing.

Each confirmed change emits an event. Alert rules subscribe to events, not to the database: "competitor undercut us by more than 5% on a top-1,000 product", "minimum advertised price violated by a reseller", "competitor went out of stock on an item we have". Because alerts fire only on validated, confirmed changes, a broken selector produces a dip in extraction success, not a flood of false undercuts.

The Assembled System

Price monitoring system
due URLs within budgetpage or JSONchanged? → new intervalconfirmed priceURL → your SKUchange eventsAdaptive schedulernext_check per URLFetch ladderfeed → HTTP → browserExtract + validateJSON-LD firstProduct matchingIDs, then scoresPrice historychange pointsAlerts + repricingon confirmed changes
An adaptive scheduler releases due URLs within each domain's budget; fetchers climb a cost ladder; extraction and validation produce confirmed prices; matching links them to your catalog; changes are stored as change points and drive alerts.

Test Yourself

Check yourself

Q1What's the main difference between a web crawler and a price tracker?

Q2A product's price hasn't changed in its last ten checks. What should the scheduler do?

Q3An extracted price is 70% below yesterday's. What happens before an alert fires?

Cheat sheet
Price monitoring system design

Scheduling

  • Budget, not schedule: fetches per day per domain are fixed.
  • Interval halves on change, grows ×1.5 without, within tier bounds.
  • Priority multipliers, sale-event overrides, and jitter.

Fetching

  • Cheapest rung first: feed, conditional GET, JSON endpoint, browser, residential.
  • Learn each site's rung; try stepping down periodically.
  • Respect robots.txt and per-site rate limits.

Data quality

  • JSON-LD Offer first, then selectors, then an LLM extractor.
  • Check currency, unit and history; refetch to confirm big moves.
  • Match by GTIN/MPN first; attributes are hard checks.

Storage and alerts

  • Short-lived raw observations; history as change points.
  • Alerts on confirmed change events, not on raw fetches.

Sources