What crawl speed should I use on Amazon?

Careful. Amazon actively monitors automated access, and slower extraction with proper session preparation (a normal, aged Chrome profile, no VPN) is far less likely to trigger a CAPTCHA than running at Balanced or Aggressive speed.

How do I track bestseller rank changes over time?

Run the same extraction daily and save each export with a datestamp in the filename, e.g. headphones_bestsellers_2026-06-20.csv. Crawl Pilot's History tab lets you re-run the same job with one click each day.

How to Scrape Amazon Best Sellers

Amazon Bestseller Rank (BSR) is one of the clearest demand signals you can get without a paid research tool. A product ranked #1 in Kitchen Appliances is outselling everything in that category. Combined with price and review count, even 30 days of historical BSR data tells you more than most market reports.

The challenge is Amazon's detection system. Unlike a practice site, Amazon actively monitors automated access and serves incomplete data to sessions it has flagged.

This guide covers how to extract bestseller data with Crawl Pilot's List Extractor, what breaks, and how to structure your data for repeated tracking.

Before You Start: Session Preparation

Amazon's detection system treats fresh, cookieless sessions as suspicious. Do this before opening the extension:

02

Open a normal Chrome session (not Incognito) and browse Amazon naturally for 2–3 minutes — load the homepage, click a department, view a product. This establishes the cookie chain Amazon expects.
04

Use your regular Chrome profile, not a throwaway. A session with purchase history and browsing history has a higher tolerance for extraction than a brand-new session.
06

Don't use a VPN. Your IP should match your browser's timezone. Geographic inconsistency between IP and timezone is a detection signal.
08

Scroll to the bottom of the bestseller page before starting extraction. Amazon lazy-loads some products — they don't render in the DOM until you scroll to them. Wait 2–3 seconds after scrolling for everything to settle.

Navigate to Your Target Category

Go to https://www.amazon.com/Best-Sellers/zgbs and drill into your category using the left sidebar. The more specific the subcategory, the more comparable the data.

For this guide: Electronics → Headphones.

Wait until the full product grid is visible — typically 30–50 products.

Open CrawlPilot and Connect the Tab

Click the CrawlPilot icon in your Chrome toolbar. The side panel opens on the right.

If the banner shows "Extension not connected to this tab", click Connect. Click List Extractor in the left sidebar — it runs as a three-step wizard.

Step 1: Wizard Step 1 — Pick a Repeating Item

You don't map out columns or hunt for containers. You click one product card and CrawlPilot detects the rest.

02
Click Select on Page, then hover over one individual product card and click it.
04
CrawlPilot detects the pattern and shows a green "Repeating Pattern Found · N matching items on page" card — N should match the visible product count (30 or 50 depending on Amazon's layout for that category). It auto-extracts the fields it found on the card: product title, price, star rating, and the link (the ASIN is embedded in it, e.g. /dp/B09XS7JWHH/).

Before continuing:

Click Preview Sample Data (database icon) to confirm the rows look right before the full run.
Use Highlight on page (eye icon) to verify every product card was caught.
Use the + to add fields auto-detection missed — for example rank (#1, #2) or review count. Click +, then click that element on the card. Rank is typically inside a span with class zg-bdg-text and review count in a span whose aria-label contains "ratings"; use Chrome DevTools (right-click → Inspect) if you need to confirm what you're clicking.
Rename columns to friendly names and remove any you don't need.

Your fields will end up something like:

text

rank           → Rank position (#1, #2, ...)
product_title  → Full product name
price          → Price as text (some items show ranges like "$24–$89")
star_rating    → Rating string ("4.4 out of 5 stars")
review_count   → Review count text
product_url    → Full product URL (contains ASIN)

Once the preview looks right, click Next.

Step 2: Wizard Step 2 — Choose How to Load More Items

The wizard asks "How should we load more items?" with three options: Numbered Pagination, "Load More" Button, and Infinite Scroll / Single Page.

Amazon Bestsellers show 30–50 products per page with a "Next Page" link at the bottom (up to 2 pages, ~100 products per category), so choose Numbered Pagination. The wizard asks you to point to the "Next" button — click the "Next Page" link at the bottom of the bestseller list so CrawlPilot learns the selector. Click Next.

Step 3: Wizard Step 3 — Review & Run

The final step shows a Summary and a Crawl Speed setting: Careful, Balanced, or Aggressive. On Amazon, choose Careful — slower is safer. Open Advanced Settings and set Max Pages to Extract to 2.

Click Start Extraction. The status bar shows a live item count as products are collected — a single 30-product page takes 5–10 seconds; two pages takes 20–30 seconds at Careful speed.

Step 4: Open the Results Table and Export

When it finishes, click Open Results Table. Click Export and choose CSV or JSON (Markdown, Google Sheets, Webhook, Airtable, and Zapier are also available). You get one row per product.

Your CSV output will look like:

csv

rank,product_title,price,star_rating,review_count,product_url
#1,Sony WH-1000XM5 Wireless Headphones,$279.99,4.4 out of 5 stars,"45,231",https://www.amazon.com/.../dp/B09XS7JWHH/...
#2,Apple AirPods Pro (2nd Generation),$189.00,4.7 out of 5 stars,"82,134",https://www.amazon.com/.../dp/B0BDHWDR12/...

Useful post-processing in Google Sheets or Python:

python

import re

# Extract ASIN from URL
asin = re.search(r'/dp/([A-Z0-9]{10})', row['product_url']).group(1)

# Parse rank to integer
rank = int(row['rank'].replace('#', '').replace(',', ''))

# Parse price to float (handle ranges by taking the lower bound)
price_str = row['price'].split('–')[0]
price = float(re.sub(r'[^0-9.]', '', price_str))

# Parse review count to integer
reviews = int(row['review_count'].replace(',', ''))

What Breaks and Why

Price shows as blank for some products

Amazon requires a logged-in session to show prices on some third-party marketplace listings. If you see empty price fields, make sure you're browsing Amazon logged in before running the extraction. Your session cookies carry the login state into the extraction.

Item count stops at 15 instead of 30

Some products haven't lazy-loaded yet. Scroll all the way to the bottom of the page, wait 2–3 seconds, then start the extraction. CrawlPilot picks up what's rendered in the DOM at extraction time.

CAPTCHA appears mid-extraction

Amazon's detection was triggered. The extension cannot solve CAPTCHAs. Stop the extraction, solve the CAPTCHA manually in the browser, and wait at least 10 minutes before trying again. Don't extract the same category more than once every 30–60 minutes from the same session.

Pagination shows the same 30 products on both pages

Amazon uses JavaScript-triggered pagination on some category pages — clicking "Next" reloads content without changing the URL. If you see duplicated data, handle pages manually: run one extraction per page and merge the two CSVs by opening both in the Data Table and using Merge Datasets.

Price values vary from what you see normally

Amazon A/B tests pricing constantly. Different sessions see different price formats, promo badges, and discount overlays. The extension captures exactly what your session sees — which is correct, but may differ from another user's view on the same page at the same time.

Building a Useful Dataset Over Time

A single extraction snapshot has limited value. The signal comes from tracking rank changes over time.

Run the same extraction daily for 30 days and you can answer:

Which products are consistently #1–#5 vs. occasionally appearing (trending or seasonal)?
When did a new product enter the top 10, and at what trajectory?
How do price drops correlate with rank jumps? (A product moving from $89 to $69 and jumping from #12 to #3 is a direct pricing signal.)
Which brands dominate and how does that correlate with review count vs. competitors?

Save each export with a datestamp in the filename: headphones_bestsellers_2026-06-20.csv. After 30 days you'll have a dataset that no paid market research report covers for your specific category window.

Use CrawlPilot's History tab to re-run the same job each day — the configuration is saved and you click one button to repeat it.

How to Scrape Amazon Best Sellers with Crawl Pilot: A Step-by-Step Guide