Building a Rate-Limited Scraping and Ingestion Pipeline Responsibly
A real estate analytics client needed listing data ingested from several public sources that didn't offer bulk APIs, only paginated web pages meant for browsers. Our first pass was a straightforward concurrent scraper — twenty workers pulling pages in parallel to get an initial dataset built quickly. Within four days, one source had blocked our entire IP range, and we had no visibility into why until we checked their robots.txt and terms of service, which we should have read first.
We rebuilt around explicit rate limiting per source rather than per our own convenience: a token bucket capped at one request every 2 seconds per domain, configurable per source since different providers had different published or implied tolerances. This is slower — a full crawl of one source went from 40 minutes to just under 5 hours — but it's the difference between a sustainable pipeline and one that gets blocked every time we need fresh data.
We also added proper caching and conditional requests: storing ETags and Last-Modified headers per page and sending If-None-Match on subsequent crawls, so pages that hadn't changed since the last crawl returned a cheap 304 instead of a full page transfer. For sources that supported it, this cut actual bandwidth and request volume by more than half on repeat crawls, since most listings don't change day to day.
For the two sources that did offer an official API with usage terms, we used those instead of scraping their HTML, even though the API had less data than the public pages — a stable, sanctioned integration beats a faster but adversarial one, especially for a client whose business depends on this data continuing to flow. We documented per-source what we scrape, how often, and why, both for our own maintainability and because a provider asking 'why are you hitting us' deserves a real answer.
The pipeline now runs as a set of scheduled Lambda functions per source, each with its own rate limit, retry backoff, and a circuit breaker that pauses a source entirely for 24 hours if we see a spike in 4xx responses — treating that as a signal we're doing something wrong rather than something to push through. It's slower than our first version and has run without a single block since.
