News Ingestion & Enrichment Engine
A crawl-to-publish pipeline across more than 10,000 US news sources — fetch, extract, de-duplicate, classify with models, publish — delivering fresh articles every second.
Client
EdTech (India/US)
Service
Data Engineering
Date
September 1, 2022
Challenge
Ten thousand sites have ten thousand page structures, most have no usable feed, many syndicate the same wire story under different bylines, and any of them can redesign overnight and silently break extraction. On top of that, news is only valuable while it is new.
Solution
Distributed fetch workers with per-domain scheduling and rate limits, content extraction with per-domain overrides where the generic path fails, hash and near-duplicate detection to collapse syndicated copies, a model pass for categorisation and quality, and a queue-driven publish step so freshness is measured in seconds.
The requirement
Cover US news at breadth — thousands of publishers, not a curated dozen — and have it available while it is still news. Those two requirements pull against each other. Breadth means most sources are small, badly structured and have no API. Freshness means you cannot batch overnight and reconcile in the morning.
How it was built
Fetching. A pool of distributed workers, scheduled per domain rather than per URL. Each domain gets its own politeness budget and crawl interval derived from how often it actually publishes — checking a wire service every few seconds and a small regional publisher every few minutes, rather than hammering all ten thousand on the same cadence. That single decision is what made the freshness target affordable: attention goes where news actually appears.
Extraction. A generic content extractor strips navigation, advertising and boilerplate to recover the article body, headline, byline and timestamp. It works on most of the web and fails on some of it, so there is a per-domain override path for the sites that matter enough to hand-tune. Extraction quality is monitored, because a site redesign shows up as a sudden drop in recovered body length long before anyone notices the articles look wrong.
De-duplication. A single wire story can appear across hundreds of outlets within minutes, sometimes verbatim, sometimes lightly rewritten. Exact-content hashing catches the verbatim copies cheaply; near-duplicate detection catches the rewrites. Without this layer the output is unusable — one event floods the feed and everything else disappears beneath it.
Enrichment. Models run over the extracted article for categorisation, relevance to the verticals being served, and a quality signal used to suppress low-value churn. This is where a crawl becomes a product: the pipeline emits classified, ranked, de-duplicated articles rather than a pile of scraped HTML.
Publishing. Queue-driven, so each stage backpressures independently and a slow model pass delays enrichment without stalling fetch. Steady state processed the full 10,000-source surface with new articles landing every second.
What I would tell someone building this
The crawler is the easy part. The cost is in extraction breakage and duplicate collapse, and both are ongoing operational work rather than one-time engineering. Build the monitoring for those two before scaling the source count, not after.