Why your scraper returns fewer rows every month

A scraper that breaks is a good outcome. Somebody notices within a day, the parser gets fixed, and the gap in the data is obvious and bounded. Nobody enjoys it, but the failure is honest.

The expensive version is different. The pipeline keeps running, the job goes green, the table fills up, and the row count drifts down by four per cent a month. Six months later somebody builds a market report on data that is missing a quarter of its listings, and there is no incident to point at, because nothing ever failed.

How the loss actually happens

Almost never through a total redesign. It happens through small changes that each remove a slice of coverage:

• A source adds a second listing template for one category. Your parser matches the original template, finds nothing, and returns an empty result for those pages rather than an error.
• Pagination changes from a page parameter to a cursor. You keep getting page one. Page one is real data, so nothing looks wrong.
• A result cap that used to be 1,000 becomes 400. Your geographic tiles were sized for the old cap, so dense areas now truncate silently.
• A field moves inside a JSON blob. The selector no longer matches, the field becomes null, and your COALESCE in the warehouse quietly substitutes a default.
• Rate limiting tightens. Your retry logic swallows the 429s, logs a warning nobody reads, and the run finishes with 80 per cent of its targets.

Every one of those returns a plausible dataset. None of them trip a job-level alert, because the job succeeded. It did what it was told. It was told the wrong thing.

The fix is not better selectors

The instinct is to make parsing more robust — fall back to a second selector, guess harder, tolerate more shapes. This makes the problem worse. Tolerant parsers are exactly the mechanism by which a broken source becomes an unremarkable-looking table.

The parser should be strict and the pipeline should be suspicious. Three things carry most of the weight:

Expected volume per unit. Every crawl unit — a market, a tile, a category — has a rough expected count based on history. A tile that returned 340 rows last week and 190 today is an incident regardless of whether every one of those 190 parsed cleanly. This single check catches truncation, pagination changes, and rate-limit loss.

Field-level null rates, tracked over time. Not "is this field nullable" but "what proportion of records had it yesterday". A field that was 98 per cent populated and is now 61 per cent populated has broken, even though the run succeeded and the schema still validates.

Template fingerprints. Hash the structural shape of the page — tag path skeleton, not content. When a page arrives whose fingerprint has never been seen, that is a new template, and it should be flagged for a human before its records enter the dataset. This is what catches the second listing template on day one instead of month four.

Fail closed, quarantine, and report

Once you can detect drift, the remaining decision is what to do about it. The default should be to fail closed: a run that trips a threshold does not overwrite yesterday's data. Records that fail validation go to a quarantine table with the reason attached, not to /dev/null and not into the main table with nulls.

Then report coverage as a first-class output. Every delivery should carry a manifest: units crawled, units expected, rows per unit against the trailing average, null rate per field, and the count quarantined. If the person consuming the data cannot see that the run was thin, they will assume it was complete, and they will be right to — you told them it succeeded.

Why this is a service problem, not a code problem

None of the above is difficult to build. It is difficult to keep. Volume baselines need updating when a market genuinely grows. Fingerprints need reviewing when a source legitimately redesigns. Thresholds that fire too often get muted, and a muted alert is worse than no alert because it comes with the feeling of being covered.

That maintenance is the actual product in managed extraction. The crawler is a week of work. Noticing, within a day, that a source changed something on a Tuesday night — and having somebody whose job it is to fix it before your Monday report — is the part you are really buying.

If you are running extraction in-house, the question worth asking is not "does it work". It is: if this quietly lost twenty per cent of its coverage tonight, how long until somebody knew?