HTTP 200 OK, Wrong Data: The Silent Scraper Failure That Costs You a Quarter

TL;DR — The worst data bug doesn't throw an error. It returns HTTP 200, writes plausible-looking values, and waits. More than 60 percent of production pipelines without automated contract testing experience schema drift; the average detection window is 8–72 hours, and McKinsey research finds silent failures consume up to 30 percent of data engineering time on reactive investigation. This post catalogs the five canonical silent-failure patterns, explains why alt-data, events, and job feeds are the hardest hit, and lays out the verification architecture that catches them — the same architecture that produces Crawlify's 99.5 percent verified field-accuracy SLA.

The failure mode no monitoring system catches

Every production data team has lived through some version of this incident. A scheduled scraper has been running for months. The dashboard is green. The status indicator says OK. Latency is normal, error rate is zero, the volume metric looks reasonable. And then — sometimes weeks later — a customer asks why every row in last Tuesday's pricing batch reads 0.00, or why the same Senior Engineer role from one city was billed as five different listings, or why the model trained on Q1's data is producing nonsense on Q2's.

The pipeline isn't broken. The pipeline is fine. The pipeline is doing exactly what it was told to do. What broke is upstream of the pipeline and downstream of what the monitoring system can see. That is a silent failure, and it is the most expensive class of bug a data product carries.

Definition check. A silent failure is a pipeline that completes successfully — HTTP 200, status green, no alert fires — but writes incorrect, missing, or stale data. The most common cause is schema drift: an upstream change in field names, types, or structure that corrupts values without raising an error. Other patterns include partial loads, dropped records, currency or unit shifts, and silent insertions or deletions. Each one passes every check that a typical "is it running?" monitor performs.

What the numbers say

Three recent pieces of research line up around the same observation:

• HST Solutions, 2026 — more than 60 percent of production pipelines without automated contract testing experience schema-drift incidents; average detection window is 8 to 72 hours after the triggering change.

• McKinsey — silent pipeline failures consume up to 30 percent of data engineering time on reactive investigation rather than planned development. Cited consistently across data-quality research from 2024–2026.

• Gartner — enterprise data downtime and silent failures cost an average of $12.9 million per year, with 60 percent of AI projects lacking data quality controls projected to be abandoned through 2026.

That last number rhymes with the cost figures from Blog 1 in this series. They aren't two different problems. They are the same problem viewed from two different organizations — once as a category-level data-quality cost (Gartner) and once as a vendor-specific accuracy SLA (Crawlify).

The five canonical silent-failure patterns

After running production pipelines across pricing, jobs, and events for two years, the failure modes resolve into five distinct patterns. None of them throw exceptions. All of them are catchable with the right verification layer.

1. Schema drift

An upstream API renames a field, switches a numeric type to a string, or restructures a nested object. The pipeline keeps running because the change is technically valid JSON. Downstream, the field becomes null, or zero, or a string that the type cast silently coerces. Schema drift is the most common silent failure because it looks like success.

2. Unit and currency drift

A retailer that quoted prices in USD starts quoting in EUR for European pageloads. A nightly-rates site flips between local-currency and base-currency display. The price field still parses as a float. The number is still positive. It is the wrong number in the wrong unit, and the model downstream treats it as comparable to last week's.

3. Partial loads and dropped records

Yesterday's batch delivered 10,000 rows. Today's delivered 7,000. Row count alerts catch this if they exist; in practice most teams don't have row-count baselines per source. The 3,000 missing records are the ones that quietly disappear from downstream aggregates.

4. Cross-page duplicates

The same role posted across five cities. The same conference listed on five aggregator sites. The same product showing up under multiple SKU IDs. AI extraction returns five confident rows. Without cross-page dedup, the downstream system treats them as five entities and the analytics layer over-counts.

5. Silent insertions and deletions

A source site adds three new records and removes one. The pipeline still returns the same row count by coincidence. Cell values look plausible. The deleted row's data, frozen at its last good value, persists in the customer's warehouse for weeks until someone notices the company in row 47 closed last quarter.

From the field. I ran a same-day audit on one customer's pricing feed in May. The feed had been green for 14 days. 22 percent of the rows had a currency-drift problem: a single retailer had started quoting nightly rates in local currency for one country segment, and the pipeline was casting the value to USD by raw float. The feed was technically fresh, technically complete, and technically wrong. Their on-call would not have caught it for another week.

Why the worst-hit verticals are the ones with the highest stakes

Three verticals carry the heaviest silent-failure cost. They share a property: there is no obvious external validation a downstream consumer can perform on the data, which is exactly what makes the silent version of the failure dangerous.

Alternative data for hedge funds

Web-scraped data is now the largest single category of alternative-data spend — roughly 15 percent of a $2.8 billion market per Neudata's 2025 sizing. Funds aren't short on sources. They are short on feeds that survive a page change without silently corrupting. One alt-data team I spoke with last year was maintaining 2,000 in-house scrapers; new source onboarding ran 2 to 4 weeks; their primary fear wasn't a scraper breaking visibly. It was the feed running quietly wrong for a week before someone noticed in a model backtest.

A 15 percent silent error rate on a feed that drives a trading signal doesn't look like a system error. It looks like a bad trade.

Events data

Event listings are a dedup nightmare. The same conference shows up on five sites with three different dates, two different venues, and four different submission deadlines. AI extraction matches them, picks one, and writes a confident row. Without human-verified dedup, an event aggregator quietly inflates the size of the calendar and ships the wrong date to the customer who registered. The ScholarMeet pipeline that Crawlify was first built to power exists because academic event data has zero tolerance for the wrong date — every PI in the field will notice.

Job-board and labor-market feeds

Ghost listings are a vertical-specific silent failure. The posting is technically present at the source, technically returns a 200, and technically passes every freshness check that does not cross-reference the source ATS. As Blog 2 in this series laid out, 27.4 percent of US LinkedIn listings and 18–22 percent of Greenhouse postings carry this exact failure mode — "successful" extraction of data that no longer represents a real hiring opportunity.

The detection-time math

The 8-to-72-hour window is the part of the silent-failure cost most teams underestimate. The cost of a wrong row in your warehouse is roughly proportional to how long it stays wrong before someone catches it. A schema drift that triggers an alert in five minutes is an inconvenience. The same schema drift that goes undetected for 72 hours is a data-remediation project. The same drift undetected for a week becomes a multi-quarter trust problem with the customer who relied on the data.

Most monitoring systems are tuned to catch infrastructure failures (CPU, memory, HTTP 5xx, latency spikes). Almost none are tuned to catch correctness failures. The correctness check requires either an external source of truth or a structurally separate verification pass. Both cost time. Both pay back the first time they prevent a silent failure from reaching a customer.

Sources: HST Solutions ("Common Causes of Data Flow Failures," 2026); Seattle Data Guy ("5 Silent Failures in Data Pipelines," April 2026); McKinsey data-quality research, ongoing.

Run a silent-failure audit on a sample of your feed. Send Crawlify a CSV of any source feed you currently rely on. Within five business days you get a defect report scoring each of the five canonical patterns, the estimated detection time at your current monitoring setup, and a worked example of where the failures hide. crawlify.ai/audit.

What an architecture that catches silent failures actually looks like

Detection costs more than monitoring because detection requires comparing the pipeline output to something the pipeline did not generate. Four layers do the work, each catching a different class of failure:

Layer 1 — Schema contracts at extraction

Every source has a typed schema declared in code. Field names, types, ranges, enumerated values, formats (ISO dates, ISO currency codes, ISO country codes). Anything that arrives outside the contract is flagged before it is written. This single layer catches more than half of silent failures because schema drift is the most common pattern.

Layer 2 — Volumetric and row-count baselines

Every source has a rolling baseline for row count, field-presence rate, and value distributions. A batch that delivers 30 percent fewer rows than yesterday, or where a price field is suddenly 40 percent null when it was 0.5 percent null last week, triggers a review. The threshold is not absolute — it is relative to the source's own history.

Layer 3 — Targeted human verification on every batch

We do not have a person look at every row. We have a person look at every row that matters: high-impact fields (price, currency, dates, job status, event status), low-confidence model outputs, schema-edge values, and a random stratified sample for accuracy measurement. The reviewer sees the source page snapshot and the extracted row side by side. Their call becomes the ground truth that feeds the accuracy number.

Layer 4 — Same-source cross-checking

For sources where it's possible (employer ATS endpoints for job feeds, retailer-side APIs for pricing), the verification pass cross-references the source itself. A listing missing from the source ATS today is removed from the feed today, not next week. This is the layer that closes the ghost-job problem from Blog 2.

The 99.5% verified field-accuracy SLA is the contractual output of this four-layer architecture. It is calculated on a stratified weekly sample, the methodology is in the pilot agreement, and there is a remediation clause if a month underperforms.

Why in-house teams discover silent failures last

A point from Blog 3 in this series rhymes here. An in-house data team builds the pipeline they need. They build the monitoring that catches the failures they've already lived through. The silent-failure architecture above takes 6 to 12 months of post-incident learning to build in-house — every layer corresponds to a failure mode that someone's on-call already paid the cost of discovering.

A managed pipeline absorbs that learning. The verification architecture is what the customer is buying. The in-house equivalent is the same architecture built incident-by-incident, paid for in customer complaints.

Five questions to ask your current pipeline this week

1. What is your detection time on a silent schema drift, measured from the upstream change to your alert firing? If you don't know the number, the honest answer is days.

2. What baseline do your row-count alerts compare against — absolute thresholds, or per-source rolling averages?

3. How would you find out a field that used to always be present is suddenly 30 percent null this week?

4. Who is performing the targeted verification on high-impact fields — a person, a contract test, or nobody?

5. What is your remediation playbook when a silent failure is caught at hour 36? At day 6? At day 14?

The honest answers determine the cost shape of the next silent failure, which is happening now in someone's pipeline as you read this.

Closing the month

This is the fourth and last post in a single-month sequence. The thesis across all four was the same: accuracy in web data is not what the category measures by default, and the cost of that gap shows up in places the category does not measure either. Uptime SLAs don't catch silent failures. Self-serve infrastructure doesn't catch them. In-house teams catch them eventually, expensively. A managed pipeline with a published, methodology-backed field-accuracy number is what catches them on the same batch.

That number, for Crawlify, is 99.5 percent. It is calculated on a stratified weekly sample, the methodology is in the pilot agreement, and there is a remediation clause if a month underperforms. It exists because the four failure patterns in this post are real, common, and very rarely measured by anyone with skin in the game.

A month of posts. One number. If you run a pricing tool, a job board, an events platform, or an alt-data desk and you cannot tell me your current field-accuracy number, that is the conversation worth having. Free 5-day pipeline audit on your own data. crawlify.ai/audit · hello@crawlifyai.com.