The failure that does not announce itself
Traditional software fails loudly. A service crashes, errors spike, pages stop loading, someone gets paged. The feedback loop between failure and awareness is measured in seconds.
Machine learning does not work that way. A model keeps running as long as it can process the input it receives — even if that input is biased, incorrect, or unlike anything it was trained on. It returns a prediction. The prediction is confidently wrong. Nothing crashes. No alert fires.
Roughly 91% of ML models degrade over time, and the mechanism is almost always the same: the model did not break in any software sense. The data around it changed, and the model kept running on assumptions that were no longer true.
The industry term is silent failure, and it is the dominant failure mode in production ML.
A worked example
Consider a credit card fraud model trained on pre-2020 spending patterns. It learned that rapid shifts to online merchants, cross-category spending spikes, and unfamiliar vendors were strong fraud signals.
Then consumer behaviour changed. E-commerce volumes surged, new merchants appeared overnight, and previously rare behaviours became ordinary.
Watch how the failure propagates:
Between stage one and stage three: weeks, sometimes months. Throughout that entire period the model returns confident predictions, latency stays inside SLA, the error rate is zero, and every dashboard is green.
Why APM structurally cannot see this
The reason is architectural, not a gap in configuration.
Application performance monitoring was designed for software that behaves deterministically. A request arrives, follows a defined code path, returns a response. Failures are obvious: a service is up, down, or returning a 500.
That assumption does not hold for ML. As one practitioner put it: a response that is fast, available, and incorrect is byte-for-byte indistinguishable from a correct one to an APM tool.
The tracing shows a clean span. The latency histogram looks excellent. And the model has just approved something it should have flagged.
The framing worth internalising: these are not infrastructure failures. They are decision failures. You cannot monitor the quality of a decision with a stack built to monitor the availability of a service.
One arXiv analysis quantified the gap, finding that traditional APM covers only 8 of 20 reliability factors — roughly 40% — and that the most discriminative signals fall entirely outside its observability scope.
There is a second trap here. AIOps does not close the gap either. It analyses your existing telemetry more intelligently — but it adds no new signal types. It is a smarter way to process insufficient data, not a solution to the insufficiency.
The ground truth delay
Even teams that build proper ML monitoring hit a structural constraint that is rarely designed for.
Most production systems receive ground truth days or weeks after the prediction. A fraud flag is confirmed at chargeback. A defect classification is verified at downstream inspection. A loan decision resolves over months.
If your only degradation signal requires labels, you are structurally blind for the length of your label delay. A thirty-day feedback cycle means a thirty-day blind spot, during which the model may be making thousands of confidently wrong decisions.
This is why label-independent signals matter so much. Input drift and prediction drift require no ground truth at all — they are computed entirely from what the model receives and what it emits. They are weaker evidence than accuracy, but they arrive first.
The practical posture is a joint condition: alert on input drift plus a measurable evaluation drop. Drift without impact is a false alarm that burns on-call attention; accuracy alone arrives too late.
Three drift types, three different responses
Conflating these produces the wrong remedy, which is worse than none because it consumes a retraining cycle and fixes nothing.
Data drift — the input distribution changed. Detected per feature via PSI, with values above 0.2 typically indicating significant drift requiring investigation. For continuous features the Kolmogorov-Smirnov test compares cumulative distributions; for categorical features, chi-square or Jensen-Shannon divergence measures changes in category frequency.
Concept drift — the relationship between input and correct output changed. The inputs may be identical while the right answer has moved. Requires ground truth, and it is the most consequential type.
Prediction drift — the model's output distribution shifted. If a classifier historically predicted a class 12% of the time and now predicts it 31%, something has changed even if accuracy has not visibly fallen. No labels required, and it frequently precedes measurable accuracy loss.
For unstructured data — images, video, audio — feature-level statistics do not exist. The practical approach is embedding drift: compute a centroid for the reference window and the production window, then track cosine distance between them.
Aggregate metrics conceal the failure that matters
A single accuracy number is the most misleading measurement in production ML.
A defect classifier at 94% aggregate, decomposed: Class A, 78% of volume, 98% accurate. Class B, 18% of volume, 91% accurate. Class C, 4% of volume, 61% accurate.
The headline looks healthy. Class C is failing badly — and Class C is almost always the rare, high-consequence category the system exists to catch.
Slice decomposition is not a refinement. Without it, the failures that matter most are the ones most effectively hidden by the reporting.
The problem is about to get structurally worse
Two forces are converging.
Model count is outpacing oversight. Gartner predicts 40% of enterprise applications will embed task-specific AI agents by the end of 2026, up from less than 5% in 2025. Most teams are not adding monitoring capacity at anything like that rate.
Provider updates are invisible. When a foundation model provider ships an update, every enterprise on that API receives it silently. A Stanford study documented significant behavioural changes in GPT-4 across quarterly updates — tasks that worked reliably in one version failing in the next, with no notification and no version pin.
And the regulatory clock is running. Under the EU AI Act, from 2 August 2026 providers and deployers of high-risk AI systems must operate continuous monitoring programmes, track real-world performance, and report serious incidents within strict timeframes. The Act does not distinguish between a system that crashes and one that quietly produces wrong answers.
How Concave AI approaches this
Detection is the easy half. What makes silent failure expensive is that finding the cause takes weeks of manual archaeology.
Three-way drift detection. Data drift via PSI computed per feature, with drifted features named individually. Concept drift by comparing historical against recent accuracy. Prediction drift via PSI on output distribution — label-independent, so it surfaces while ground truth is still in flight.
Diagnosis, not just measurement. The three signals are read together. Low accuracy with a stable distribution means the model never converged — a training data problem, not drift, and relabelling will not fix it. Degraded accuracy with drift means genuine distribution shift. Healthy aggregate with one failing slice means a localised data defect. Each maps to a different action.
Slice decomposition on every evaluation. Performance broken out by category with per-slice health verdicts, sorted worst-first. The failing segment surfaces immediately rather than being averaged away.
Traceability to training records. Because labelling, versioning, and observability sit in one platform, a failing production slice can be traced to the training data that shaped it — which is what turns an alert into a remedy.
Correction that returns to the pipeline. Suspect records are flagged in place within the original dataset, reviewed, and versioned as a correction round with a full lineage delta. No temporary datasets, no severed provenance.
Conclusion
Silent failure is not an exotic edge case. It is the normal way production ML degrades, and the tooling most teams rely on is structurally incapable of seeing it — not misconfigured, incapable.
Green dashboards are not evidence of a working model. They are evidence of a working server.
The distinction between those two things is where the risk lives, and closing it requires monitoring built for decisions rather than availability — connected to the data layer, because the data layer is where the fix has to happen.
