The failure is universal
Roughly 91% of machine learning models degrade over time. This is not a defect rate — it is the expected behaviour of a static function applied to a non-static world.
The root cause is almost always a data mismatch. The model has not changed. The data flowing into it has.
What makes this expensive is not the degradation itself but the diagnostic gap that follows it. Data science teams spend excessive time manually investigating performance degradation rather than systematically addressing root causes. The typical sequence is: a metric falls, someone notices, someone else is asked to investigate, and three weeks later the answer is a subset of training data that was mislabeled or unrepresentative.
Observability exists to close that gap. Most implementations only close half of it.
Monitoring versus observability
The distinction is not semantic.
Monitoring answers: is the system behaving within expected bounds? It tracks known metrics against known thresholds and alerts when they are breached. Latency, throughput, error rate, aggregate accuracy.
Observability answers: given unexpected behaviour, can I determine why? It requires enough instrumented context to interrogate a failure that nobody anticipated.
Applied to models, monitoring tells you accuracy fell from 94% to 87%. Observability tells you the fall is concentrated in a specific input segment, that the segment's share of production traffic tripled in the last month, and that the corresponding region of the training distribution was thinly covered.
The second is actionable. The first is an alarm.
Three distinct drift types
Conflating these leads to the wrong remediation.
A complete observability layer computes all three, because each catches a failure mode the others miss.
Aggregate metrics hide the failures that matter
A single accuracy number is the most misleading measurement in production ML.
Consider a defect classifier at 94% aggregate accuracy. Decomposed by class: Class A, 78% of volume, 98% accurate. Class B, 18% of volume, 91% accurate. Class C, 4% of volume, 61% accurate.
The aggregate looks healthy. Class C is failing badly, and Class C is very likely the rare, high-consequence category — the one the system exists to catch.
Slice analysis decomposes performance across categories, input segments, time windows, and metadata dimensions. Without it, the failures that matter most are the ones most effectively concealed by the reporting.
The corollary is that observability must operate at the level of individual predictions, not aggregate statistics. You cannot decompose what you did not retain.
Detection without remediation is an incomplete loop
This is where most observability tooling stops.
The typical stack detects drift, raises an alert, and displays a chart. What follows is manual: someone exports predictions, investigates by hand, forms a hypothesis about which training data is responsible, assembles a corrective dataset, arranges for it to be labeled, and eventually retrains.
That sequence takes weeks and is entirely undocumented. It is also disconnected — machine learning production issues often stem from organisational disconnects between data science teams building models and engineering teams maintaining production systems, where data scientists lack visibility into production data characteristics.
Retraining without diagnosis compounds the problem. Retraining on more of the same distribution that caused the failure does not fix the failure; it entrenches it.
The loop is only closed when a drift signal can be traced to specific training records, those records can be corrected, and the correction can be versioned and returned to training as a documented change.
What a complete observability layer requires
- Prediction retention. Individual input-output pairs with timestamps and confidence, not aggregate counters.
- Ground truth linkage. Where available, actual outcomes joined to predictions. Where unavailable, confidence and prediction-distribution proxies.
- Three-way drift computation. Data, concept, and prediction drift, computed independently on a regular cadence.
- Slice decomposition. Performance by class, segment, and metadata dimension, with health thresholds per slice.
- Confidence calibration in production. Whether stated confidence still corresponds to observed accuracy, which frequently degrades before accuracy does.
- Traceability to training data. The ability to move from a failing production slice to the specific training records that shaped that slice's behaviour.
- A remediation path. A mechanism to correct those records and return them to training.
The last two are what separate an observability product from an observability feature.
How Concave AI approaches this
Datalier's observability layer is connected to the labeling and versioning layers by design, which makes remediation a platform operation rather than a manual project.
Prediction ingestion, three paths. Upload a prediction log file with automatic format detection, import directly from a connected storage backend using the same connection already configured for data import, or stream continuously via a webhook SDK snippet inserted into the inference path.
Three-way drift detection. Data drift computed per feature via PSI with drifted features identified individually. Concept drift computed by comparing historical to recent accuracy across the metrics history. Prediction drift computed via PSI on output distribution. Each is reported with severity and history, so trajectory is visible rather than a point measurement.
Slice analysis. Performance decomposed by category with per-slice accuracy and a health verdict, sorted worst-first. The failing segment surfaces immediately instead of being averaged into an aggregate.
Automated re-labeling with strategy selection. Once failing slices are identified, the platform assembles a targeted correction dataset — failing slices, low confidence, recent errors, or drift-affected records. The output is not a random sample; it is the specific records most likely to improve model performance when corrected.
Return to the pipeline. The correction dataset routes through the labeling layer, receives AI annotation and human verification, is frozen as a new version with a full lineage report, and exports in the training format the customer's framework expects.
Detection identifies the problem. The loop is what fixes it.
Conclusion
Ninety-one percent of models degrading in production is not a statistic teams can dashboard their way out of. It requires observability that goes past the alert — one that can name the specific slice failing, trace it back to the training records responsible, and route a correction back into the pipeline without three weeks of manual archaeology in between. Detection without that loop is just a more sophisticated way of watching things break.
