The ratio nobody defends but everybody accepts
Andrew Ng has been making the same observation for years: roughly 80% of an AI developer's time is spent on data preparation, leaving 20% for the work the role was actually created to do.
Nobody argues with the number. Teams quote it in interviews, in postmortems, in conference talks. And then they carry on operating as though it were a fact of nature rather than an engineering failure.
It is not a fact of nature. It is the predictable output of a tooling gap — and it carries a cost that is measurable, compounding, and almost entirely absent from any budget line.
What the 80% is actually spent on
Break it down and the work is startlingly unglamorous.
None of that is machine learning. All of it is done by machine learning engineers.
Four to seven tools, and the cost is in the gaps
The typical enterprise stack for moving one dataset from raw to training-ready spans four to seven separate products: object storage, a transformation or ETL layer, an annotation platform, some form of version control, a quality or validation tool, a monitoring product, and a training framework.
Each is competent in isolation. The cost lives in the handoffs.
Every boundary loses context. The annotation platform knows the confidence score assigned to each label. The training framework never sees it. Information that would be genuinely useful downstream is discarded at the interface because the interface is a file format.
Every boundary is a manual step. Export here, transform, import there. Automated with a script if the team is disciplined, done by hand if they are busy — and they are always busy.
No boundary carries provenance. By the time a dataset reaches training, the record of where it came from and what was done to it exists only in the memory of whoever did it.
This is why the question how was this model trained is so consistently difficult to answer. The information was never lost through carelessness. It was never captured, because nothing in the chain was responsible for capturing it.
The cost is measured, and it is large
Gartner estimates that poor data quality costs organisations an average of $12.9 to $15 million annually. IBM's Institute for Business Value found that 43% of chief operations officers name data quality as their most significant data priority, with more than a quarter of organisations losing over $5 million a year and 7% reporting losses above $25 million.
For AI specifically the failure rate is stark. Gartner has found that at least 30% of generative AI projects are abandoned after proof of concept, with poor data quality among the leading causes. Cloudera's 2026 research found that only 7% of enterprises consider their data entirely AI-ready, while 73% struggle with AI data preparation.
And the reason it stays invisible: roughly 59% of enterprises do not measure data quality at all. The cost is real, ongoing, and unbudgeted, because no line item exists for it.
The compounding version of the problem
A single dataset handled badly is an inconvenience. The structural damage appears at scale, and it takes three forms.
Irreproducibility. A training run that cannot be reproduced makes every subsequent experiment uninterpretable. When model v3 outperforms v2, you cannot tell whether the architecture improved or the data changed. Months of experimentation produce no reliable knowledge.
Undiagnosable failure. When a production model degrades, the investigation runs backwards through the pipeline. Without lineage that path does not exist, so the team retrains on more data and hopes. Sometimes it works. Nobody learns why.
Talent attrition. ML engineers are expensive, in demand, and did not take the job to write conversion scripts. The 80% is not just a productivity cost; it is a retention cost, and it is paid quietly through resignation letters that cite "wanting to work on harder problems."
Why this persists
Three reasons, and none of them are stupidity.
Each tool is individually justified. Nobody chose seven tools. They chose an annotation platform, then later a version control system, then later a monitoring product — each a reasonable decision addressing a real need. The fragmentation is emergent, not designed.
The cost is distributed. No single team owns it. It is spread across every ML engineer's week in two-hour increments, which is exactly the shape of cost that never appears in a budget review.
The alternative did not exist. There has been no unified layer to migrate to. Point solutions were the only option, so point solutions is what everyone built.
That last point is what has changed.
How Concave AI approaches this
Datalier replaces the chain with a single pipeline. Six layers, one system, no handoffs.
Ingest. Connect object storage, data warehouses, and databases — S3, GCS, Azure Blob, Hugging Face Hub, PostgreSQL, Snowflake — or upload directly. Source provenance is recorded on entry and persists through every downstream operation.
Transform. Validation, deduplication, PII detection, and schema normalisation run automatically, each reporting rows before, rows after, and rows affected. Nothing is silently dropped. A composite quality score travels with the dataset.
Label. AI annotation across dozens of types spanning text, image, video, and audio, with confidence-based routing. Items above the threshold are accepted; the rest enter a keyboard-driven review stream at three to five seconds per item. Automation rate, review rate, and correction rate are reported, not estimated.
Version. Immutable, hash-verified snapshots with semantic numbering. Every version carries a quality score, a readiness verdict, and an auto-generated lineage report covering source, transformations, labelling history, and export evidence.
Observe. Production predictions flow back in. Drift is detected across three types, performance is decomposed by slice, and failing segments are identified automatically.
Govern. Provenance, audit trail, access control, and integrity verification are produced as a by-product of the operations themselves — not as a separate documentation exercise that can be skipped.
The 80% does not disappear because engineers work faster. It shrinks because the work between the tools stops existing.
Conclusion
The 80% figure has been repeated so often it has lost its force. It should not have. It describes a profession in which the majority of highly paid specialist time is spent on file management, format conversion, and archaeology.
That was defensible when no alternative existed. It is not defensible now.
The question worth asking is not how do we make data preparation faster. It is why is data preparation a separate activity from the pipeline at all. Fix that, and the 80% becomes something engineers reinvest in the models they were hired to build.
