Stop Changing Models

The reflex

When a model underperforms, the reflex in most organisations is the same: try a bigger architecture, tune the hyperparameters, swap in the newest published technique. It feels like progress because it produces something concrete to point to — a new run, a new leaderboard number, a new line in a changelog.

It is also, in most cases, the wrong lever. Andrew Ng has estimated that more than 90% of AI research effort is model-centric — improving code while holding the dataset fixed — despite growing evidence that for the majority of practical problems, the dataset is where the ceiling actually sits.

The evidence

A steel-defect-detection experiment made the comparison directly. Two teams tackled the same underperforming model. One team was restricted to model-centric changes only — architecture search, regularisation, hyperparameter tuning. The other was restricted to data-centric changes only — fixing label inconsistencies, correcting mislabeled examples, improving annotation quality on the existing dataset, with the model held constant.

The model-centric team produced 0% improvement. The data-centric team produced roughly 16% improvement — without touching the architecture at all.

Ng's own research group found a related result on a steel-defect dataset: cleaning label errors in the existing data outperformed collecting 3x more raw data. More data without quality control did not help; less data with fewer errors did.

A separate result on CIFAR-10 reinforces the same point from a different angle: a model trained on a smaller set of correctly labeled images consistently outperformed the same model trained on a larger set that included noisy labels — even though the noisy set had strictly more examples.

Scale has stopped compensating

For years, the default answer to weak model performance was simply more data — the assumption being that scale would statistically average out whatever noise the labels contained.

That assumption is breaking down. As datasets have grown into the billions of examples, curation and filtering have started to outperform raw volume scaling on a growing number of benchmarks. Several recent large-scale training runs have found that aggressively filtered, smaller-but-cleaner datasets beat larger, noisier ones on downstream evaluation.

Uncurated scale increasingly produces diminishing — and in some cases negative — returns, because the marginal example added at scale is more likely to be redundant or mislabeled than genuinely informative.

Three things a data-centric approach requires

Being data-centric is not a slogan — it is three concrete disciplines, each of which most teams currently skip.

Consistency enforced and measured. Labeling consistency has to be actively checked, not assumed. Two annotators should agree on the same example at a measured rate, and that rate should be tracked over time, not eyeballed once during a pilot.

Error analysis as the driver of what to fix next. Rather than retraining blindly after a performance drop, the team examines which specific examples the model gets wrong, looks for patterns in those failures, and directs labeling or cleanup effort at exactly that failure mode.

Data as a versioned asset. Every change to the training set — a relabel, a correction round, a filtering pass — is tracked with the same rigor as a code change, so a performance shift can be attributed to a specific data change rather than treated as unexplained variance.

Concretely, "fix the pipeline" means:

How Concave AI approaches this

Datasets as versioned assets. Every dataset carries full version history — what changed, when, and why — with the same discipline applied to code. A performance change can always be attributed to a specific, named data change.

Quality measured, not asserted. Inter-annotator agreement, label consistency, and coverage gaps are tracked continuously rather than checked once during a pilot phase and then assumed to hold indefinitely.

Label-error detection via Confident Learning. Rather than relying on manual audits to catch mislabeled examples, the platform surfaces likely label errors algorithmically, directing review effort at exactly the examples most likely to be wrong.

Error analysis wired directly into the loop. Slice-level performance breakdowns point straight at the specific examples and categories driving underperformance, so labeling effort is targeted rather than applied uniformly across the dataset.

Lineage that reaches all the way to production. A failure observed in production can be traced back through the pipeline to the specific training data responsible, closing the loop that most teams currently have to reconstruct by hand.

Conclusion

The architecture is very likely not your bottleneck. Ninety percent of the field's effort goes toward the part of the system that, for most practical problems, has the least remaining room to improve.

Stop changing the model. Fix what feeds it.