Data Layer

Versioning & Lineage

Immutable snapshots. Full lineage. Training-ready export.

What It Does

The versioning engine captures labelled datasets into immutable state snapshots, establishing a deterministic, point-in-time record of source data, labels, validation metrics, and upstream transformations. By embedding end-to-end data lineage directly into each version, every artifact remains permanently audit-ready and tamper-proof. Once committed, versions are read-only and can exclusively be exported, diffed against historical baselines, or referenced as the foundational branch for new dataset iterations.

v1.0
Major
v2.0
Schema changes, complete re-processing.
Minor
v1.1
New labels, corrections applied.
Patch
v1.0.1
Small fixes, minor issues resolved.
Versioning Model

Versions follow semantic numbering: major, minor, patch.

Each version records its parent version, the changes made, and the reason for the bump. Versions can branch to explore different labeling strategies on the same base data without touching the main line.

Lineage

Every version generates a lineage report one structured document, nine sections, auto-included with every export.

Identity
Dataset name, version number, SHA-256 hash, creation timestamp.
Source Provenance
Where the data originated, how it was imported, original format and row count.
Transformation Chain
Every cleaning and validation step applied, with before-and-after row counts.
Labeling History
Engine used, confidence threshold, auto-label vs. review split, correction rate.
Version Changes
What differs from the previous version rows added, removed, labels changed.
Quality Summary
Verified agreement, calibration error, class distribution, readiness score.
Lineage Path
The full chain from source to export, traced as one continuous path.
Export Evidence
Where the data was sent, file checksums, export timestamp.
Risk Assessment
Quality warnings, imbalance flags, or coverage gaps, surfaced before training.
Resolutions

What versioning actually resolves, six problems eliminated by making every dataset a permanent, addressable record.

01
Reproducibility
Rerun any experiment against the exact dataset version that produced it bit-for-bit, every time.
02
Traceability
See exactly how the data was collected, cleaned, and labelled, the full path from raw signal to label.
03
Collaboration
Distributed teams branch and work the same base dataset without ever overwriting each other's progress.
04
Auditability
A permanent, unforgeable record of who changed what, and why, for every version that ever existed.
05
Resolvability
Trace how a change ripples forward to see every downstream dataset, model, and report a single upstream edit would touch before it happens.
06
Optimisation
Map how data actually moves through the organisation, surfacing duplicate pipelines, outdated processes, and where resources are being spent twice.

Get data infrastructure for training AI models

Book a Demo