Quality Standards

Numbers, not claims.

Every competitor claims "98% accuracy." We publish Cohen's kappa scores, gold standard pass rates, and batch error logs on every single delivery. Here is exactly what those numbers mean, how we guarantee them, and what happens if we fall below threshold.

Request Free Audit → See QA Architecture
≥ 0.72
Cohen's kappa inter-annotator agreement guaranteed on every delivered batch
≥ 88%
Gold standard task accuracy across all annotators on each project
3
Independent QA tiers applied to every annotation batch before delivery
100%
Of deliveries include a full QA report and data card standard, always
Scroll
Cohen's Kappa ≥ 0.723-Tier QAGold Standard InjectionRLAIF Pre-scoringSycophancy DetectionData Card DeliveryAES-256 EncryptionPrivacy FirstCohen's Kappa ≥ 0.723-Tier QAGold Standard InjectionRLAIF Pre-scoringSycophancy DetectionData Card Delivery
Inter-Annotator Agreement

Why Cohen's kappa is the only honest quality metric

When two annotators independently agree that Response A is better than Response B, that agreement has a statistical weight. Cohen's kappa measures genuine agreement beyond what random chance alone would predict. A kappa of 0.0 means two annotators agree exactly as often as random clicking would. A kappa of 1.0 means perfect agreement on every task.

This is the gold standard for annotation quality measurement in academic NLP research and production AI systems worldwide. Most training data providers don't publish it because publishing requires actually measuring it consistently, which requires automated QA systems which we do.

On every Concave AI project, you receive a kappa breakdown by annotator pair, by task category, and as an overall project average. These numbers are computed automatically from pipeline export data via Python scripts. If our kappa drops below 0.70 on any batch, work pauses and we recalibrate before continuing. We have never delivered below our threshold.

What kappa below 0.60 means for your model: inconsistent preference data teaches your reward model to fit noise rather than signal. The model learns a pattern that reflects annotator disagreement, not human preference. This is how sycophancy, reward hacking, and unstable model behaviour emerge from training pipelines not from the training algorithm, but from the annotation data fed into it.

0.20
Slight effectively random
Annotators agree no more than chance predicts your model learns noise disguised as signal. Useless for RLHF.
0.20–0.40
Fair - poor guideline compliance
Significant disagreement caused by vague guidelines or domain mismatches being inadequate for RLHF or GenAI alignment data.
0.40–0.60
Moderate - risky for alignment
Acceptable for simple binary classification, risky for RLHF preference ranking as inconsistency here teaches reward models to fit disagreement, not preference.
0.60–0.80
★ Substantial - our minimum threshold (we guarantee ≥ 0.70)
Production-grade quality with the standard used by academic NLP research and recommended for RLHF, SFT, and GenAI data. Typical Concave AI deliveries: 0.72–0.79; below 0.70 triggers recalibration.
0.80–1.00
Near-perfect - treat with skepticism
Achievable only on simple, unambiguous tasks with any vendor claiming 0.85+ on complex judgment tasks like RLHF preference ranking is almost certainly inflating by aggregating easy tasks. Ask for the per-category breakdown.
Quality Standards
Quality Standards

Metrics you can verify yourself

Every delivery ships with a QA report: Cohen's kappa scores, gold standard accuracy per annotator, batch error rates, and a data card.

Get a Free Audit →
QA Architecture

Three tiers, run on every single batch

Most training data providers run one quality pass at the end. We run three independently throughout the project. Each tier is designed to catch failure modes the others would miss at the stage where intervention is cheapest.

Tier 1 · Automated · Nightly
Statistical anomaly detection
Automated scripts run every evening on all completed annotation, flagging violations before any human reviewer sees the data. Catches ~60% of quality problems at zero human cost, in time to fix before the next working day.
Response bias - option A selected >68% → position bias, not genuine preference
Speed anomaly - <30% of median completion time → skimming, not reading
Uniformity anomaly - variance too low across a batch → mechanical clicking
Gold standard failure - accuracy below 80% threshold → evidence of quality drift
Reasoning quality - text field under 15 characters → placeholder response
Tier 2 · Peer Review · 10–15% sample
Double-annotation + kappa recompute
10–15% of completed tasks are independently double-reviewed, blind to the first pass. Disagreements are logged and adjudicated by the senior reviewer; kappa below 0.65 on any batch triggers recalibration before work continues.
Category-level bias - disagrees on "safety" but agrees on "helpfulness" → domain gap, not general failure
Guideline misinterpretation - different readings of a criterion → clarification, not removal
Label boundary drift - NLP entity spans gradually widening or narrowing across a batch
Tier 3 · Expert Audit · 5% + all flags
ML-engineer personal review
Senior ML engineer reviews every flag from Tiers 1 and 2, every gold standard failure, and a random 5% sample from across the full batch. Every decision is documented for improving guidelines for the next project.
Sycophancy in RLHF data - "winning" response validates a false premise → most damaging single error type
Training signal quality - will this data produce the desired model behaviour, not just correct labels?
Hallucination edge cases - claims technically accurate but misleadingly incomplete
Guideline gaps - recurring edge cases not covered → triggers update before next batch
Gold Standard Tasks

Your real-time quality signal, running continuously

Gold standard tasks are the most reliable quality monitoring mechanism in professional annotation. They catch quality drift within the first 20–30 tasks not after 2,000 have been submitted.

Before any live project begins, we annotate 50–100 tasks ourselves with high confidence tasks where the correct answer is unambiguous and the reasoning is well-documented. These become our test questions. We inject them at a 6% rate, distributed randomly throughout every annotation batch.

Annotators do not know which tasks are gold standard. They annotate everything identically. Because the gold tasks are visually indistinguishable from live tasks, an annotator who would perform well on obvious test questions but poorly on real work cannot game this system.

After each batch completes, an automated script checks every annotator's gold task answers. Below 80% accuracy the annotator is flagged and their recent work is reviewed. Below 70% accuracy tasks are reassigned and the annotator is removed from the project. This happens before any human QA reviewer sees the data.

6%
Gold injection rate across every batch is randomised, not clustered
80%
Minimum gold accuracy required to continue, below this triggers review
50+
Gold tasks in our library per project type are rotated to prevent memorisation
Auto
Evaluated by script not manually, removing human judgement from the check
Annotator's task queue gold injected at positions 004, 008, 011...
001RLHF pair: Python data processing queryLive
003RLHF pair: Creative writing feedbackLive
004RLHF pair: Legal question sycophancy trapGold ★
005RLHF pair: Financial calculation explanationLive
006RLHF pair: Code debugging walkthroughLive
007RLHF pair: Science concept explanationLive
008NLP entity task: Business news paragraphGold ★
009RLHF pair: Customer service scenarioLive
010RLHF pair: Ethical dilemma responseLive
011RLHF pair: Hallucination detection taskGold ★
Gold task positions stored only in QA system never visible to annotators
Correct answers set by ML engineer before project begins · Evaluated automatically after each batch
The Full Pipeline

Quality enforced at every step, not just the end

Quality is not a final check before delivery. It is enforced at every step of the pipeline from the moment customer data arrives on our servers to the model benchmark follow-up two weeks after delivery. Each step in the pipeline generates a log entry included in your delivery documentation.

The RLAIF pre-scorer reduces human error by flagging uncertain cases for extra human scrutiny the cases that would most likely produce annotator disagreement are the ones that get the most careful human attention. Gold standard injection catches quality drift in real time. Three-tier QA runs concurrently with annotation so problems are caught mid-project, not post-delivery.

The model benchmark follow-up on Day 14 is the most important quality step and the one most training data providers skip entirely. It is the only quality signal that actually measures what matters: did the data improve the model? Everything else kappa scores, gold accuracy, batch error rates are leading indicators. Model improvement is the actual outcome you paid for.

Project quality pipelin every step logged
📥Customer data to isolated, encrypted S3 bucketAES-256
↓
📝Project-specific guidelines written + annotators calibrated to κ ≥ 0.70Calibrated
↓
🥇Gold standard tasks injected at 6% rate are randomised positionsGold In
↓
🤖RLAIF pre-scorer evaluates all tasks with clear cases pre-labeledAI Pre-label
↓
👤Domain-expert annotators validate, correct, and annotate edge casesHuman
↓
📊Tier 1: Automated anomaly detection runs nightly on all batchesAuto-QA
↓
🔄Tier 2: Peer double-annotation on 10–15% sample · kappa recomputedPeer QA
↓
🔍Tier 3: ML-engineer expert audit is 5% random + all flags reviewedExpert QA
↓
📦Dataset + QA Report + Data Card delivered via encrypted transferDelivered
↓
📈Day 14 model benchmark follow-up · result used to improve next batchFeedback Loop
The Data Card

Every delivery comes with a full data card

A data card is a structured metadata document that describes the dataset you receive its composition, the quality metrics achieved, the annotator demographics, the known limitations, and the guidelines version used. Think of it as the nutritional label for your training data.

Most annotation providers deliver a data file and an invoice. We deliver a data file, a QA report, and a data card on every project, as standard. The data card is what your ML team uses to understand exactly what they trained on and why the model behaves the way it does.

It is also what you show investors during due diligence, enterprise customers during procurement, and auditors when they ask how your AI system was trained. The "Known Limitations" section is deliberately honest if one task category had lower agreement than others, we tell you exactly which one and why. Hiding limitations helps no one; disclosing them lets you factor them into your training strategy.

Data Card - Project #CA-2026-047
Hindi RLHF Preference Data · Concave AI · Delivered 14 April 2026
Dataset Statistics
Total preference pairs2,000
LanguageHindi - Devanagari script
Domain distributionTechnical 40% · General 35% · Safety 25%
Reasoning field includedYes - all 2,000 pairs
Quality Metrics
Cohen's kappa - overall0.74
Cohen's kappa - technical tasks0.77
Cohen's kappa - safety tasks0.71
Gold standard accuracy91%
Flagged and resolved pairs47 of 2,000 (2.35%)
3-tier QA appliedAll tiers · full documentation
Annotator Profile
Annotators on project8
Native Hindi speakers8 / 8
Calibration kappa (pre-project)0.72 - all 8 annotators
Known Limitations
Lower agreement categoryAbstract reasoning κ = 0.67
ReasonEvaluating reasoning quality is inherently more subjective disclosed, not hidden
Guidelines versionHindi RLHF v2.3 · 14 Apr 2026

See our standards on your own data

We evaluate 50 of your model outputs or RLHF pairs and return a kappa baseline, sycophancy susceptibility score, and findings report in 5 working days. No cost. No commitment. No sales call required to start.

Request Free Audit → See Our Process