Annotation Quality Frameworks
The frameworks that make labels trustworthy — multi-layer QA, native-linguist review, the seven transcription standards and inter-annotator review.
This page documents Cognegica's annotation quality frameworks: the multi-layer QA process, native-linguist review, the seven transcription standards, and inter-annotator review. Together they turn raw labelling into data a model can trust — with disagreements adjudicated rather than averaged, and quality reported per batch against an agreed bar.
Why frameworks, not vibes
Quality is a process, not a final inspection
Quality that's checked only at the end is quality you can't defend. Our frameworks build review into every stage — calibration before work starts, two passes during, native-linguist review against documented standards, and adjudication of disagreement. The result is labels with a recorded chain of decisions, not an opaque number.
The seven transcription standards
Transcription work follows a defined set of seven standards covering verbatim conventions, speaker labelling, timestamping, handling of disfluencies and non-speech, code-switching, and formatting — so two transcribers on the same audio produce comparable output.
Native-linguist review
A native linguist reviews against the guidelines for each language, catching the dialect, register and cultural nuances that a generic checker cannot. This is the layer that makes low-resource work trustworthy.
Inter-annotator review
We measure agreement between annotators and adjudicate disagreement, reporting the agreed quality bar per batch. The target bar is set with you per project — figures here are editable placeholders, not claims.
The QA layers
Multi-layer QA framework
Each layer has an owner and a purpose. Quantitative targets are set per project and shown here as editable placeholders.
| Layer | Owner | Purpose | Target (placeholder) |
|---|---|---|---|
| 1. Guideline calibration | Lead linguist | Align annotators on the spec before work starts | Calibration sign-off |
| 2. First-pass labelling | Annotator | Produce labels against the seven standards / guidelines | Per-task SOP adherence |
| 3. Second-pass review | Reviewer | Independent review of first-pass output | <set per project> |
| 4. Native-linguist QA | Native linguist | Dialect, register and cultural-nuance check | <set per project> |
| 5. Adjudication | Lead linguist | Resolve inter-annotator disagreement | Agreed quality bar per batch |
Representative QA layers. Inter-annotator agreement targets and pass rates are scoped per project and available under NDA — no fixed accuracy figures are claimed here.
About the frameworks
Questions about annotation quality
- What is multi-layer QA?
Quality is built into every stage — calibration, two-pass labelling, native-linguist review and adjudication — rather than a single final inspection.
- Do you report inter-annotator agreement?
Yes. We measure agreement and adjudicate disagreement, reporting the agreed quality bar per batch. The target bar is set with you per project.
- What are the seven transcription standards?
A defined set covering verbatim conventions, speaker labelling, timestamping, disfluencies and non-speech, code-switching and formatting — so output is comparable across transcribers.
Know which labels you can trust.
Set the quality bar with us and we'll run your data through multi-layer QA.