Every AI project that learns from examples eventually hits the same wall: somebody has to label the data. It is unglamorous work, it scales badly, and it is usually the largest hidden line item in a training budget. The good news is that most teams overspend on labelling not because annotation is inherently expensive, but because of fixable process problems. Ambiguous instructions, redundant reviews and blind spot-checks quietly multiply the bill. Here is how to get accurate labels for a fraction of the usual spend.
Start with the cheapest fix: sharper guidelines
Before you negotiate annotation rates or hire more reviewers, look at your instructions. In most projects the biggest cost driver is rework, and most rework traces back to guidelines that two reasonable people can read differently. A strong guideline document includes a worked example for every label, a short list of the edge cases that actually appear in your data, and an explicit tie-breaker rule for anything ambiguous — for example, "when a review mentions both shipping and product quality, assign the topic the customer spent most words on."
Run a pilot on 50 to 100 examples and measure how often independent labellers disagree. If disagreement lands above roughly ten percent, revise the guidelines before scaling up. One day spent sharpening instructions routinely saves weeks of correction work later, which makes it the highest-return investment in the whole pipeline.
Match the method to the question
Not every label needs a human starting from scratch. Some labels are already sitting in your systems: cancelled subscriptions tell you about churn, support ticket tags tell you about issue categories, click-through logs tell you about relevance. Deriving labels from behaviour is the cheapest annotation there is, so always ask what your logs already know before commissioning new labelling.
- Simple judgements — binary or small-choice tasks tolerate a large, well-instructed pool at low rates.
- Specialist judgements — domain expertise is expensive, so reserve it for the disputed cases, not the easy ones.
- Consensus by design — one labeller on easy items, two on ambiguous ones, with disagreements escalated to a domain owner.
That design concentrates your most expensive people on the hardest five percent of examples instead of the easiest ninety-five.
Sample your quality checks instead of reviewing everything
Reviewing every label doubles your cost and adds little beyond what sampling tells you. A better default: inspect a random sample from every batch, set an acceptance threshold, and send whole batches below it back with targeted feedback. Track inter-annotator agreement with a simple overlap measure or Cohen's kappa, and watch it per class, not just in aggregate.
Two tricks keep quality high without full review. Golden questions — known-answer items seeded invisibly into the queue — tell you when attention drifts. And per-class disagreement analysis often reveals that one genuinely ambiguous category drives half of all disputes; merging or splitting that category is far cheaper than adding another layer of reviewers.
Let models handle the boring parts
Model-assisted labelling is now standard for a reason. A pre-labelling model drafts its best guess, and humans correct instead of creating from nothing, which is routinely two to four times faster on repetitive tasks. Active learning goes further: label only the examples the model is least confident about, and skip the ones it already gets right. On a well-behaved classification task this can cut the required label count dramatically.
One caution: pre-labels anchor people. Reviewers rubber-stamp a plausible suggestion more often than they should, so audit a sample of agreed pre-labels, rotate which items receive model suggestions, and keep a purely human-labelled holdout to detect drift.
Measure cost per usable label, not cost per click
Billing by items submitted rewards speed over correctness. The number that matters is cost per label that survives quality assurance. Track rework rate as a first-class metric, publish it per batch, and reward the teams that reduce it. When rework concentrates on one category or one guideline section, fix the cause at the source instead of paying for another review pass.
Budget discipline in labelling is mostly a process problem. Sharper guidelines prevent disputes, sensible workforce design concentrates expertise where it counts, sampled quality checks catch problems early, and model assistance removes the most repetitive work. Teams that adopt these habits typically cut annotation spend in half while improving label quality — no rate negotiation required.
