Classification looks like the simplest job in machine learning — read a text, output a label — and it is exactly where fine-tuning earns its keep. Support-ticket routing, lead scoring, content moderation, intent detection: these are all classification problems, and a model fine-tuned for them routinely beats both keyword rules and general-purpose chat models on speed, cost and consistency. This walkthrough covers the full path from a blank page to a classifier you can put in front of real traffic.
Why fine-tune instead of prompting a large model
A general-purpose LLM can classify text with no training at all, so why bother? Three reasons show up in almost every project. First, cost: a small fine-tuned model serves requests at a fraction of the price of a frontier model. Second, latency: a compact model answers in milliseconds, which matters when classification sits inside a live workflow. Third, consistency: a fine-tuned model has absorbed your label definitions, so it applies them the same way at 3 p.m. and 3 a.m., while a prompted model drifts with phrasing. Prompting is still the right first experiment — but once a category set is stable and volume is real, fine-tuning usually wins.
Step 1: write the label book first
The most important artefact is not the model — it is the document that defines every label. For each category, write what it means, what belongs in it, what looks similar but belongs elsewhere, and two or three real examples. Ambiguity you leave in the label book becomes noise in the training data and arguments in production. If two labels overlap so much that examples could sit in either, merge them or add a decision rule. Ten well-defined labels beat forty fuzzy ones.
Step 2: build the dataset
With labels defined, gather real examples from the source the model will actually see — live tickets, genuine product reviews, authentic chat logs — rather than invented ones. Then assemble the training set:
- Aim for a few hundred examples per class to start; simple problems often converge with less, nuanced ones need more.
- Keep classes roughly balanced, or the model learns to over-predict the popular label.
- Include hard cases deliberately: sarcasm, mixed intent, borderline wording — the examples that made humans hesitate.
- Hold out a validation and test split you never train on, drawn from the same distribution as production traffic.
- Label anything ambiguous a second time and keep the disagreements; they are your quality signal.
Clean, honest data at this stage is worth more than any training trick later. Most disappointing classifiers were not trained badly — they were fed a dataset that quietly disagreed with itself.
Step 3: choose a base model and train
For most classification work you do not need the biggest model you can find. A compact encoder or a small instruction-tuned model fine-tunes faster, costs less to serve and is easier to debug. Start with conservative training choices: a low learning rate, a handful of epochs, and early stopping on validation performance rather than a fixed schedule. Watch the validation loss, not the training loss — the moment validation loss turns upward while training loss keeps falling, the model is memorising instead of learning. If data is scarce, freeze most of the backbone or use parameter-efficient methods such as LoRA, which reach good quality with a fraction of the compute and keep checkpoints small.
Step 4: evaluate like production will judge you
Accuracy alone hides too much. Report a per-class breakdown — precision, recall and F1 — so you can see exactly which labels the model confuses and what it confuses them with. Build a small confusion matrix and read it: systematic pairings, such as billing versus refunds or login versus permissions, tell you where the label book or the dataset needs another pass. Then check the metric that matches the business risk. A moderation model with poor recall lets harmful content through; a routing model with poor precision sends customers to the wrong queue. Decide the acceptable trade-off before deployment, not after an incident.
Step 5: ship with confidence thresholds and a fallback
A classifier in production needs an exit. Calibrate the output probabilities and set a threshold below which the model does not trust itself: route low-confidence cases to a human reviewer or a slower, larger fallback model instead of guessing. Log every prediction with its confidence, sample the low band for review each week, and fold corrections back into the training set for the next version. That review loop is what keeps a classifier sharp as language, products and customers drift.
None of these steps are exotic, and that is the point: fine-tuning for classification is a data project with a training loop attached. Teams that invest in the label book and honest evaluation ship models that hold up under real traffic; teams that rush the data spend their months debugging surprises instead.
