Fine-tuning gets blamed for a lot of problems that actually start in the dataset. A model trained on duplicated, inconsistently formatted or quietly mislabelled examples will reproduce those flaws faithfully, and usually with more confidence than the source material deserved. The good news is that dataset preparation is mostly unglamorous, deterministic work. You can inspect it, measure it and improve it without paying for another training run.
This is the order we work in when preparing a fine-tuning corpus. It runs from the cheapest fixes to the most expensive, so effort is always spent where it still changes the outcome.
1. Collect everything in one place
Before cleaning, assemble every source you intend to use: support tickets, documentation, spreadsheets, PDFs, chat logs and internal wikis. Record where each file came from and who owns it. That provenance list is what lets you remove a source later without re-auditing the whole corpus.
- Convert every source into a single plain-text or JSONL format.
- Keep the original file alongside the converted version.
- Note the collection date, because stale sources are a common cause of wrong answers.
2. Remove duplicates, including near-duplicates
Exact duplicates are easy to find; near-duplicates are what damage a model. The same question asked five slightly different ways teaches the model that one answer is enough for all five, which flattens exactly the variety you wanted to capture.
- Drop exact matches on a normalised key such as lowercased text with collapsed whitespace.
- Use shingle overlap or embedding similarity to cluster the near-duplicates that remain.
- Keep the best-written instance of each cluster rather than the first one you happened to find.
3. Normalise the formatting
Inconsistent structure is the second most common defect. If half your examples are conversational and half are instruction-following, the model learns a blurry average of both. Pick one schema and enforce it everywhere.
- Define a single JSON shape for every record and validate it programmatically.
- Strip navigation text, page numbers, headers and footers introduced by extraction.
- Standardise dates, units and currency formats across the corpus.
- Remove template placeholders and instructional phrases that leaked in from prompt text.
4. Check the answers, not just the questions
Inputs are usually clean because people read them. Outputs are where silent errors hide: truncated answers, answers that contradict the source document, or a confidently wrong figure copied from an outdated version of a file.
- Flag every answer that falls under a minimum length for its question type.
- Sample-check answers against the source document rather than against memory.
- Remove examples where the supposed correct answer is really an opinion or a guess.
5. Balance the distribution
A corpus that is ninety percent one topic will produce a model that is ninety percent one topic. Count records by intent and category before training, then compare that count with how you expect the model to be used in production.
- Report counts per intent, per language and per source.
- Cap any single source so it cannot dominate the distribution.
- Add a small set of deliberately hard edge cases that your users will hit.
6. Split before you tune anything
The most damaging mistake in the whole process is splitting after cleaning decisions have been tuned on the full corpus. Any choice you make by looking at the test set has already leaked into the model, and the resulting evaluation numbers will flatter you.
- Hold out validation and test sets first, then clean only the training portion by hand.
- Split by document or by customer, never by random row, so near-duplicates cannot cross the boundary.
- Keep the test set untouched until final evaluation.
7. Run the human review pass
Automated checks catch structure; a short human review catches meaning. Budget for a reviewer to read a random sample of a few hundred records with no context other than the source document, and turn every defect they find into a new automated check.
What good enough looks like
You are ready to train when the schema validates without exceptions, duplicates sit below a small percentage, the intent distribution matches your expected traffic, and a human reviewer cannot find a wrong answer in a random sample. That is a far higher bar than most teams apply, and it is usually the difference between a pilot that stalls and a model you can actually ship.
