Most teams hit the same wall with large language models: the model sounds confident, but it knows nothing about your products, policies, or documents. Retrieval-augmented generation (RAG) is the practical fix. Instead of retraining a model, you let it look things up. At question time the system searches your own content, hands the most relevant passages to the model, and asks it to answer using only those. The result is an assistant that stays current, cites its sources, and costs a fraction of a full fine-tune. Here is how RAG works, and how to build your first pipeline without overcomplicating it.
How RAG works in plain terms
RAG has two halves that run on different schedules. Indexing happens offline: you collect documents, clean them, split them into chunks, convert each chunk into an embedding — a list of numbers that captures its meaning — and store those vectors in a vector database next to metadata such as source, title, and section. Answering happens per question: the question is embedded the same way, the database returns the chunks whose vectors sit closest in meaning, and those passages are placed into the model's prompt as the ground truth to answer from. Because embeddings group similar meanings together, a question about password resets lands near the help page that explains them, even when the wording differs. No model weights change at any point in this process — that is what separates RAG from fine-tuning.
Building your first pipeline, step by step
Resist the urge to index everything at once. Pick one narrow domain — a single help centre, product manual, or policy set — and get that answering well before widening the net. You can assemble the whole thing in a day with a few hundred documents. A first pipeline looks like this:
- Collect and clean the documents. Strip navigation, headers, and duplicates; broken text produces broken answers.
- Split content into chunks of roughly 300 to 500 tokens with a little overlap, cutting at headings and paragraphs rather than mid-sentence.
- Generate an embedding for every chunk and store it in a vector database such as pgvector or a lightweight embedded store, keeping source, title, and section as metadata.
- At question time, retrieve the five to ten closest chunks. Hybrid search — combining keyword matching with vector similarity — is noticeably more robust than vectors alone.
- Prompt the model to answer only from the retrieved context and to say so when the documents do not contain the answer.
- Show citations. Linking every answer back to its source chunk is the fastest way to earn trust and to spot bad retrievals.
- Log real questions together with the chunks retrieved for them. Thirty to fifty logged questions become your evaluation set.
The details that make or break answer quality
The model gets the credit and the blame, but retrieval quality decides most outcomes. Structure-aware chunking beats blind cutting; metadata filters for product, section, or date keep irrelevant material out of the prompt; and a reranking pass over the top candidates often lifts precision more than any prompt tweak. Keep answers grounded: if the retrieved passages do not support a reply, the assistant should say so instead of guessing. Track two numbers from day one — how often answers are fully supported by their citations, and how often the assistant correctly declines. Finally, plan for freshness and re-index whenever a source document changes, because a RAG system is only as current as its last index update.
Beginner mistakes to avoid
Nearly every rocky first RAG project traces back to a handful of habits:
- Cutting chunks mid-sentence, which shreds the context the model needs.
- Skipping metadata, which makes filtering and citations painful later.
- Stuffing too many chunks into the prompt and burying the useful one.
- Leaving out a refusal path, so the model invents an answer for missing information.
- Never testing with the questions real users actually ask.
- Treating the pipeline as finished — retrieval quality drifts as your content changes.
When RAG is the right tool
RAG shines when answers must reflect documents that change: policies, manuals, knowledge bases, internal wikis. It is the wrong tool for teaching a model a new style or output format — that is fine-tuning territory, and the two approaches can complement each other. Start small: one source, one week, thirty test questions. When citation quality holds and users stop asking whether they can trust the answers, widen the corpus. That steady expansion — not a big-bang launch — is how retrieval-augmented generation earns its place in production.
