RAG: answers from your own documents

A model knows the public internet up to its cutoff and nothing about your wiki. RAG fixes that without training: split documents into chunks, embed them into vectors, and at query time retrieve the few chunks closest to the question and paste them into the prompt.

It wins when answers must cite internal sources and the corpus changes faster than you could ever fine-tune. It loses when the task needs new behavior rather than new facts — then train or write skills instead. Start with chunk size around 500 tokens and overlap, measure answer correctness, and only then touch anything else.

How RAG fails

Three classic failures. Wrong chunks retrieved: fix with better chunking and metadata filters before touching the model. Right chunks, wrong answer: the prompt buries them — put retrieved text first and instruct the model to cite it. Stale index: the documents moved on and the vectors did not — re-embed on a schedule, not on complaints.

Evaluate with twenty fixed questions and human-graded answers. If you cannot tell whether a change helped, you are decorating, not engineering.

Costs people forget

Embeddings cost money or GPU time on every reindex, vector storage grows with the corpus, and each query pays for the retrieved context twice (embedding the question, then reading the chunks). Cap top-k at five to start, cache frequent questions, and keep an eye on the token bill for a month before declaring victory.