RAG — retrieval-augmented generation — is one of the most useful and most oversold patterns in applied AI. Strip away the jargon and it is a simple idea: before the model answers, go find the relevant material and put it in front of the model. This guide explains the problem RAG solves, how retrieval actually works, where it shines, where it disappoints, and how to decide between RAG, a bigger context window, and building nothing at all.
The problem RAG solves
A model knows only what it learned during training. It does not know your company's internal wiki, the contents of a PDF you have not shown it, yesterday's numbers, or anything past its training cutoff. Ask about those and it will either refuse or, worse, invent a confident answer. You could paste the relevant document into the prompt every time, but that does not scale when the knowledge is thousands of documents, changes frequently, or is too large to fit. RAG automates the paste: given a question, it retrieves the handful of passages most likely to contain the answer and hands them to the model along with the question.
Embeddings and retrieval, plainly
The trick to finding relevant passages is matching by meaning, not keywords. An embedding model converts a chunk of text into a list of numbers — a vector — positioned so that texts with similar meaning land near each other in that mathematical space. How do I reset my password and steps to recover account access use different words but end up close together. You embed all your documents once and store the vectors in a database. At query time you embed the question the same way and retrieve the stored chunks whose vectors sit nearest to it. That nearest-neighbor search is the retrieval in retrieval-augmented generation.
Those retrieved chunks are then inserted into the prompt — here is the user's question, and here are the passages we found; answer using them — and the model generates its response grounded in that material. A well-built system also passes along where each passage came from, so the answer can cite its sources and you can check them.
Chunking, the unglamorous part that decides everything
You cannot embed a whole 80-page manual as one vector and expect precise retrieval, so documents are split into chunks. How you chunk quietly determines whether RAG works. Chunks that are too large dilute the meaning and drag in irrelevant text; chunks that are too small lose the context that makes a passage make sense. A sentence pulled out of a procedure is useless without the step it belongs to.
Modern systems chunk along a document's natural structure — sections, paragraphs, headings — rather than blindly cutting every N characters. A common pattern indexes small, precise chunks for matching but returns the larger parent section they belong to, so the match is accurate and the model still gets enough surrounding context to answer. If a RAG system gives fragmentary or off-target answers, the chunking is the first thing to inspect, not the model.
Where RAG shines and where it disappoints
- Question-answering over a known, changing corpus — support docs, policies, product manuals, a knowledge base — where you want current answers without retraining anything.
- Grounding and citations. Because the model is handed specific passages, it can point to where an answer came from, which matters for anything auditable.
- Freshness and updateability. Change a document and re-embed it; the system knows the new version immediately, with no model training involved.
- Cost control. Retrieving five relevant passages is far cheaper than stuffing an entire library into every prompt.
- Questions that need the whole corpus at once. Summarize every contract we signed this year is not a nearest-neighbor lookup; retrieval returns a few chunks, not a global view.
- Reasoning that spans many scattered pieces. If the answer requires connecting facts from ten documents, retrieving the top few will miss most of them.
- When retrieval quality is poor. RAG is only as good as what it retrieves. Bad chunking, a weak embedding model, or an ambiguous question surfaces the wrong passages, and the model faithfully answers from the wrong material — often the hardest kind of error to catch.
- Anything requiring the model to behave differently, not know more. RAG adds knowledge; it does not change tone, format, or skill. That is a different problem, addressed in the fine-tuning guide.
RAG versus long context
Every time context windows grow, someone declares RAG dead: why retrieve when you can paste the whole thing? For a single moderate document that fits comfortably, that is often the right call — just put it in the prompt. RAG earns its keep when the knowledge is larger than any window, changes often, must be cited, or is queried so frequently that stuffing a giant context into every call is wasteful. And as the context-window guide notes, a model's usable recall falls off well before its advertised limit, so a huge paste is not the reliability win it appears to be. The two also combine: retrieve the relevant sections, then use a generous context to reason over them. It is rarely all-or-nothing.
Build versus use built-in features
You do not always have to build a retrieval pipeline. Many chat products now let you upload files or connect a knowledge source and handle the retrieval for you; for personal use or a small document set, that is usually enough and far less work. Building your own RAG system is worth it when you need control — over how documents are chunked, which embedding model is used, how results are ranked and filtered, and how the whole thing is evaluated — or when it must integrate with your own data and application. A reasonable path: start with a built-in file or knowledge feature, confirm that retrieval is actually your bottleneck, and only then build the custom pipeline you now understand you need.
RAG is not magic and not dead. It is plumbing: find the right text, hand it to the model, cite it. When the failure is missing knowledge in documents you control, it is often the highest-leverage thing you can add — and when it disappoints, the cause is almost always retrieval, not the model.