RAG (Retrieval-Augmented Generation)
A technique that improves LLM answers by retrieving relevant passages from an external knowledge base and adding them to the prompt before generation, reducing hallucinations and giving the model access to information beyond its training data.
Large language models are limited by their training data cutoff and by what fits in the context window. Retrieval-Augmented Generation (RAG) adds a retrieval step before generation: a search system pulls relevant documents or chunks from an external knowledge base and inserts them into the model's context window.
How it works
A typical RAG pipeline has three stages:
- Index: Documents are split into chunks (paragraphs, pages or semantic units), converted into embedding vectors and stored in a vector database.
- Retrieve: The user's query is embedded with the same model, and the database returns the chunks whose vectors are most similar, usually by cosine similarity or dot product.
- Generate: The retrieved chunks are added to the prompt alongside the query, and the model answers from that grounded context, ideally citing which passages it used.
Why developers use it
RAG addresses three LLM limitations: knowledge cutoffs (answering questions about recent or proprietary data), hallucination (answers are grounded in retrieved passages that can be cited and checked), and context cost (only the relevant pieces are sent instead of the whole knowledge base).
Common patterns
- Naive RAG: Simple top-k retrieval and injection into the prompt. Useful as a baseline.
- Hybrid search: Combine dense vector retrieval with sparse keyword retrieval (BM25). Improves recall for exact matches like product SKUs or proper nouns.
- Reranking: Use a cross-encoder or reranking model to reorder the top-k chunks before passing them to the LLM. Cuts irrelevant context.
- Agentic RAG: The model gets search as a tool and decides when to retrieve, what to search for and whether to search again, which handles multi-hop questions better than a single retrieval pass. Self-RAG, a research method, trains the model to decide when to retrieve and to critique its own outputs.
- Managed retrieval: Hosted tools such as the file search tool in OpenAI's API handle chunking, embedding and search for you, with less control over each step.
RAG vs long context
Million-token context windows are now common on cheap models (Claude Haiku 5.5 offers 1M tokens and GPT-6 Luna 1.05M), so a small knowledge base can simply be pasted into the prompt. RAG still wins on cost and freshness for anything larger or frequently changing. A prompt that fills most of Claude Haiku 5.5's window, say 900K tokens, costs about $0.45 in input alone, because prompts over 100K tokens are billed at $0.50 per million input tokens instead of $0.10. A RAG pipeline that sends 5,000 relevant tokens costs a fraction of a cent. Prompt caching narrows the gap when the same large context is reused across many requests.
Common failure modes
RAG quality depends heavily on chunking strategy and embedding model choice. Poorly chunked documents lose context at chunk boundaries. Queries and documents must be embedded with the same model; mixing models or model versions makes similarity scores meaningless. Retrieval is usually the bottleneck, not generation. Retrieved context can also add noise that degrades answers, particularly when passages are only loosely relevant or contradict each other. Retrieval can miss relevant passages when the query uses different vocabulary from the documents, which is why techniques like query rewriting, hybrid search and multi-stage ranking are common.
Related terms
Models relevant to RAG (Retrieval-Augmented Generation)
Claude Haiku 5.5
Anthropic's fastest, lowest-cost current model at $0.10 per million input tokens with a 1M context window
View model →GPT-6 Luna
OpenAI's cheapest GPT-6 model, and since October 2026 the default for free ChatGPT users
View model →Gemini 3.8 Flash
Google's previous-generation Flash model for coding and agents, at half Gemini 4 Hydrogen's price until December 31, 2026
View model →