LLMs
Retrieval-Augmented Generation (RAG)
Follow retrieval into a grounded prompt, then examine chunking, ranking, evidence quality, and generation failures.
Jump to a section
Why this matters
Retrieval-augmented generation adds retrieved information to a model's context before it produces an answer. The system prepares an index, retrieves candidate passages for a query, and places selected evidence into the prompt.
Retrieval and generation are separate failure points. Finding relevant evidence is necessary but not sufficient: the generator can still misread it, omit an important qualification, or make an unsupported claim.
Build the mental model
Offline, you chunk your documents and embed each chunk into a vector, stored in a vector database. At query time, you embed the user's question into the same space, find the nearest chunks (the ones most semantically similar), and paste them into the prompt as context. The model answers using that retrieved evidence rather than its parametric memory alone.
Retrieval adds context before generation
Follow the question through retrieval into the model’s prompt. Retrieved passages provide evidence for the answer, not a guarantee of correctness.
The visual loads as you reach this section.
Read the connections
Question -> Encode query : query Encode query -> Retrieve chunks : vector Retrieve chunks -> Build prompt : passages Build prompt -> Generate answer : question + context
The retriever's job is to surface the right evidence; the generator's job is to synthesize it. If retrieval misses essential evidence, the answer cannot be grounded in that evidence.
Work through the math
Each document chunk and the query are mapped to vectors by an embedding model. Relevance is similarity in that space, typically cosine or dot product:
Retrieve the top- chunks by score:
then build the prompt as and decode. Two design levers dominate quality: chunking (chunks must be small enough to be specific but large enough to be self-contained) and retrieval quality (often improved with hybrid keyword + vector search and a re-ranker that re-scores the top candidates with a stronger model). Approximate nearest-neighbor indexes (HNSW, IVF) make top- search fast over millions of vectors.
Key takeaway: RAG swaps knowledge in at inference time: embed chunks offline, retrieve top-k by similarity, generate grounded in them. Missing evidence limits what the answer can substantiate, even if the model can produce a plausible response.
| RAG | Fine-tuning | |
|---|---|---|
| Adds knowledge | At inference, via the index | Baked into weights |
| Updating knowledge | Re-index documents | Retrain |
| Fresh/private data | Natural fit | Requires new training runs |
| Citations/grounding | Yes: show retrieved sources | Not directly |
| Best for | Facts, lookup, changing corpora | Style, format, task behavior |
Read the implementation
1import numpy as np
2
3def embed(texts): # stand-in for a real embedding model
4 rng = np.random.default_rng(abs(hash(tuple(texts))) % 2**32)
5 return rng.normal(size=(len(texts), 384))
6
7def cosine(a, b):
8 return (a @ b.T) / (np.linalg.norm(a, axis=1)[:, None] * np.linalg.norm(b, axis=1))
9
10docs = ["Batch norm normalizes per feature.", "Attention scales by sqrt(d_k).", "RAG retrieves context."]
11doc_vecs = embed(docs) # offline: index the chunks
12
13def retrieve(query: str, k: int = 2) -> list[str]:
14 q_vec = embed([query])
15 scores = cosine(q_vec, doc_vecs)[0] # similarity to every chunk
16 top = np.argsort(scores)[::-1][:k] # top-k indices
17 return [docs[i] for i in top]
18
19context = retrieve("how does attention scale scores?")
20prompt = f"Context:\n" + "\n".join(context) + "\n\nQuestion: ...\nAnswer using only the context."Questions and trade-offs
- Conceptual: When would you use RAG instead of fine-tuning? (When knowledge is large, changing, or proprietary, and you need grounding/citations: RAG updates by changing the index, not the weights.)
- Implementation: Why does chunking strategy matter so much? (Chunks too large dilute relevance and waste context; too small lose the surrounding meaning needed to answer: both hurt retrieval and generation.)
- Applied: How does RAG reduce hallucination? (It supplies the model with the actual source text to answer from, so it's not forced to invent facts from parametric memory.)
- Systems-level: What's a re-ranker and why add one? (A stronger model that re-scores the top-k retrieved candidates for relevance: cheap vector search casts a wide net, the re-ranker sharpens precision.)
- Failure modes: What happens when retrieval fails, and how do you catch it? (Generation is grounded in wrong/irrelevant context and answers confidently wrong; you evaluate retrieval (recall@k) and generation (faithfulness/groundedness) separately.)
Check your understanding
Without looking: draw the retrieve→augment→generate pipeline, write the similarity-and-top-k retrieval step, and name the two levers (chunking, retrieval quality) that most affect RAG quality. Check against Stages 2–3.
Oliver Perrin
Machine Learning Engineer · Founder, LiminalML
LiminalML brings together concept explanations, mathematical examples, and working visuals. Find more writing by Oliver on the LiminalML Substack.
Keep exploring the idea.
Ask a follow-up in a guided session, work through the self-check, or open the visual in Studio to build on it.
Related concepts
LLMs
Embeddings
Compare vectors by direction and magnitude, and connect embedding geometry to retrieval and representation learning.
LLMs
Tokenization & BPE
Follow a pair of vocabulary merges and distinguish learning merge rules from applying a tokenizer to new text.
LLMs
KV Cache
See which token projections caching saves, what attention must still read, and how memory use grows with context.