Retrieval-Augmented Generation (RAG)

Retrieval-augmented generation adds retrieved information to a model's context before it produces an answer. The system prepares an index, retrieves candidate passages for a query, and places selected evidence into the prompt.

Retrieval and generation are separate failure points. Finding relevant evidence is necessary but not sufficient: the generator can still misread it, omit an important qualification, or make an unsupported claim.

One common implementation chunks documents offline and embeds each chunk into a vector index. RAG can also use keyword or hybrid retrieval; a vector database is not required. At query time, you embed the user's question into the same space, find the nearest chunks (the ones most semantically similar), and paste them into the prompt as context. The model answers using that retrieved evidence rather than its parametric memory alone.

The retriever's job is to surface the right evidence; the generator's job is to synthesize it. If retrieval misses essential evidence, the answer cannot be grounded in that evidence.

Each document chunk $d_i$ and the query $q$ are mapped to vectors by an embedding model. Similarity in that space, often cosine or dot product, is used as a relevance estimate:

Retrieve the top-$k$ chunks by score:

then build the prompt as $[\text{instructions};\, d_{(1)}, \dots, d_{(k)};\, q]$ and decode. Two design levers dominate quality: chunking (chunks must be small enough to be specific but large enough to be self-contained) and retrieval quality (often improved with hybrid keyword + vector search and a re-ranker that re-scores the top candidates with a stronger model). Approximate nearest-neighbor indexes (HNSW, IVF) make top-$k$ search fast over millions of vectors.

Key takeaway: RAG swaps knowledge in at inference time: embed chunks offline, retrieve top-k by similarity, generate grounded in them. Missing evidence limits what the answer can substantiate, even if the model can produce a plausible response.

This runnable toy uses word-count vectors, not learned semantic embeddings, so retrieval is lexical. It demonstrates indexing, cosine ranking, and prompt assembly without an external service. A semantic retriever would replace embed with a trained encoder.

1import re
2import numpy as np
3
4docs = ["Batch norm normalizes per feature.", "Attention scales by sqrt(d_k).", "RAG retrieves context."]
5
6def words(text):
7    return re.findall(r"[a-z_]+", text.lower())
8
9vocab = sorted({word for doc in docs for word in words(doc)})
10
11def embed(texts):
12    return np.array([[words(text).count(word) for word in vocab] for text in texts], dtype=float)
13
14def cosine(a, b):
15    norms = np.linalg.norm(a, axis=1)[:, None] * np.linalg.norm(b, axis=1)
16    return np.divide(a @ b.T, norms, out=np.zeros((len(a), len(b))), where=norms > 0)
17
18doc_vecs = embed(docs)                          # offline: index the chunks
19
20def retrieve(query: str, k: int = 2) -> list[str]:
21    q_vec = embed([query])
22    scores = cosine(q_vec, doc_vecs)[0]         # similarity to every chunk
23    top = np.argsort(scores)[::-1][:k]          # top-k indices
24    return [docs[i] for i in top if scores[i] > 0]  # no lexical match: no evidence
25
26query = "how does attention scale scores?"
27context = retrieve(query)
28assert context == [docs[1]]
29prompt = "Context:\n" + "\n".join(context) + f"\n\nQuestion: {query}\nAnswer only if the context supports it."

1. Conceptual: When would you use RAG instead of fine-tuning? (When knowledge is large, changing, or proprietary, and you need grounding/citations: RAG updates by changing the index, not the weights.)
2. Implementation: Why does chunking strategy matter so much? (Chunks too large dilute relevance and waste context; too small lose the surrounding meaning needed to answer: both hurt retrieval and generation.)
3. Applied: How does RAG reduce hallucination? (It supplies source text that can support the answer. Retrieval and generation can still fail, so evaluate claim support and abstain when evidence is missing.)
4. Systems-level: What's a re-ranker and why add one? (A stronger model that re-scores the top-k retrieved candidates for relevance: cheap vector search casts a wide net, the re-ranker sharpens precision.)
5. Failure modes: What happens when retrieval fails, and how do you catch it? (The generator may answer from irrelevant context or unsupported memory; you evaluate retrieval (recall@k) and generation (faithfulness/groundedness) separately.)

Without looking: draw the retrieve→augment→generate pipeline, write the similarity-and-top-k retrieval step, and name the two levers (chunking, retrieval quality) that most affect RAG quality. Check against Stages 2–3.

Sources and further reading

• Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: combining retrieval with generation; the toy code here illustrates prompt assembly, not the paper's full trained architecture.