¶What does retrieval-augmented generation add beyond the model's stored knowledge?
Your study assistant can explain database transactions correctly and still invent the deadline printed in your course notes. This chapter turns it into an evidence pipeline: retrieve relevant material, assemble a bounded context, generate an answer, and expose which sources support it—or decline when none do.
A model’s stored knowledge is not a searchable copy of your course folder. Its parameters encode patterns learned during training, not a dependable lookup table with page numbers. Even when an answer is correct, asking “Which document supports that?” can produce a fabricated reference. I would not accept a fluent answer as evidence that the assistant has consulted the notes.
Retrieval-augmented generation, or RAG, supplies external evidence at answer time. The retrieval system from Chapter 8 finds candidate passages; the application gives selected passages to the model alongside the question. The model’s weights remain unchanged in the pipeline built here. Updating an indexed document can therefore change the available evidence without retraining the generator, provided the updated content has actually reached the index.
¶Give the model something it could not reliably know
Consider an illustrative course notice, not a measured production result: “Database assignment 2 is due on 18 September at 17:00.” A student asks, “When is assignment 2 due?” Without the notice, the model should not guess. With the right passage, it can answer with the date, time, and a reference to the notice.
The engineering work sits between those two sentences. Your application retrieves, say, 5 candidate chunks, selects the relevant notice, and preserves its source_id, page_number, and chunk_id. It assigns a short citation_label for the prompt and retains the mapping back to the original passage. These are application fields, not a universal RAG standard. I prefer explicit fields because a missing source mapping should become a visible application failure, not a convincing-looking citation.
The evidence might occupy 700 tokens inside a 1,000-token context allowance you chose for retrieved material. Those figures are an illustrative budget, not recommended defaults. If the selected passages exceed that allowance, context assembly must omit or shorten material deliberately. Silent truncation can remove the deadline while leaving the assignment title, producing an answer that looks grounded but lacks the decisive fact.
Retrieval adds access, not certainty. A search for “assignment 2” might return last semester’s notice; a passage might mention the assignment without its deadline; the generator might cite the correct page but state the wrong time. None of these failures requires an API exception. A successful request can still deliver an unsupported answer.
I therefore treat source selection and generation as separate responsibilities. Unless the selected material supports the requested fact, the assistant should say, “I could not find that deadline in the available notes.” That statement describes the evidence available to the application. It does not claim the deadline does not exist.
¶The paper and the application pattern are not identical
The 2020 RAG paper by Lewis and colleagues proposed a specific training approach combining a pretrained sequence-to-sequence generator with a dense Wikipedia index accessed through a neural retriever. Its two formulations allowed the same retrieved passages to condition an entire answer or different passages to contribute at different token positions.
The common application pattern now called RAG is broader: retrieve documents, put evidence into a prompt, and call a generator. That is the pattern used here. It does not reproduce the paper’s training recipe, and I would not describe a vector-search wrapper as an implementation of the original architecture.
Nor would I retrieve for every request. “Explain recursion with a short example” may need no course-specific evidence, unless the student explicitly asks for the lecturer’s treatment. “What does my lecturer require?” does. Retrieval also adds a search step, context tokens, and another place for stale data to enter; it earns that cost when answers need external, changeable, or traceable information.
Try changing the notice to 19 September while leaving the model untouched. If the assistant still answers 18 September, inspect the retrieved passage before rewriting the prompt. Which version of the notice actually reached the generator?
¶How do we assemble retrieved passages within a fixed context budget?
A retriever can return the right paragraph and your application can still hide it from the model. Duplicate chunks consume the allowance, a long passage crowds out a useful definition, or automatic truncation removes the exception that changes the answer. Retrieval selects candidates. Context assembly decides which evidence the model actually receives.
I start with a deterministic packer, not another model call. Its job is to select complete passages, preserve their source identifiers, and account for the serialized request. Unless you can inspect that request, a retrieval success can disguise a context-assembly failure.
¶Reserve the answer before packing the evidence
For a model whose context window includes input and output, subtract the output reservation first. With an illustrative 8,192-token window, a 1,024-token answer reservation, a 256-token safety margin, and 620 tokens of instructions, question, and message formatting, the evidence allowance is 6,292 tokens. These are planning figures, not a model specification. Check your endpoint’s separate input and output limits as well.
Count using the generator’s tokenizer, not the embedding model’s tokenizer. For chat models, include the chat template and generation prefix discussed earlier. Counting only passage text underestimates the request because source labels, separators, and message wrappers also occupy tokens. A margin covers uncertain overhead; it does not repair the wrong tokenizer.
I would reject an oversized fixed prompt before retrieval. Silently shortening the student’s question can change the task. Depending on the serving system, an oversized request may produce a context-limit error or undergo truncation; neither belongs in your packing policy.
¶Pack complete evidence, not arbitrary token slices
Keep candidates in retrieval order initially. Remove exact duplicate text, then accept a passage only when the complete rendered prompt still fits. If one passage is too large, skip it and try the next; stopping immediately wastes space that a shorter passage could use. This greedy policy does not find an optimal evidence set, but it is understandable and testable.
The following example requires the already introduced transformers package and downloads the gpt2 tokenizer on its first run. It demonstrates counting a plain-text prompt, not a chat request or answer generation. Its deliberately small 256-token window is a test constraint, not GPT-2’s context limit. Replace the tokenizer and serialization together for your generator.
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
window, output_reserve, margin = 256, 64, 16
input_limit = window - output_reserve - margin
question = "How do gradient accumulation and batch size relate?"
candidates = [
{
"id": "notes-p12-a",
"source": "training-notes.pdf, page 12",
"text": "Gradient accumulation adds gradients across microbatches. " * 100,
},
{
"id": "notes-p13-a",
"source": "training-notes.pdf, page 13",
"text": "Four microbatches of 8 examples give an effective batch of 32.",
},
{
"id": "notes-p13-b",
"source": "training-notes.pdf, page 13",
"text": "Four microbatches of 8 examples give an effective batch of 32.",
},
{
"id": "notes-p14-a",
"source": "training-notes.pdf, page 14",
"text": "Accumulation need not match a larger batch with batch-dependent layers.",
},
]
def render(passages):
evidence = "\n\n".join(
f"[{p['id']}] {p['source']}\n{p['text']}" for p in passages
)
return (
"Answer from the evidence. Cite source IDs. "
"If evidence is insufficient, say so.\n"
f"Evidence:\n{evidence}\nQuestion: {question}\nAnswer:"
)
def count_tokens(text):
return len(tokenizer.encode(text, add_special_tokens=False))
if count_tokens(render([])) > input_limit:
raise ValueError("Fixed prompt exceeds input budget")
selected, decisions, seen = [], [], set()
for passage in candidates:
text = passage["text"]
if text in seen:
decisions.append((passage["id"], "duplicate"))
continue
seen.add(text)
trial = selected + [passage]
if count_tokens(render(trial)) <= input_limit:
selected.append(passage)
decisions.append((passage["id"], "included"))
else:
decisions.append((passage["id"], "over_budget"))
prompt = render(selected)
assert count_tokens(prompt) + output_reserve + margin <= window
print(decisions)
print("Input tokens:", count_tokens(prompt))
print(prompt)
The packer records over_budget for the long passage and duplicate for the repeated sentence. It retains the batch-size example and its qualification. That qualification matters: cutting after the first convenient sentence could produce an unsupported claim that accumulation always matches a larger batch.
Inspect decisions, selected, and count_tokens(prompt) during development. Preserve the mapping from notes-p14-a to the original document location outside the prompt too. Otherwise, a plausible citation can become an unresolvable source reference. For an empty selection, route to insufficient-evidence handling rather than treating the empty prompt as permission to answer from memory.
I avoid filling every remaining token merely because space exists. More overlapping passages can bury the useful sentence without adding evidence. Exact deduplication is only a baseline: overlapping chunks with different wording survive it. Keep their provenance, and inspect overlap before adding more selection machinery.
¶A boundary test worth keeping
Change the example’s window to 32. The fixed prompt should raise ValueError with Fixed prompt exceeds input budget, rather than quietly discarding the question. Then restore 256 and move the short qualification ahead of the long candidate. It should still survive. These tests distinguish budget handling from accidental list-order behavior.
An interviewer might ask why retrieving 20 passages does not imply sending 20 passages. My answer: retrieval proposes evidence; assembly must satisfy the generator’s budget while retaining useful, traceable support. Once the request fits, the next question is harder: does the generated answer actually use that support?
¶Can the assistant produce answers whose citations point to supporting passages?
A citation can look convincing while pointing to a passage that says nothing about the answer. Worse, a model can invent a document title and page number that your interface renders as a legitimate source. I would rather ship an explicit “not supported by these notes” response than a polished answer with decorative references. For the study assistant, citations must be data your application resolves, not bibliography text the model improvises.
The context assembled in the previous section gives you a bounded set of passages. Assign each included passage a request-local identifier, such as S1, and retain its document revision, page, and chunk identifier outside the generated response. Only passages actually sent to the model belong in this registry. If retrieval found 20 chunks but the budget admitted 5, the other 15 are not valid citation targets for that generation.
¶Ask for claims and evidence, not a finished bibliography
I prefer a small structured response: a claims list, with each claim containing text and an evidence list. Each evidence item contains source_id and quote. The quote is an exact excerpt from the supplied passage; the identifier lets your application attach trusted source metadata. Requiring an excerpt makes inspection easier, although it does not prove that the excerpt supports the claim.
Use this generation instruction alongside the structured-output contract already established:
Answer using only the supplied passages. Split the answer into independently checkable claims. For each claim, return at least one source identifier and an exact supporting excerpt. Do not supply filenames, links, or page numbers. If the passages cannot support an answer, return an empty claims list.
A single claim should not combine “dropout randomly zeros activations” with “dropout always improves accuracy.” A passage may support the first statement while providing no basis for the second. Putting both under 1 citation hides the unsupported assertion. Keep claims narrow, unless several passages jointly support the full statement.
¶Resolve citations before rendering
The following executable fixture isolates citation handling from model variability. It uses author-written course-note text, not an external PDF. In the application, response would be the parsed, schema-validated model response; sources would contain the exact passage strings used in that request.
sources = {
"S1": {
"document": "regularization-notes",
"revision": "v1",
"page": 3,
"chunk_id": "regularization-003",
"text": (
"During training, dropout randomly sets some activations "
"to zero. It can reduce overfitting."
),
}
}
response = {
"claims": [
{
"text": "During training, dropout randomly zeros some activations.",
"evidence": [
{
"source_id": "S1",
"quote": (
"During training, dropout randomly sets some "
"activations to zero."
),
}
],
}
]
}
def resolve_claims(response, sources):
resolved = []
for claim in response["claims"]:
if not claim["evidence"]:
raise ValueError("missing_evidence")
citations = []
for evidence in claim["evidence"]:
source_id = evidence["source_id"]
if source_id not in sources:
raise ValueError("unknown_source_id")
source = sources[source_id]
quote = evidence["quote"]
if not quote.strip() or quote not in source["text"]:
raise ValueError("quote_not_in_passage")
citations.append({
"source_id": source_id,
"document": source["document"],
"revision": source["revision"],
"page": source["page"],
"chunk_id": source["chunk_id"],
"quote": quote,
})
resolved.append({"text": claim["text"], "citations": citations})
return resolved
for claim in resolve_claims(response, sources):
print(claim["text"])
for citation in claim["citations"]:
print(
f'[{citation["source_id"]}] '
f'{citation["document"]}, revision {citation["revision"]}, '
f'PDF page {citation["page"]}: {citation["quote"]}'
)
This boundary rejects 3 concrete failures: missing_evidence prevents uncited claims, unknown_source_id prevents invented references, and quote_not_in_passage prevents invented excerpts. On any failure, withhold the candidate answer and use the bounded retry policy from the API chapter or return an explicit validation failure. Never silently delete a broken citation and publish its claim.
Exact matching is deliberately strict. If PDF extraction introduced unusual whitespace, a model’s cleaned-up quotation can fail even when its wording is faithful. I would first send and validate against the same normalized passage string, preserving the original provenance separately. Do not repair a failed quote with fuzzy matching unless you also expose the resolved excerpt; otherwise the validator starts guessing on the model’s behalf.
Render citations beside each claim, not as an undifferentiated list beneath the answer. Clicking S1 should reveal the excerpt and the stored page locator. Label page 3 as “PDF page 3” if it means the third file page rather than the printed page number. Preserve the revision too: a link to the latest notes can stop supporting an answer generated against an older edition.
¶A valid pointer is not proof
An experienced interviewer may ask: “Does this validator guarantee a grounded answer?” No. It guarantees that cited passages exist in the supplied context and quoted text occurs there. The model can still attach a genuine dropout quotation to a false claim about learning rates. Semantic support needs separate examination; Chapter 10 develops that evaluation rather than disguising string matching as understanding.
For an exercise, change the fixture’s claim to “Dropout guarantees better test accuracy,” leaving its evidence untouched. The validator still accepts it. Now change source_id to S9; it must fail. Those two outcomes separate citation integrity from support. What should the assistant do when every pointer is valid, but the available passages still cannot answer the student’s question?
¶When should insufficient or conflicting evidence cause abstention?
A study assistant that always answers will eventually turn a missing lecture note into an invented course policy. The dangerous answer is often plausible: “The assignment deadline is Friday,” supported by a passage that mentions Friday’s lecture but says nothing about submission. A valid citation does not rescue an unsupported claim.
I would make abstention an explicit application outcome, not a polite sentence the model may choose to append. Abstention means declining to assert what the available evidence cannot establish. It need not mean refusing the entire question.
¶Separate missing evidence from broken retrieval
Give the response contract a status field with outcomes such as answered, partial, insufficient_evidence, and conflicting_evidence. Keep operational failures separate: a retrieval timeout should produce retrieval_error, not “Your notes do not contain an answer.” The first describes a failed lookup; the second makes an unjustified claim about the collection.
Likewise, 0 retrieved passages justify stopping generation, but not concluding that the answer does not exist. A PDF extraction failure, an outdated index, or a poorly phrased query can all hide relevant material. Say “I could not find supporting passages in the indexed notes,” and offer a concrete next step: upload the missing lecture or specify the course week.
I avoid universal similarity cutoffs. A cosine score of 0.82 is not an 82% probability that a passage supports the answer. Even the top 5 results can all discuss the wrong assignment. A threshold is useful only when its behavior has been checked for your embedding model and collection; the evaluation chapter develops that work.
¶Decide at the claim level
Suppose a student asks, “What is the late penalty, and when is Assignment 2 due?” One passage explicitly states a 10% daily penalty; none gives a deadline. Return the penalty with its citation and decline the deadline. Do not let support for 1 claim spread to the other.
The application can route these decisions deterministically once support has been assessed. This Python example assumes that an earlier step supplies one assessment per requested claim. It does not detect semantic support itself.
def route_evidence(claim_states, retrieval_failed=False):
if retrieval_failed:
return "retrieval_error"
states = list(claim_states)
allowed = {"supported", "missing", "conflicting"}
if any(state not in allowed for state in states):
raise ValueError("Unknown claim evidence state")
if "conflicting" in states:
return "conflicting_evidence"
if not states or all(state == "missing" for state in states):
return "insufficient_evidence"
if "missing" in states:
return "partial"
return "answered"
assert route_evidence(["supported", "missing"]) == "partial"
assert route_evidence(["conflicting"]) == "conflicting_evidence"
assert route_evidence([], retrieval_failed=True) == "retrieval_error"
print(route_evidence(["supported"]))
The hard part remains upstream: deciding whether each passage actually supports each claim. An instruction to the generator can request that assessment, but cannot guarantee it. I would retain the supporting passage identifiers and the proposed claim text so that a suspicious decision can be inspected. A nonexistent identifier is an output-validation failure; an existing identifier attached to unrelated text is a support failure. Neither should silently become an answer.
¶Do not resolve contradictions by vote
Imagine 2 retrieved passages: a course handbook says the deadline is 12 March, while an assignment announcement says 14 March. First check scope: are these the same assignment, semester, and student cohort? Different scopes are not necessarily a contradiction.
If the scopes match, use an authority rule only when the application has a defensible one. An explicitly labeled correction can supersede the original announcement. A later upload timestamp alone cannot: someone may have uploaded an older PDF yesterday. Three duplicate chunks repeating 12 March do not outweigh one official correction.
When authority remains unresolved, show both dates with their citations and say that the sources disagree. Do not average dates, pick the more convenient answer, or ask the model which sounds right. Other supported parts of the response can still be delivered, provided the disputed claim stays visibly unresolved.
For a small exercise, remove the deadline passage, then add a contradictory announcement, then simulate a retrieval timeout. Your assistant should produce 3 different explanations—not the same generic apology. Before allowing it to answer again, what evidence would resolve each condition?
¶Which controlled experiment separates missing evidence from a generation failure?
A wrong answer does not tell you which component to fix. If the study assistant invents an assignment deadline, changing the prompt may hide the symptom while leaving the relevant course notice outside the context. I start with an evidence substitution experiment: keep the question and generation settings fixed, but replace the retrieved context with a passage known to support the answer. This is a diagnostic intervention, not a new retrieval strategy.
¶Change the evidence, not the whole application
Use 3 conditions: the context your application actually sent, a manually selected supporting context, and an empty context. Call the manually selected passage oracle evidence: evidence selected with knowledge of the correct answer. It must come from an authorized source, not from an answer you wrote yourself. Otherwise, you are testing whether the model can repeat your explanation.
For an illustrative study-assistant test, suppose an instructor notice says, “Assignment 2 is due on 14 September.” The question asks for that deadline, but retrieval returns a general submission policy mentioning a 7-day extension. The retrieved answer might incorrectly report the extension as the deadline. The oracle condition supplies the notice; the empty condition checks whether the assistant declines rather than inventing a date.
Freeze the model identifier, instruction text, question, output limit, and sampling settings. Save the final assembled messages, not merely the documents returned by search. A supporting passage can survive retrieval and still disappear during context assembly. Unless you inspect the exact request, that failure looks like a generation error.
This runnable Python fixture creates the 3 requests without making paid API calls. The dates and passages are invented test data, not claims about a real course.
import json
question = "When is Assignment 2 due?"
instruction = (
"Answer only from the supplied sources. Cite the source ID. "
"If the sources do not support an answer, say "
"'Insufficient evidence.'"
)
contexts = {
"retrieved": (
"[policy-p2] Approved extensions allow submission "
"up to 7 days after the deadline."
),
"oracle": (
"[notice-p1] Assignment 2 is due on 14 September."
),
"empty": "",
}
requests = {}
for condition, context in contexts.items():
requests[condition] = [
{"role": "system", "content": instruction},
{
"role": "user",
"content": (
f"Sources:\n{context}\n\nQuestion:\n{question}"
),
},
]
with open("evidence_probe.json", "w", encoding="utf-8") as handle:
json.dump(requests, handle, indent=2, ensure_ascii=False)
print("Saved", len(requests), "conditions to evidence_probe.json")
Send each entry in requests through your existing model adapter. Record the returned text beside its condition in evidence_probe.json or a separate result file. Keep instruction and question unchanged; only context varies. Inspect role and content in the saved messages if the adapter transforms requests. A timeout, authentication error, or malformed response is an execution failure, not evidence that the model could not answer.
¶Read the result without claiming too much
If the retrieved condition fails but the oracle condition answers “14 September” with notice-p1, the generator can answer under the supplied supporting evidence. Trace the notice backward: was it absent from the extracted corpus, absent from search results, or dropped from the final context? Those are different repairs. I would not change the model until that trace is complete.
If the oracle condition also fails, inspect the request before blaming model capability. A truncated passage, contradictory instruction, or invalid citation can explain the result. If the source is intact and the answer still says “7 September,” you have a generation-side failure under this test condition. If it answers correctly but cites policy-p2, the date is right and the evidence link is broken.
Success with one clean passage does not prove the generator handles a crowded context. Replacement changes both evidence availability and distraction. As a follow-up, add the supporting passage to the original context, provided it fits the same budget without displacing other text. If replacement succeeds but addition fails, inspect interference and passage placement. I prefer this second probe to immediately increasing the context window; a larger window can preserve the very distraction causing the error.
The empty condition should return Insufficient evidence. A correct date without sources still violates this assistant’s contract. Repeat an unstable result—for example, 3 trials per condition—but do not mistake those trials for a reliable quality estimate or assume zero temperature guarantees identical outputs.
For practice, preserve one failed question and explain which observation would change your diagnosis. The next chapter needs these saved requests and outcomes to turn individual debugging probes into systematic retrieval and answer evaluation.