Fix the Failing Stage Before Buying a Bigger Model

¶How do answerable, unanswerable, and adversarial questions form a useful RAG test set?

A study assistant can produce a wrong answer because the passage never reached the model, because the model misread it, or because the notes never contained an answer. Buying a larger model before distinguishing those failures makes the experiment more expensive without making its result more informative.

I would start with a small, inspectable test set rather than hundreds of automatically generated questions. For this project, 30 carefully reviewed cases—12 answerable, 10 unanswerable, and 8 adversarial—are a reasonable first diagnostic batch, not a statistically reliable benchmark. Keep the held-out split established earlier untouched while you develop these cases.

¶Label what the evidence permits

Answerable means the frozen course corpus contains enough evidence to satisfy the question. General knowledge does not qualify. If the assistant correctly explains gradient descent but the uploaded notes contain only a course timetable, it has failed this application's grounding requirement.

Unanswerable means the corpus does not support the requested answer. Include plausible omissions, not just unrelated questions: a missing examination date, an unspecified optimizer setting, or a comparison for which only one method is documented. Unless you inspect the relevant documents, “unanswerable” may merely mean your search missed the passage.

Adversarial questions deliberately expose misleading behavior: false premises, near-matching terminology, contradictory passages, or pressure to provide certainty. This is not a third answerability state. Record it as a separate flag, because a difficult question can still be answerable. Security-focused attacks belong to the later safety chapter; here, the target is evidence handling.

Consider an illustrative, author-written course note: “The lab uses batch size 32. The submission deadline is 18 September. No optimizer learning rate is specified.” These cases exercise different obligations:

¶Store an evidence contract, not just a reference sentence

I prefer required facts and forbidden claims over one “perfect” answer string. Exact wording is rarely the behavior worth protecting. Store case_id, question, answerable, adversarial, required_facts, forbidden_claims, and evidence_ids. Attach corpus_version so a changed PDF cannot silently change the test's meaning.

This runnable fixture validates the minimum annotation contract. It does not grade generated answers:

cases = [
    {
        "case_id": "batch-premise",
        "question": "Why does the lab require batch size 64?",
        "answerable": True,
        "adversarial": True,
        "required_facts": ["The lab uses batch size 32."],
        "forbidden_claims": ["The lab requires batch size 64."],
        "evidence_ids": ["lab-note:p1"],
        "corpus_version": "course-notes-v1",
    },
    {
        "case_id": "missing-rate",
        "question": "What learning rate does the lab use?",
        "answerable": False,
        "adversarial": False,
        "required_facts": ["The note does not specify a learning rate."],
        "forbidden_claims": ["The learning rate is 0.001."],
        "evidence_ids": [],
        "corpus_version": "course-notes-v1",
    },
]

seen = set()
for case in cases:
    if case["case_id"] in seen:
        raise ValueError("duplicate case_id")
    seen.add(case["case_id"])
    if case["answerable"] and not case["evidence_ids"]:
        raise ValueError("answerable case needs evidence_ids")
    if not case["required_facts"]:
        raise ValueError("case needs an expected behavior")

print(f"Validated {len(cases)} cases")

Preserve a short supporting quotation and page location alongside each evidence identifier in the full dataset. Prefer stable source locations to chunk identifiers alone: rechunking can replace a chunk without changing the source. For an unanswerable case, an empty evidence list records no known supporting passage; it does not prove absence. Record which documents the reviewer checked.

Use explicit failure labels such as missing_evidence, unsupported_claim, wrong_citation, and false_abstention. A correct number with a citation to an unrelated paragraph is still a citation failure. A refusal on every question avoids unsupported claims but fails the answerable slice. I would reject a single aggregate score that hides either behavior.

Report counts separately. On a slice of 10 unanswerable questions, one extra fabricated answer changes the observed failure rate by 10 percentage points; do not mistake that jump for a precise population estimate. Likewise, model-generated questions need human review because they often copy the passage's wording, making retrieval artificially easy.

¶Make the diagnosis explainable

An interviewer asks, “The assistant abstained; is that a success?” My answer is conditional: only if the corpus cannot support the requested answer. If supporting evidence exists but retrieval missed it, abstention is appropriate for the supplied context while the overall system still failed.

For an exercise, add one paraphrase, one missing-detail question, and one false-premise question to a real course passage. Write the expected evidence before running the assistant. When a case fails, can you show whether the supporting passage existed, was retrieved, and actually reached the answer prompt?

¶What do Recall@k, MRR, and nDCG reveal that answer accuracy cannot?

A correct answer can hide broken retrieval. Your study assistant might explain deadlock from the model’s training data while retrieving three irrelevant paragraphs. Answer accuracy passes; the evidence pipeline fails. Conversely, the right passage may arrive too far down the ranking to survive context assembly. I would inspect retrieval scores before changing the generator: a larger model cannot reliably cite a passage it never receives.

Use the answerable questions from your test set, with relevance judgments attached to stable passage identifiers. Keep unanswerable questions separate: when no relevant passage exists, recall has a zero denominator. Reporting that case as zero confuses “nothing should be retrieved” with “the retriever missed the evidence.”

¶Coverage, first useful result, and ranking quality

Recall@k asks how much relevant evidence reached the first k results. Divide the number of distinct relevant passages retrieved by the total number labeled relevant. If 2 passages are relevant and only 1 appears in the first 5 results, Recall@5 is 0.5. Its weakness is deliberate: a relevant passage at rank 1 counts exactly as much as one at rank 5.

I start with recall because missing evidence limits everything downstream. But label completeness matters. Unless you have judged alternative valid passages, a retriever can find useful evidence and still receive a low score. Treat unlabeled passages as unjudged during error analysis, not automatically useless.

Mean reciprocal rank, or MRR, asks how soon the first useful result appears. For each question, take 1 divided by the first relevant rank; use zero if none appears within the cutoff. Average those values across questions. A first relevant result at rank 2 contributes 0.5. State the cutoff explicitly as MRR@5, for example, because an unrestricted ranking answers a different question.

MRR is useful when one passage usually suffices. It can mislead you on comparison questions: retrieving one definition first earns a perfect reciprocal rank even when the second required definition is missing. That is a coverage failure, not a generation failure.

Normalized discounted cumulative gain, or nDCG, rewards putting stronger evidence earlier. Assign grades such as 0 for irrelevant, 1 for helpful, and 3 for directly sufficient evidence. Here, DCG sums each result’s gain, 2 ** grade - 1, divided by log2(rank + 1). Divide DCG by the ideal DCG obtained by sorting all judged passages by grade and taking the same cutoff. State this gain convention; linear gains produce different scores.

¶A ranking you can audit by hand

Suppose the deadlock question has two relevant passages: deadlock-definition, graded 3, and deadlock-example, graded 1. The returned order is an irrelevant passage, the example, then the definition. Recall@3 is 1.0, reciprocal rank is 0.5, and nDCG@3 is approximately 0.541. All evidence arrived, but the strongest evidence arrived last.

from math import log2

def retrieval_scores(ranked_ids, grades, k):
    if k < 1:
        raise ValueError("k must be positive")
    if len(ranked_ids) != len(set(ranked_ids)):
        raise ValueError("duplicate passage IDs")
    if any(grade < 0 for grade in grades.values()):
        raise ValueError("relevance grades must be nonnegative")

    relevant = {pid for pid, grade in grades.items() if grade > 0}
    if not relevant:
        raise ValueError("no positive relevance judgments")

    top = ranked_ids[:k]
    recall = len(set(top) & relevant) / len(relevant)
    reciprocal_rank = next(
        (1 / rank for rank, pid in enumerate(top, 1) if pid in relevant),
        0.0,
    )

    def dcg(values):
        return sum(
            (2 ** grade - 1) / log2(rank + 1)
            for rank, grade in enumerate(values, 1)
        )

    actual = dcg([grades.get(pid, 0) for pid in top])
    ideal = dcg(sorted(grades.values(), reverse=True)[:k])
    return recall, reciprocal_rank, actual / ideal

grades = {"deadlock-definition": 3, "deadlock-example": 1}
ranking = ["cpu-scheduling", "deadlock-example", "deadlock-definition"]
scores = retrieval_scores(ranking, grades, 3)
print(dict(zip(("recall", "reciprocal_rank", "ndcg"),
               (round(value, 3) for value in scores))))

retrieval_scores returns per-question values; average each component across eligible questions to obtain the suite scores. The implementation treats absent judgments as grade 0 for scoring. Duplicate identifiers raise ValueError with duplicate passage IDs, rather than allowing repeated evidence to inflate ranking gain. Distinct overlapping chunks still require a labeling policy; identifiers alone cannot detect duplicated meaning.

I would report scores both before and after context assembly if truncation or filtering changes what the generator sees. High retrieval recall does not excuse a passage silently dropped from the prompt.

Interview check: Can perfect Recall@5 coexist with a wrong answer? Yes: the ranking may bury decisive evidence, or the generator may ignore it. Exercise: move the definition to rank 1 and rerun the code. Recall stays fixed while the ranking metrics improve. If that happens on your real test set, is evidence ordering—not model capacity—the next failure to investigate?

¶When do hybrid retrieval and cross-encoder reranking recover the missing evidence?

Your study assistant answers a question about BCNF with a generic paragraph on normalization. The source passage exists, but dense retrieval ranked broader explanations above the exact rule. Buying a larger generator does not repair that failure. I would first inspect whether the passage is absent from the candidate pool or merely buried inside it. Those are different problems: hybrid retrieval can expand the pool; reranking can reorder only what reaches it.

¶Recover candidates before improving their order

Hybrid retrieval combines lexical and dense search results. For a query containing “BCNF,” a course code, or an error string such as ModuleNotFoundError, lexical retrieval can preserve exact matches that embeddings underweight. Dense retrieval earns its place when students paraphrase: “a fact depending on only part of a combined key” may retrieve a passage about partial dependencies without sharing its terminology. Neither branch is automatically correct.

I start with reciprocal rank fusion rather than adding raw BM25 and cosine scores. Their scales are not comparable; a weighted sum becomes fragile unless you calibrate it against representative queries. Reciprocal rank fusion assigns each passage a contribution of 1 / (offset + rank) from each list, then adds those contributions. Ranks begin at 1. The illustrative offset of 60 below softens differences near the top; it is a starting value, not a property of your notes.

def reciprocal_rank_fusion(rankings, offset=60):
    scores = {}
    for ranking in rankings:
        unique_ids = dict.fromkeys(ranking)
        for rank, passage_id in enumerate(unique_ids, start=1):
            scores[passage_id] = (
                scores.get(passage_id, 0.0)
                + 1.0 / (offset + rank)
            )
    return sorted(scores, key=lambda item: (-scores[item], item))


dense_ids = ["overview", "normalization", "bcnf"]
lexical_ids = ["bcnf", "exam", "normalization"]

fused_ids = reciprocal_rank_fusion([dense_ids, lexical_ids])
assert fused_ids[0] == "bcnf"
print(fused_ids)

This fixture is not a benchmark. In dense_ids, the supporting passage is third; in lexical_ids, it is first; reciprocal_rank_fusion puts it first overall. dict.fromkeys prevents duplicate identifiers within one list from voting twice. Use stable passage identifiers across both indexes: if the same paragraph has 2 identifiers, fusion treats it as 2 documents and duplicate evidence can crowd out another source.

¶Spend reranking effort on plausible candidates

A cross-encoder scores the question and one candidate passage together, allowing token interactions across the pair before producing a relevance score. Unlike independently computed embeddings, those scores depend on the particular question–passage combination. That extra computation can distinguish a passage stating the BCNF condition from one merely mentioning BCNF. It does not establish factual truth, and a high relevance score is not a calibrated probability that the answer is supported.

As an initial experiment, retrieve 20 candidates from each branch, deduplicate their union, rerank at most 40 passages, and send the best 5 onward. These are trial settings, not measured recommendations. At most 40 question–passage pairs need scoring; actual latency depends on the checkpoint, input lengths, and hardware. Record the reranker checkpoint revision and its input limit. If truncation removes the sentence containing the exception, the reranker can confidently promote the wrong passage. A larger candidate pool does not fix text the scorer never sees.

I would reject a reranker that improves aggregate ranking while consistently discarding the course’s exceptions. Inspect those failures individually. Dense retrieval can miss an exact identifier; fusion can promote keyword-heavy distractors; a cross-encoder can prefer a fluent but incomplete explanation. None of these failures necessarily throws an exception. The request succeeds while the evidence disappears.

¶Make the upgrade earn its cost

Hold chunks, queries, relevance labels, and the final context budget fixed. Compare dense-only, lexical-only, fused retrieval, and fusion followed by reranking. Measure candidate Recall@40 before reranking and nDCG@5 afterward, alongside retrieval latency. If the gold passage never enters the union, investigate extraction, chunk boundaries, or candidate depth before changing the scorer. If it enters but falls below position 5, reranking is a justified experiment.

For an interview or project review, defend the change with paired query results, not one attractive example. Which previously failing questions gained their supporting passage—and, with that evidence now present, does the generated answer actually use it?

¶How should humans and model judges assess faithfulness, relevance, and citation support?

A fluent answer can fail even when retrieval finds the right passage. The study assistant may add an unsupported deadline, answer a neighbouring question, or attach a genuine citation to a claim that the source never makes. A single “quality” score hides those differences. I would keep three separate judgements, because each points to a different repair.

¶Score claims, not the impression an answer leaves

Faithfulness asks whether the supplied evidence supports the answer’s factual claims. Relevance asks whether the answer addresses the question, including its requested scope. Citation support asks whether each cited passage actually supports the claim attached to it. A statement can be true from general knowledge yet unfaithful to the supplied notes; a faithful answer can still answer the wrong question.

Use this deliberately small, illustrative fixture. The student asks, “When is Assignment 2 due, and are late submissions allowed?” Passage notes-17 says: “Assignment 2 is due on 14 March. Late submissions are accepted within 48 hours with a penalty.” Passage notes-18 says: “Assignment 2 contributes 20% of the course grade.” The assistant replies: “Assignment 2 is due on 14 March. Late submissions receive no penalty [notes-17].”

Split that answer into 2 factual claims. The date is supported; the penalty claim contradicts the evidence. Claim-level faithfulness is therefore 1/2, or 50%, under an equal-weight rubric. That figure is a diagnostic, not a declaration that the answer is halfway usable. The contradiction could cause a student to lose marks. Record its severity separately.

The answer addresses both requested topics, but addresses one incorrectly. I score topical relevance separately from correctness: otherwise the same error silently lowers every metric. For relevance, use 0 for off-topic, 1 for partial coverage, and 2 for direct coverage of every requested part. Under that rubric, this answer receives 2. Its high relevance does not excuse its false claim.

Citation placement introduces another decision: does the trailing citation cover both sentences or only the second? Declare that convention before annotation. I prefer explicit claim-to-source links in evaluation records. A valid source identifier is not proof of support. Check citation precision—the proportion of claim–citation links that are supported—and citation coverage—the proportion of evidence-requiring claims with at least one supporting citation. These catch different failures: misleading references and missing attribution.

¶Give humans a reproducible annotation job

Store question, answer, retrieved_passages, and claims together. For each claim, record support_label, evidence_ids, evidence_quote, and severity. Distinguish supported, contradicted, and insufficient-evidence labels. “Not found” is not the same as “false.” If a citation identifier cannot be resolved, record citation_not_found; do not let the evaluator silently skip it.

For an initial calibration exercise, have 2 people independently label 20 answers, without seeing model names. Compare labels before discussing disagreements. If they agree on 16 answers, report 80% raw agreement, then inspect the remaining 4. This is a proposed exercise, not a measured benchmark. Unless reviewers agree on claim boundaries and citation scope, their aggregate scores are not comparable.

Include abstentions. “The supplied notes do not state the deadline” can be appropriate when evidence is absent, but is an unnecessary refusal when the retrieved passage contains it. Judge that distinction against the evidence actually supplied, while retaining the dataset’s answerability label.

¶Use model judges as reviewers, not authorities

I would automate the rubric only after humans can apply it consistently. Give the judge the question, answer, numbered passages, and exact scoring rules. Require structured labels, evidence identifiers, and short supporting quotations. Validate that quoted text exists in the referenced passage; a judge can fabricate evidence too. A matching quotation still needs a semantic support check.

Pin the judge’s model version where available and save its prompt and raw response. Treat malformed output as an evaluation failure, not a zero-quality answer. Model judges can favour verbosity or familiar phrasing, and using the answer generator as its own judge does not provide independent confirmation. A different model is not automatically unbiased either.

I do not know whether a judge is reliable on your course notes until it is checked against human labels. Audit contradictions, abstentions, and a random sample of passes—not just failures. For your next experiment, add the 20% grading statement to the fixture: does the judge reward accurate but unrequested detail, or preserve the relevance boundary you specified?

¶Which hallucination mitigations survive ablations, judge disagreement, and added latency?

A study assistant can appear safer simply because it stops answering. If unsupported answers fall from 12 to 4 while useful answers also disappear, you have changed the failure mode—not necessarily improved the product. I would choose a mitigation only after checking three things together: which errors it removes, which correct answers it damages, and how long the student waits.

¶Make each intervention earn its place

Start with the frozen questions and stage-level records from this chapter. Keep the model revision, document snapshot, retrieval settings, and generation budget fixed unless one is the variable under test. Store question_id, variant, supported_answer, unsupported_answer, abstained, and elapsed_ms for each response. Preserve the evidence and answer too; a score without its underlying text is difficult to audit.

Test the cheapest plausible correction first. For an answer that invents an exam deadline absent from the notes, I would first tighten the evidence-only instruction and abstention rule. A reranker cannot recover a deadline that the corpus never contained. Conversely, if the deadline exists but falls outside the selected passages, another generation instruction addresses the wrong stage.

The following numbers are an invented accounting exercise, not benchmark results. Assume 100 questions: 70 answerable and 30 unanswerable. “Useful” means a supported, relevant answer to an answerable question; unsupported answers are counted across all 100. Latency is the mean end-to-end time under identical test conditions.

The verifier is a second model pass that checks claims against supplied passages and rejects or revises unsupported content. Here it removes 3 unsupported answers relative to the instruction-only variant, but loses 1 useful answer and adds 1,170 ms. I would not select it by default for an interactive study assistant. That choice could change if an unsupported answer carries substantially greater harm than a refusal.

Inspect the individual transitions. Did the verifier fix an unsupported claim, replace it with an abstention, or reject a correct answer? Those are different outcomes. Record failure_reason values such as missing_evidence, unsupported_claim, and incorrect_abstention. Also record verifier timeouts separately: if a timeout silently becomes “insufficient evidence,” an infrastructure failure masquerades as caution.

¶Try to overturn the apparent win

For the preferred combination, remove one component at a time. If removing the reranker leaves the same corrected answers, its apparent contribution may have come from another change. Where generation remains variable, repeat paired runs rather than trusting one response per question. Keep the same question pairs in every comparison; otherwise, easier questions can manufacture an improvement.

Judge disagreement matters most near the decision boundary. Suppose reviewers disagree on 7 answers, including 2 of the reranker’s 3 additional useful answers. Have humans adjudicate those cases against the passages, with variant names hidden. Until that happens, report the gain as uncertain. I would not settle a grounding dispute by taking a majority vote from several similar model judges; shared blind spots can produce confident agreement.

Report latency for the complete path, including retrieval, checking, and any retries. If the verifier repeats after malformed output, those calls belong in both elapsed_ms and request_cost. A citation checker can reject an unknown source identifier, but a valid identifier does not establish that the passage supports the claim. That failure still needs semantic assessment.

Exercise: choose a maximum acceptable wait and an unsupported-answer limit before selecting a row. Then explain which choice changes if disputed answers are scored against your preferred variant. In an interview, “the judge score improved” is weaker than “the paired cases improved, human review confirmed the disputed gains, and the added delay met the application’s budget.”

The next chapter’s adaptation decisions need this frozen baseline and its unresolved failure cases. Otherwise, changing model weights risks becoming an expensive response to a retrieval problem.