¶Which failures call for better prompts, better retrieval, or changed model weights?
Fine-tuning the study assistant before locating its failures can make it more consistent at giving unsupported answers. This chapter uses the existing evaluation suite to decide whether training is justified, then limits the experiment to behavior that prompts and retrieval have not fixed.
I would not allocate a GPU because an answer “feels weak.” That description hides different defects: the required passage never reached the model, the instructions were ambiguous, or the model repeatedly ignored a clear requirement despite sufficient evidence. Each defect needs a different intervention.
¶Find the first broken stage
Take a failed question from the study assistant and inspect the retrieved passages, the assembled model input, and the answer. Use the stage-isolation tests from Chapter 10. A passage appearing somewhere in the search results does not count as available evidence unless it survives context assembly and reaches the generator.
For this investigation, save the question and expected evidence in diagnostic_cases.jsonl, retrieved passages in retrieved_context.jsonl, and generated answers in baseline_answers.jsonl. These are suggested artifact names, not library requirements. Keep the model revision and generation settings fixed; otherwise, a changed answer cannot reliably identify which intervention helped.
- Missing or wrong evidence: If the question asks about the revised examination policy but retrieval returns last year’s policy, repair ingestion, filtering, or ranking. Fine-tuning cannot reliably supply a document revision it has never received.
- Available evidence, unclear instructions: If the answer omits citations because the prompt merely says “be helpful,” first specify the citation requirement and provide a representative example. I reach for this cheaper intervention before constructing training data.
- Available evidence, persistent behavioral failure: If the model repeatedly invents citations or ignores the required answer structure despite clear instructions and examples, adaptation becomes a candidate—not an automatic remedy.
Some errors belong outside all 3 categories. An answer rejected with JSONDecodeError might reflect malformed model output, but it might also reflect your application truncating the response. Likewise, context_length_exceeded is an input-budget failure, not evidence that the weights need changing. Fix the application before training against its symptoms.
¶Run a small counterfactual experiment
Consider an illustrative diagnostic set of 40 questions, not a reported benchmark. Suppose 12 answers fail. Replace retrieved passages with manually verified evidence while leaving the prompt unchanged. If 9 failures disappear, those 9 cases point toward evidence delivery. Training on their desired answers would conceal the retrieval defect rather than repair it.
For the remaining 3 cases, keep the verified evidence and compare the original prompt against a clearer prompt containing 2 representative examples. If 2 cases now pass, only 1 remains unexplained. That is too little evidence to justify training: expand the diagnostic set and inspect whether the failure repeats across different topics and phrasings.
Record these interventions separately in intervention_results.jsonl, including evidence_present, format_valid, and citation_supported. Do not collapse them into one score. An answer can satisfy the schema while citing a passage that supports none of its claims.
¶Require a learnable behavior
Changing weights makes sense when you can state the desired behavior consistently, demonstrate it across representative inputs, and measure improvement on held-out cases. Fine-tuning adjusts the model’s output probabilities through training; it does not install an authoritative course database. I would keep changing course facts in retrieval and consider adaptation for stable conventions such as answer structure, terminology, or evidence-based abstention.
Even then, persistent failure may indicate insufficient model capability, not insufficient training. Compare a stronger baseline before committing, provided its cost is acceptable. More examples will not necessarily teach reasoning that the chosen model cannot perform reliably.
For an interview, explain why correct evidence plus repeated instruction failure motivates adaptation; missing evidence motivates retrieval repair. For your own experiment, select 10 failures and label the earliest broken stage. Which recurring behavior can you demonstrate in training examples without accidentally teaching the assistant to memorize the evaluation answers?
¶How do we construct adaptation examples without leaking the evaluation set?
A fine-tuned study assistant can appear better because it has rehearsed the exam. If a training answer is a paraphrase of a held-out answer, a higher score no longer demonstrates generalization. I would rather trust a modest improvement on untouched questions than a large improvement on contaminated ones. More training cannot repair that measurement.
Keep the existing final evaluation set frozen. Use development failures to identify behavior categories—unsupported citations, missed abstentions, inconsistent formatting—but write training examples from separate source families. If you inspect a final-test answer and then teach its correction, that item has become development data. Retire it from the final test; changing its wording does not restore independence.
¶Separate families before generating examples
For the study assistant, record example_id, source_family, question_family, parent_id, split, and the reviewed messages. A source family includes a PDF, its revised editions, and derivative notes. A question family includes paraphrases and synthetic descendants. Unless your evaluation explicitly measures new questions about familiar documents, keep both families within one partition.
I would start an illustrative pilot with 120 training examples, 30 validation examples, and 40 untouched test examples, provided enough independent families exist. Those counts are experiment sizes, not evidence of statistical adequacy. Assign families first; accept uneven row counts rather than splitting related examples to hit an 80/10/10 ratio. Validation guides checkpoint selection; the final test must not become another tuning loop.
This standard-library check catches family collisions before training. It raises ValueError with split leakage when a family crosses partitions:
def check_splits(rows):
owners = {}
for row in rows:
for field in ("source_family", "question_family"):
key = (field, row[field])
previous = owners.setdefault(key, row["split"])
if previous != row["split"]:
raise ValueError(f"split leakage: {key}")
rows = [
{"source_family": "transactions-v1",
"question_family": "atomicity-explanation", "split": "train"},
{"source_family": "networks-v1",
"question_family": "packet-loss-explanation", "split": "test"},
]
check_splits(rows)
print("Family checks passed")
Passing check_splits() proves only that the labels agree. It cannot discover an unlabeled copied paragraph. Normalize whitespace, compare exact-text hashes, and review suspicious near-duplicates across partitions. Semantic similarity can nominate pairs for inspection, but no universal similarity threshold proves independence. Two legitimate questions about different subjects may share nearly identical wording.
¶Teach the response, not the answer key
Each example should contain the instructions, retrieved evidence, user question, and desired assistant response in the same arrangement used at inference. Preserve source identifiers. If training supplies clean excerpts while production supplies noisy retrieval, the adapter learns under easier conditions than deployment provides. Include realistic distractors only when a reviewer can still determine the supported answer.
For example, supply [note-17] stating that atomicity makes a transaction all-or-nothing. Ask, “Does atomicity prevent concurrent transactions from interfering?” The target should distinguish atomicity from isolation and abstain from unsupported details. If isolation is absent from the evidence, do not silently import its definition into a supposedly evidence-only target. That produces a grounding failure even when the definition is correct.
I prefer a small reviewed collection to thousands of unchecked synthetic answers. A generator can multiply one mistaken citation into 200 examples; all 200 then reward the same error. Check answer support, citation resolution, and output format before admitting a row. Reject broken records with explicit reasons such as missing evidence or unresolved citation, rather than silently dropping context.
Interview check: Why is random row splitting insufficient? Because related documents and paraphrases can cross partitions, allowing memorization to masquerade as transfer. For an exercise, create 3 paraphrases of one training question and deliberately assign one to validation. Does your lineage record catch the contamination before any model sees it?
¶Why does LoRA train low-rank updates instead of every model parameter?
A training run can exhaust GPU memory before its first optimizer step, even when inference fits comfortably. Full fine-tuning needs gradients and optimizer state for every trainable weight, alongside the weights and saved activations. For the study assistant’s narrower goal—following an answer format and abstaining consistently—I would start with LoRA rather than pay that cost immediately.
Low-Rank Adaptation (LoRA) freezes the pretrained weights and learns an additive correction represented by two smaller matrices. It restricts the update, not the original model’s capacity. The frozen model still performs its ordinary computation; the correction changes how selected layers transform their inputs.
¶Replace one large update with two small matrices
For a linear layer, let W have shape (d_out, d_in). Full fine-tuning can change every entry. LoRA instead learns A, shaped (r, d_in), and B, shaped (d_out, r). Its effective weight is W + s * B @ A, where s scales the correction. The rank—the maximum number of independent directions in that correction—is at most r.
A 4,096-by-4,096 weight matrix contains 16,777,216 parameters. With rank 8, the two trainable matrices contain 65,536 parameters: 256 times fewer for this layer. That is not a 256-fold reduction in total training memory. The base weights remain resident, and activations still consume memory. If activations dominate, lowering rank may barely change the memory peak.
This executable PyTorch example uses smaller dimensions to expose the same computation. torch.nn.functional.linear applies each matrix without constructing the full correction during the forward pass.
import torch
import torch.nn.functional as F
torch.manual_seed(7)
d_in, d_out, rank = 32, 24, 4
scale = 1.0
W = torch.randn(d_out, d_in) # Frozen: requires_grad is False.
A = torch.nn.Parameter(torch.randn(rank, d_in) * 0.01)
B = torch.nn.Parameter(torch.zeros(d_out, rank))
x = torch.randn(3, d_in)
base = F.linear(x, W)
adapted = base + scale * F.linear(F.linear(x, A), B)
assert torch.equal(base, adapted)
target = torch.ones_like(adapted)
loss = F.mse_loss(adapted, target)
loss.backward()
assert W.grad is None
assert A.grad is not None
assert B.grad is not None
print("Full matrix parameters:", W.numel())
print("Adapter parameters:", A.numel() + B.numel())
print("Initial A gradient norm:", A.grad.norm().item())
print("Initial B gradient norm:", B.grad.norm().item())
The counts are 768 and 224. Because B starts at zero, the initial adapter changes nothing. Its gradient can nevertheless be nonzero. The initial gradient of A is zero because it passes through zero-valued B; after B changes, both matrices can learn. Initializing both matrices to zero would stall this branch: each matrix’s gradient would depend on the other zero matrix.
¶What freezing does—and does not—buy
Freezing means not updating the base weights; it does not mean removing the base model from backpropagation. Earlier adapters still need gradients propagated through later frozen operations. Wrapping the whole model in torch.no_grad() is therefore a footgun: it can disconnect the loss, producing RuntimeError with does not require grad in the message. Likewise, an optimizer that omits the adapter parameters can leave training apparently running while no intended weights change.
I treat rank as an experiment setting, not a quality dial. A larger rank permits more update directions, but does not guarantee better held-out behavior. LoRA’s usefulness rests on empirical evidence that useful adaptation often needs fewer directions than a full matrix permits—not a theorem that every task has a low-rank solution. If training loss falls while citation support deteriorates, the adapter has learned a regression, not an improvement.
For the study assistant, I would test a modest rank before increasing it, keeping the same evaluation suite. Applying adapters to more attention and feed-forward projections also expands capacity; unless those targets stay fixed, a rank comparison confounds two changes. Exact projection names depend on the architecture, so inspect the model rather than copying another model’s configuration.
At inference, compatible LoRA updates can be merged into ordinary weights, avoiding a separate adapter computation. That claim needs qualification for quantized weights, where merging may involve dequantization or requantization. Before choosing LoRA or its quantized-base variant, QLoRA, the next question is concrete: which memory allocation actually prevents your experiment from fitting?
¶How does QLoRA combine frozen quantized weights, NF4, double quantization, and paged optimizers?
A LoRA adapter can be small while its training job still exhausts GPU memory. The frozen base model must remain available during computation, and activations still consume memory. I would try ordinary LoRA first if it fits: fewer precision choices make failures easier to isolate. QLoRA becomes useful when base-weight storage prevents the study assistant’s adaptation experiment from running.
The mechanism is precise: store the frozen base weights in 4-bit form, reconstruct higher-precision values for computation, and backpropagate through those operations into the adapters. The base weights receive no optimizer updates. This is not training every parameter with 4-bit gradients. In a Transformers setup using BitsAndBytesConfig, load_in_4bit requests quantized loading, while bnb_4bit_compute_dtype selects the computation dtype. Storage precision and computation precision are separate decisions.
¶What NF4 and double quantization save
NF4, or 4-bit NormalFloat, uses 16 representable levels arranged for approximately normally distributed weights rather than evenly spaced levels. Quantization works on blocks: each block has a scale, and its weight values are represented using the quantization levels. NF4’s advantage depends on that distributional assumption; “optimal” does not mean optimal for every tensor or every task. The setting bnb_4bit_quant_type selects "nf4".
Those block scales also occupy memory. Double quantization compresses the quantization constants themselves, rather than repeatedly quantizing the model’s weights. In this integration, bnb_4bit_use_double_quant enables it. The extra saving comes from metadata, not another reduction from 4-bit weights to 2-bit weights.
For a hypothetical 7-billion-parameter model, raw 16-bit weights require approximately 14 GB; raw 4-bit codes require approximately 3.5 GB, using decimal units. Neither number is a training-memory estimate. Scales, unquantized tensors, adapters, gradients, optimizer states, activations, and temporary buffers remain. A checkpoint that loads successfully can still fail during backward computation with torch.OutOfMemoryError. I would budget from a complete training step, not a successful model load.
¶What paged optimizers do—and cannot do
Paged optimizers use unified-memory mechanisms to move optimizer-state pages between GPU and host memory when necessary. Their purpose is to absorb memory spikes rather than reserve all optimizer state permanently on the GPU. They do not make activations disappear. Unless the host has sufficient RAM and the backend supports paging, this escape route is unavailable; frequent transfers can also stall training.
The original QLoRA paper demonstrated a 65-billion-parameter model on a single 48 GB GPU. That is a reported experiment, not a promise for arbitrary context lengths, batches, or implementations. For the GPU path discussed here, you need a GPU supported by the installed quantization backend, compatible PyTorch and backend builds, sufficient device memory, and host RAM for any paging. Use bfloat16 computation only when your GPU and software support it. An import succeeding proves none of the remaining requirements.
I would run a hardware preflight with microbatch size 1 and a 512-token sequence, then test the intended maximum length. A backend mismatch may fail before training; insufficient host RAM can kill the process during paging; an unsuitable precision configuration can produce a non-finite loss. Treat each as a different diagnosis, not a reason to keep lowering adapter rank.
¶One experiment worth recording
For the study assistant, hold the examples, adapter configuration, and evaluation questions fixed. If both fit, compare ordinary LoRA against QLoRA and record peak device memory, step time, and answer-quality regressions. No universal slowdown figure is defensible without that hardware and workload.
An interviewer asks, “Why can QLoRA still run out of memory?” Your answer should separate compressed frozen weights from uncompressed training allocations. Then test your explanation: does doubling sequence length break the run even though the stored weights have not changed?
¶Can a small adaptation improve study-answer format without damaging general behavior?
A study assistant can learn your preferred headings while becoming less willing to admit that the notes contain no answer. That is a failed adaptation, even if its responses look more polished. I would accept a small LoRA adapter only when it improves the requested behavior without introducing unacceptable failures on the existing evaluation suite. Lower training loss does not settle that question.
Keep the experiment narrow: produce an answer, supporting citations, and a short revision question in the required order. Do not simultaneously change retrieval, the system prompt, and model weights. Otherwise, an improvement has no clear cause. Compare the frozen base model against that same model with the adapter enabled, using identical retrieved passages, chat templates, generation limits, and decoding settings. For QLoRA, compare against the same quantized base without the adapter; otherwise, quantization becomes another variable.
¶Measure the behavior you wanted—and the behavior you might lose
For a first comparison, I would reserve 50 held-out cases: 30 formatting cases, 10 ordinary instruction-following cases, and 10 questions whose answers are absent from the supplied notes. This is an illustrative experiment design, not a measured result or a sufficient production benchmark. Keep checkpoint selection on validation data; inspect this test set only after choosing the candidate.
Score format separately from factual support. A response can pass the heading check while inventing a citation. Record malformed output, unsupported claims, inappropriate answers instead of abstentions, and instruction-following regressions as distinct failures. If generation ends at max_new_tokens, record truncation rather than silently treating the missing final heading as evidence of poor learning. Inspect the generated token count and stopping condition first.
The following dependency-free Python example exercises the comparison logic with fabricated pass/fail fixtures. It performs no model inference. Replace these fixtures with judgments from your existing evaluator before drawing any conclusion about an adapter.
def compare_candidate(baseline, candidate):
if baseline.keys() != candidate.keys():
raise ValueError("evaluation groups differ")
accepted = True
for group, before in baseline.items():
after = candidate[group]
if not before or len(before) != len(after):
raise ValueError("paired case counts must match and be nonzero")
gains = sum(not old and new for old, new in zip(before, after))
regressions = sum(old and not new for old, new in zip(before, after))
print(
f"{group}: {sum(before)}/{len(before)} -> "
f"{sum(after)}/{len(after)}; "
f"gains={gains}, regressions={regressions}"
)
if group == "format":
accepted = accepted and (gains - regressions >= 3)
else:
accepted = accepted and regressions == 0
return accepted
baseline = {
"format": [True] * 24 + [False] * 6,
"general": [True] * 10,
"abstention": [True] * 10,
}
candidate = {
"format": [True] * 28 + [False] * 2,
"general": [True] * 9 + [False],
"abstention": [True] * 8 + [False] * 2,
}
print("Accept candidate:", compare_candidate(baseline, candidate))
Here, compare_candidate rejects the candidate despite formatting improving from 24/30 to 28/30. The general group loses 1 pass, and abstention loses 2. The gains and regressions counts matter because unchanged aggregate accuracy can conceal newly broken cases. The ValueError checks catch mismatched groups or case counts, but your evaluation records must also preserve case identifiers and ordering.
I prefer this conservative decision rule for a student’s first adaptation: require at least 3 additional formatting passes and no new failures in the protected groups. Those thresholds are experiment choices, not universal constants. With only 10 abstention cases, zero observed regressions does not establish that general behavior is undamaged. It establishes something narrower: this candidate survived these checks.
¶Investigate the rejected candidate
If formatting improves but abstention deteriorates, inspect the training examples before increasing rank or training longer. A dataset containing only confident answers teaches an unfortunate association: every question deserves a completed answer. Include permitted, correctly formatted examples where insufficient evidence is the right outcome, then repeat validation. Keep the final test set untouched during that repair.
As an exercise, change the fabricated candidate so aggregate instruction-following accuracy stays constant while one previously passing case fails. Your comparison should still reject it. For an interview explanation, I would say: “I adapted response structure, held retrieval constant, and checked paired regressions—not just average accuracy.” That describes an engineering decision rather than a training command.
The next chapter needs a documented model choice, a reproducible baseline, and known failure cases from this experiment—not an adapter accepted merely because training completed.