¶How do corpus construction and pretraining objectives shape model behavior?
Your study assistant can produce a fluent, confident explanation that contradicts the course notes. To understand why, you must separate what pretraining rewards from what instruction tuning and preference optimization later change: sounding helpful is not the same as being correct.
I start with the training data, not the parameter count. Pretraining adjusts model weights using a large corpus—a collection of training documents—and a prediction objective. The corpus determines which patterns are available to learn; the objective determines which prediction errors drive updates. Neither automatically supplies a test for factual truth.
¶The corpus is already an engineering decision
A web crawl is not a balanced sample of human knowledge. It contains duplicated pages, navigation menus, outdated tutorials, copied answers, and languages represented at very different frequencies. Unless construction removes or reweights these patterns, they become part of the training signal. A widely copied misconception can receive more training exposure than a carefully written correction.
Corpus construction therefore includes selecting sources, filtering documents, removing duplicates, and choosing sampling proportions. These operations change behavior, not just storage requirements. If a small programming collection is sampled repeatedly, its influence can exceed its share of stored bytes. Conversely, an aggressive quality filter can discard useful informal explanations when its scoring rule mistakes unfamiliar language for poor writing.
I would inspect duplicates before celebrating another million documents. This runnable experiment isolates why repetition matters; it does not simulate neural-network training.
from collections import Counter
documents = [
"The exam permits calculators.",
"The exam permits calculators.",
"The exam prohibits calculators.",
]
def label_counts(texts):
return Counter(
"permits" if "permits" in text else "prohibits"
for text in texts
)
raw_counts = label_counts(documents)
unique_documents = list(dict.fromkeys(documents))
unique_counts = label_counts(unique_documents)
print("Raw:", dict(raw_counts))
print("Deduplicated:", dict(unique_counts))
raw_counts reports 2 occurrences of permits and 1 of prohibits; unique_counts reports 1 each. Counter counts the labels, while dict.fromkeys removes exact duplicate strings. With uniform document sampling, the repeated statement receives twice the exposure. Real pipelines often sample tokens or sequences instead, so document length and sampling weights also matter. Exact matching misses near-duplicates, and deduplication cannot decide which exam rule is true.
Another failure is evaluation contamination: if benchmark questions or answers enter training, a high score can reflect prior exposure rather than generalization. Keep the split discipline established earlier, but recognize that checking overlap becomes harder when the corpus contains paraphrases rather than identical strings.
¶The objective decides what counts as an error
For an autoregressive model, the next-token loss from Chapter 3 penalizes assigning too little probability to the observed continuation. A false sentence in the corpus still supplies target tokens. Unless another signal distinguishes accuracy from plausibility, predicting that sentence well lowers the loss. This is why declining training loss does not certify a reliable study assistant.
The original BERT work selected 15% of token positions for its masked-language objective, predicting those targets using surrounding context; it also used a next-sentence prediction objective. GPT-3, published in 2020 with 175 billion parameters in its largest model, instead used autoregressive prediction. Those objectives help explain their different uses, but do not establish that either model verifies facts.
I avoid treating “trained on textbooks” as a reliability guarantee. Textbooks can conflict, and training does not preserve a dependable document-to-answer audit trail. For the study assistant, missing course-specific rules can produce unsupported answers even when general subject explanations are strong. When a model’s corpus details are undisclosed, I cannot determine its exposure to your syllabus from fluent output alone.
Interview check: does adding 3 worked examples to a prompt repeat pretraining? No. GPT-3’s few-shot experiments used examples without gradient updates; the examples changed the input, not the weights. If prediction training explains fluent continuation but not dependable instruction following, what additional training signal teaches the model to answer the user’s request?
¶Why does BERT's masked language modeling differ from causal language modeling?
Give the study assistant “The exam is on” and ask it to continue. A causal language model is trained for that direction of work. Original BERT is not: its training prepares it to recover missing tokens using surrounding text. Treating those objectives as interchangeable leads to a model-selection mistake before you write any application code.
I choose the objective by the information available at prediction time, not by whether a model is described as “understanding language.” If the answer must extend an unfinished sentence, causal training matches that constraint. If a task involves interpreting a complete passage, bidirectional representations can be useful—provided you also have a suitable task-specific output layer.
¶The missing token can see its right-hand neighbours
In BERT’s 2018 pretraining setup, masked language modeling selects 15% of token positions as prediction targets. Of those selected positions, 80% are replaced with [MASK], 10% with a random token, and 10% remain unchanged. These percentages describe the original recipe, not requirements for every masked model.
For the study note “The exam is on Monday,” imagine selecting “Monday” as the target. BERT predicts the original token from the corrupted sequence. More revealing is masking “exam”: the model can use both “The” and “is on Monday.” Its attention is bidirectional within the permitted input, using the encoder structure from Chapter 4.
Only selected target positions contribute to the masked-token prediction loss; the other positions still supply context. The unchanged 10% are not an implementation error. Together with random replacements, they reduce reliance on an artificial marker that ordinary downstream text does not contain. They do not eliminate the mismatch between corrupted pretraining inputs and clean application inputs.
¶Causal training forbids the convenient clue
A causal model instead predicts each next token from its preceding tokens. When predicting “exam,” it cannot inspect “is on Monday.” Allowing that access during training would leak the answer-bearing future: training loss could look excellent while continuation quality fails. Shifting prediction targets is not enough unless the attention mask also blocks future positions.
This dependency-free experiment shows the information boundary, not model predictions. Each string is treated as one token for readability; actual tokenization may split it differently.
tokens = ["The", "exam", "is", "on", "Monday"]
target_index = 1
masked_input = tokens.copy()
masked_input[target_index] = "[MASK]"
causal_prefix = tokens[:target_index]
print("Target:", tokens[target_index])
print("Masked-model input:", " ".join(masked_input))
print("Causal-model input:", " ".join(causal_prefix))
assert masked_input[target_index] == "[MASK]"
assert causal_prefix == ["The"]
assert "Monday" not in causal_prefix
Here, target_index selects the same answer for both tasks, but masked_input and causal_prefix expose different evidence. Change the index to 4: the distinction becomes less obvious because the target is last. Test an interior position when checking your understanding.
Original BERT also used a next-sentence prediction objective. That historical detail matters, but it does not make BERT an autoregressive generator. Likewise, its masked predictions do not automatically define a left-to-right probability for an entire document. Comparing a masked-model score directly with causal-model perplexity can therefore produce a misleading ranking unless the scoring procedures account for their different conditioning.
¶What this changes for the study assistant
I would start with a causal model for composing multi-sentence explanations. Repeatedly inserting [MASK] into BERT is not a drop-in substitute: predictions at several masked positions do not automatically enforce a coherent generation order, and errors can leave the paragraph inconsistent. Conversely, choosing a causal model does not establish that its explanation is correct.
Interview check: “Does bidirectional attention make BERT better than GPT?” A defensible answer names the task and available context rather than declaring a winner. GPT-3’s 2020 paper studied an autoregressive model, including tasks supplied through text examples without gradient updates. But why can such a model continue an instruction convincingly while still failing to follow it?
¶How do instruction tuning and few-shot learning change behavior in different ways?
A study assistant can complete your question instead of answering it. Ask for an explanation of deadlock, and a pretrained model might continue with another examination question: that is a plausible continuation, but the wrong application behavior. I separate two remedies immediately: instruction tuning changes the model’s weights; few-shot learning supplies examples in its current input. Calling both “teaching the model” hides the distinction you need to debug them.
¶Instruction tuning changes the default behavior
Instruction tuning trains a model on demonstrations pairing requests with desired responses. In supervised fine-tuning, the optimizer updates weights to make those responses more likely. For a causal language model, the objective remains next-token prediction, typically with loss calculated on the response tokens rather than the request tokens. The difference is the training material: examples reward answering a request instead of merely continuing arbitrary text.
For the study assistant, demonstrations might pair “Explain deadlock briefly” with a concise explanation, or “Compare these concepts” with a comparison. After training, the changed weights influence later requests even when those demonstrations are absent. That persistence does not mean every instruction will be followed, nor that every learned behavior generalizes to unfamiliar subjects.
The 2022 InstructGPT study began with supervised demonstrations, then applied preference-based training. Its reported preference for a 1.3-billion-parameter InstructGPT model over 175-billion-parameter GPT-3 concerned the study’s human evaluations and prompt distribution. It was not evidence that supervised instruction tuning alone produced the entire improvement. I would not use that comparison to promise that a smaller current model will outperform a larger one on your course material.
¶Few-shot learning changes the input, not the weights
Few-shot learning supplies demonstrations inside the prompt. Zero-shot supplies no demonstrations; one-shot supplies 1; few-shot supplies several. In ordinary inference, there is no optimizer step. The model conditions its continuation on the examples through the attention machinery already covered, while its weights remain fixed.
The 2020 GPT-3 paper evaluated a 175-billion-parameter model this way, without gradient updates. “Learning” here names changed behavior within context, not a saved training update. Unless the application supplies those examples again, a later independent request does not inherit them through this mechanism.
This Python experiment constructs 2 inputs for an illustrative study-assistant task. It does not run a model or demonstrate an accuracy gain.
instruction = "Label the question as RECALL or COMPARE."
demonstrations = [
("What is a process?", "RECALL"),
("How does a thread differ from a process?", "COMPARE"),
]
question = "How does paging differ from segmentation?"
zero_shot = f"{instruction}\nQuestion: {question}\nLabel:"
examples = "\n\n".join(
f"Question: {text}\nLabel: {label}"
for text, label in demonstrations
)
few_shot = f"{instruction}\n\n{examples}\n\nQuestion: {question}\nLabel:"
for name, prompt in [("zero-shot", zero_shot), ("few-shot", few_shot)]:
print(f"--- {name} ---")
print(prompt)
Compare zero_shot and few_shot: only the input text differs. The demonstrations communicate the intended distinction and label spelling. An instruction-tuned model can still benefit from these examples; the techniques are complementary, not competing model categories.
I would start with a few demonstrations when an otherwise capable model misunderstands a narrow convention. They are cheap to revise, provided they fit within the context window. But if every example is labeled RECALL, a model may overproduce that label; if examples contradict each other, the intended rule becomes ambiguous. A response such as COMPARE can also be correctly formatted yet factually wrong when the task requires knowledge beyond classification. Neither technique verifies truth.
¶The interview distinction worth defending
If asked whether a model “learned from 3 examples,” answer by identifying where the examples went. Training examples can drive weight updates; prompt examples condition inference without those updates. Then ask what persists after the request ends. For an exercise, change one demonstration’s label and predict the possible failure before testing it. If human preferences guide further training instead, what behavior does that training actually reward?
¶What do reward models, RLHF, and direct preference optimization actually optimize?
A study assistant can become better at giving answers students like without becoming better at giving correct answers. A confident explanation may beat a cautious correction in a preference comparison, especially when the evaluator cannot check the mathematics. I treat preference optimization as training toward a particular judgment process—not as installing a truth detector.
¶From human comparisons to a learned score
After instruction tuning, a common reinforcement learning from human feedback (RLHF) pipeline collects several responses to each prompt and asks people to rank them. A reward model learns to score prompt-response pairs so that preferred responses receive higher scores. Its training data says “A beats B,” not necessarily “A is factually correct.”
A typical pairwise objective models the preference probability as sigmoid(reward_chosen - reward_rejected). If the scores are 2 and 1, that probability is approximately 0.731. Minimizing the negative log probability encourages a larger score gap. Neither score means “73.1% factually accurate”; only their difference enters this comparison.
For the study assistant, imagine 2 answers to “Does gradient descent always find the global minimum?” One correctly limits that guarantee to suitable conditions; the other says yes and supplies a polished explanation. If evaluators reward confidence over correctness, the reward model learns the wrong distinction. This is a label-quality failure, not a problem that more optimization automatically repairs.
¶RLHF optimizes the proxy, with a restraint
The language model being updated is called the policy. In a common RLHF setup, it generates responses, receives reward-model scores, and updates its weights to increase expected reward. Proximal Policy Optimization (PPO) is one algorithm used for those updates. A separate penalty discourages excessive departure from a frozen reference policy, often the instruction-tuned model.
That restraint commonly uses Kullback–Leibler (KL) divergence, a measure of distributional difference. Without sufficient restraint, the policy can exploit reward-model errors: verbosity, flattering agreement, or repeated phrases might increase predicted reward while making answers worse. This failure is called reward hacking. Strong restraint limits movement, but can also prevent useful changes.
I would inspect the ranking instructions before choosing an optimization algorithm. Unless evaluators are equipped to recognize unsupported claims, “helpful” labels can teach persuasive mistakes.
The 2022 InstructGPT paper reported that evaluators preferred its 1.3-billion-parameter model over 175-billion-parameter GPT-3 on the paper’s prompt distribution. That is evidence for the training procedure under those evaluation conditions, not evidence that a smaller aligned model always beats a larger model—or stops making factual errors.
¶DPO moves the comparison into the training loss
Direct preference optimization (DPO) avoids training a separate reward model and running a reinforcement-learning optimization loop in its standard offline form. It updates the policy directly using preferred and rejected responses plus a frozen reference policy.
The comparison uses each response’s sequence log probability under both policies. Define chosen_log_ratio as the policy’s log probability minus the reference’s for the chosen answer; define rejected_log_ratio similarly. The DPO loss is -log(sigmoid(beta * (chosen_log_ratio - rejected_log_ratio))), where beta is a positive coefficient connected to the reference-policy constraint in the derivation.
import math
beta = 0.1
policy_chosen, reference_chosen = -8.0, -10.0
policy_rejected, reference_rejected = -9.0, -9.0
chosen_log_ratio = policy_chosen - reference_chosen
rejected_log_ratio = policy_rejected - reference_rejected
margin = beta * (chosen_log_ratio - rejected_log_ratio)
loss = max(0.0, -margin) + math.log1p(math.exp(-abs(margin)))
print(f"Preference margin: {margin:.3f}")
print(f"DPO loss: {loss:.3f}")This arithmetic experiment prints a margin of 0.200 and loss of 0.598; it does not train a model. Increase policy_chosen to −7 and observe the loss decrease. The objective rewards a relative preference shift against the reference, not a standalone correctness score.
I would start with DPO rather than PPO for a bounded preference-training experiment, provided fixed comparison data is available; it removes substantial training machinery. It does not remove mislabeled preferences or distribution shift. Nor must the chosen answer’s absolute probability increase: its relative advantage can improve while both answers become less probable.
Interview question: If DPO loss falls while the study assistant makes more factual errors, has training failed? Answer by separating optimization success from application failure: the model may be learning the supplied preferences correctly. Which comparisons would you inspect first to find what those preferences actually reward?
¶Why can an instruction-following model still hallucinate, reproduce bias, or miss new facts?
Your study assistant can follow every instruction, write a polished explanation, and still invent the date of an examination. Nothing needs to crash. The failure is a plausible answer that a student acts on. I treat fluent instruction-following as a usability improvement, not a certificate of factual reliability.
¶A preferred answer is not necessarily a true answer
The objectives discussed earlier reward patterns and preferred behavior; they do not give the model a universal fact-checker. A hallucination is generated content that is false or unsupported by the available evidence, despite being presented as an answer. It can be an invented reference, a wrong calculation, or an explanation that quietly adds a condition absent from the course notes.
Preference optimization can reduce these failures when the feedback reliably distinguishes correct answers from convincing ones. But if a reviewer cannot verify a specialist claim, confident prose may receive the better rating. The model can then learn features associated with approval without learning the missing fact. This is a limitation of the signal, not proof that human feedback is useless.
The 2022 InstructGPT study makes the distinction concrete: evaluators preferred outputs from a 1.3-billion-parameter InstructGPT model over those from 175-billion-parameter GPT-3 on the study’s prompt distribution. The authors also reported improved truthfulness and reduced toxic output. They still reported simple mistakes. I would cite those findings as evidence that alignment can improve behavior, never as a current-model accuracy guarantee.
Lowering temperature does not repair this gap. It changes sampling, not the evidence available to the model. Unless the correct answer already receives sufficient probability, less randomness can simply make the same wrong answer appear more consistently.
¶Bias can survive polite wording
A model may avoid offensive language while still making unequal assumptions about people. In an illustrative study-assistant test, changing only a student’s name could change the difficulty of the recommended exercises. That would be a harmful difference even if both responses sounded encouraging.
Pretraining can absorb associations from the corpus; alignment can suppress some expressions while leaving other associations intact. Feedback itself also reflects the judgments and coverage of its contributors. Unless comparable cases are examined, a harmless-looking demonstration tells you little about behavior across different names, languages, or social groups. I would investigate a repeated difference, but not label one differing answer proof of systematic bias: sampling and wording can also affect the result.
¶New instructions do not create new knowledge
Suppose the project contains an old file, exam_schedule_2025.txt, and a replacement, exam_schedule_2026.txt. These are hypothetical project files, not model features. The replacement moves an examination from 12 May to 19 May. If the assistant receives neither document, asking it to “use the latest schedule” does not supply the missing date.
Even a published training cutoff is not a promise that every earlier fact was learned accurately. Coverage can be incomplete, contradictory, or uneven. Conversely, a model can use a newer fact supplied in its context without any weight update. Keeping those two mechanisms separate is essential.
For this example, record both published_at and effective_from: the newest publication is not necessarily the rule currently in force. A stale answer can otherwise become a missed examination rather than an obvious software error. Document access and source handling belong to the later retrieval chapters.
As an exercise, explain why “answer confidently using current information” fixes neither the absent schedule nor a biased assumption. An interview-quality answer names the missing evidence and the limits of the training objective, rather than blaming every failure on insufficient model size.
The next chapter needs this distinction: instructions can change the response without updating weights or establishing truth. Carry that boundary into application code before treating any generated answer as a dependable result.