Generation Is a Distribution, Not a Database Lookup

¶What makes a model generative rather than merely predictive?

If you treat the study assistant’s answer as a database result, you will mistake a plausible sentence for a retrieved fact. This chapter explains how models produce samples from learned distributions, why different sampling mechanisms matter, and how that process eventually becomes next-token generation.

Consider two jobs for the assistant. The first assigns a question to one of 3 subjects: databases, networks, or operating systems. The second writes an explanation of database indexing. Both involve prediction. The difference is not that one predicts and the other creates; the second uses predictions to construct an output whose possible forms are not restricted to those 3 labels.

A generative model learns a distribution over data, possibly conditioned on an input, from which a generation procedure can produce samples. Here, “conditioned” means that the distribution depends on supplied information: a question changes which answers are likely. In probability notation, a classifier might estimate p(label | question), while an answer generator models p(answer | question).

That distinction needs care. A classifier can sample a label rather than choose its highest-scoring label; randomness alone does not make it a useful content generator. Conversely, a generative model can produce the same answer repeatedly if its decoding procedure always chooses the highest-scoring continuation. I judge the system by what distribution it models and how outputs are constructed, not by whether a demo looks creative.

¶A distribution is not a correctness check

Suppose an illustrative generator assigns 3 candidate explanations probabilities of 0.6, 0.3, and 0.1. Sampling permits the third explanation even if it is wrong. Choosing the first every time removes that particular randomness, but it still fails whenever the highest-scoring explanation is wrong. Neither procedure checks a textbook.

The following standard-library experiment isolates sampling. It is not a trained language model: the answers and weights are handwritten. I prefer this small experiment before introducing a neural generator because it exposes a mistake that larger models can hide behind fluent prose.

import random
from collections import Counter

answers = [
    "An index can reduce the rows examined by a query.",
    "An index can speed reads but adds storage and write work.",
    "An index makes every query faster.",
]
weights = [0.6, 0.3, 0.1]

if len(answers) != len(weights):
    raise ValueError("Each answer needs one weight.")
if any(weight < 0 for weight in weights) or sum(weights) <= 0:
    raise ValueError("Weights must be nonnegative with a positive total.")

rng = random.Random(7)
samples = rng.choices(answers, weights=weights, k=1000)

for answer, count in Counter(samples).items():
    print(f"{count:4d}  {answer}")

random.Random(7) makes the experiment repeatable in your environment; it does not improve answer quality. rng.choices draws 1,000 samples, and Counter records their frequencies. Expect proportions near the supplied weights, not exactly 600, 300, and 100. If the counts differ slightly, that is sampling variation, not a broken implementation.

The third answer is an overgeneralization. Its nonzero weight lets the generator emit it. The checks raise ValueError for mismatched lengths or invalid weights, but no exception announces “this explanation is false.” That is the uncomfortable boundary: valid computation can produce invalid content.

¶The interview distinction worth keeping

Question: “Is next-token prediction merely classification, or is it generation?”

Answer: “At an individual step, the model scores possible next tokens. A generation procedure selects a token, appends it, and repeats with the updated input. Repeated conditional predictions construct a sequence, so classification-like steps can support generation.” Tokenization and that loop receive concrete treatment later in this chapter.

For an exercise, set the third weight to 0 and rerun the program. That answer becomes impossible in this fixed candidate set, provided the remaining weights stay positive. A language model does not generally offer such a convenient list of complete answers. What mechanisms let it represent and sample from a much larger space?

¶How do VAEs, GANs, and diffusion trade reconstruction, competition, and denoising?

A generated diagram can look convincing while omitting half the kinds of diagrams in the training set. Another model can cover those variations but blur every label. These are different failures, produced by different training objectives. I compare generative families by what their losses reward and how they turn randomness into a sample—not by their best-looking demonstration.

¶VAEs: reconstruct, but keep the latent space sampleable

If you compress a 28 × 28 image into 16 numbers and reconstruct its 784 pixels, you have an autoencoder. But arbitrary random numbers need not decode into useful images. A variational autoencoder, or VAE, addresses that problem by learning a distribution over compressed representations, called latent variables, rather than one fixed representation per input.

The encoder usually predicts a mean and variance. Training samples a latent vector using those predictions; the decoder reconstructs the input. There are 2 competing loss terms: reconstruction error and a Kullback–Leibler divergence penalty that encourages the encoder’s distributions toward a chosen prior, commonly a standard Gaussian. Generation samples from that prior and runs the decoder, without needing an input image.

The balance matters. Too little regularization can leave poorly supported regions in the sampling space; too much can cause posterior collapse, where the decoder largely ignores the latent vector. Pixelwise reconstruction objectives can also produce blur when several plausible outputs get averaged. I would start with a VAE for a compact, sampleable representation, provided exact visual detail is not the main requirement.

¶GANs: compete for convincing samples

A generative adversarial network, or GAN, replaces explicit reconstruction with competition. A generator transforms noise into samples; a discriminator learns to distinguish generated samples from training examples. The generator receives gradients through the discriminator and learns to make its outputs harder to distinguish.

The original formulation is a minimax objective, although practical variants change the losses. Once trained, the generator can produce a sample in one forward pass. That makes sampling attractive, but moves difficulty into training: if the discriminator becomes too confident under the original saturating generator loss, the generator can receive weak gradients.

Another failure is mode collapse: many noise inputs produce nearly identical outputs. A page of beautiful samples does not establish coverage of the training distribution. Unless you check diversity as well as realism, you can reward a generator for repeating its safest answer.

¶Diffusion: learn to undo controlled corruption

Diffusion training corrupts examples with noise, then teaches a network to reverse that corruption. In a common formulation, the network receives a noisy sample and its noise level and predicts the added noise. Sampling starts from noise and repeatedly applies a learned denoising procedure. Other formulations predict the clean sample or a related quantity.

This experiment shows only forward corruption—not a trained generator:

import numpy as np

rng = np.random.default_rng(7)
clean = np.array([0.0, 0.0, 1.0, 1.0])
noise = rng.normal(size=clean.shape)

for alpha_bar in (0.9, 0.5, 0.1):
    noisy = (
        np.sqrt(alpha_bar) * clean
        + np.sqrt(1.0 - alpha_bar) * noise
    )
    error = np.mean(np.square(noisy - clean))
    print(f"alpha_bar={alpha_bar:.1f}, squared error={error:.3f}")

np.random.default_rng fixes the random stream, and rng.normal supplies Gaussian noise. np.sqrt sets the signal and noise amplitudes; np.mean and np.square measure corruption. Decreasing alpha_bar reduces the signal contribution. Exercise: change the seed and check whether error must increase at every step for every individual sample.

The sampling cost depends on network size, resolution, and denoiser evaluations. A sampler making 50 evaluations does not have the same latency as one making 20; fewer evaluations can also damage quality. I would not implement any of these families merely to make the study assistant answer text questions.

An interviewer asks: “Why not denoise everything in one step?” A defensible answer is that heavy corruption permits many plausible originals; gradual sampling lets the model resolve uncertainty across intermediate states. What changes when the output is discrete text rather than continuous pixel values?

¶Why do tokenizers split familiar words and change the length of a question?

Your study assistant receives two questions that look equally short, yet one occupies more model input positions. Counting words will not explain the difference. Neither will counting Python characters. The model receives tokens: units selected by a tokenizer and represented by integer IDs. A token might cover a word, part of a word, punctuation, or bytes that do not form a readable character on their own.

I would inspect the tokenizer before shortening a student's question. “One token is about four characters” is a rough estimate for some English text, not a rule you can safely apply to equations, Python code, Hindi notes, or unfamiliar names.

¶Familiar to you does not mean indivisible to the model

A tokenizer has a fixed vocabulary, meaning its inventory of token units. Giving every possible word its own entry would leave new words, spelling variations, and identifiers uncovered. Subword tokenizers instead reuse smaller pieces. Frequent sequences can receive single entries; less frequent sequences remain split. A familiar technical word can therefore occupy several tokens without anything being broken.

Byte-pair encoding, or BPE, builds larger units through repeated pair merges. In a simplified byte-level version, text starts as UTF-8 bytes, and learned merge priorities determine which adjacent units combine. Real implementations add preprocessing rules and special tokens. Not every tokenizer uses BPE.

The experiment below supplies merge rules manually rather than learning them. It demonstrates the mechanism, not a production model's vocabulary. There are 256 possible starting byte values; the 9 rules add larger units. The encode function applies those rules in order, while decode reconstructs the text.

MERGES = [
    (b"s", b"t"),
    (b"st", b"u"),
    (b"stu", b"d"),
    (b"stud", b"y"),
    (b" ", b"n"),
    (b" n", b"o"),
    (b" no", b"t"),
    (b" not", b"e"),
    (b" note", b"s"),
]

VOCABULARY = [bytes([value]) for value in range(256)]
VOCABULARY += [left + right for left, right in MERGES]
TOKEN_TO_ID = {token: index for index, token in enumerate(VOCABULARY)}


def encode(text):
    pieces = [bytes([value]) for value in text.encode("utf-8")]
    for left, right in MERGES:
        merged = []
        index = 0
        while index < len(pieces):
            if (
                index + 1 < len(pieces)
                and pieces[index] == left
                and pieces[index + 1] == right
            ):
                merged.append(left + right)
                index += 2
            else:
                merged.append(pieces[index])
                index += 1
        pieces = merged
    return [TOKEN_TO_ID[piece] for piece in pieces]


def decode(token_ids):
    return b"".join(VOCABULARY[index] for index in token_ids).decode("utf-8")


for question in ("study notes", "Study notes", "study notes?", "पढ़ाई"):
    token_ids = encode(question)
    pieces = [VOCABULARY[index] for index in token_ids]
    print(repr(question), len(token_ids), pieces)
    assert decode(token_ids) == question

The first input produces 2 tokens: one for the lowercase word and one containing the leading space plus the second word. Capitalizing the first letter raises the count to 6 because the lowercase merge no longer matches. Adding a question mark produces 3 tokens. These are properties of this deliberately small vocabulary, not predictions about another tokenizer.

The Hindi example exposes a different boundary: a Python character can require multiple UTF-8 bytes. Unless merges combine those bytes, several tokens can represent one character. Calling decode("utf-8") on an isolated byte token can raise UnicodeDecodeError; joining the bytes before decoding avoids that failure here.

¶What I would check before changing the text

I would preserve the original question and inspect its actual model-specific token IDs. Replacing punctuation or stripping spaces merely to reduce a count can alter equations or destroy code indentation. Also, token IDs are vocabulary addresses, not universal meanings: using another tokenizer can silently assign different units to the same numbers.

A round-trip assertion catches lost text in this example. If you delete a vocabulary entry, lookup can instead raise KeyError. Neither check proves compatibility with a real model; that requires its matching tokenizer.

Interview question: Why can two equally long questions have different token counts? A useful answer connects vocabulary coverage, merge rules, whitespace, and byte representation—not just “tokens are parts of words.” For an exercise, add a second space and predict the result before running it. Once those units become integer IDs, what makes one ID useful to a neural network rather than merely a dictionary address?

¶How do embedding tables turn token identifiers into trainable vectors?

If you feed token identifier 400 into a network as the number 400, you introduce a relationship that the tokenizer never promised: that this token is numerically “larger” than token 40. Identifiers are addresses, not measurements. For the study assistant, two neighboring identifiers might represent unrelated fragments of course notes. I would never treat their numerical distance as evidence of meaning.

An embedding table replaces each identifier with a trainable vector. With vocabulary size V and vector width d, the table has shape (V, d). Looking up identifier 3 selects row 3; it does not search for a similar word or calculate a definition. A table with 10,000 tokens and 128 coordinates per token contains 1,280,000 trainable parameters. Those coordinates usually begin as random values unless you load already-trained weights.

¶A lookup with a visible update

I reach for torch.nn.Embedding rather than constructing one-hot vectors. Multiplying a one-hot vector by the table selects the same row, but materializing mostly zeros adds work without clarifying this experiment. The following example uses 6 vocabulary entries and 4 coordinates. Its identifiers are illustrative, not the output of a particular tokenizer.

import torch

torch.manual_seed(7)
table = torch.nn.Embedding(num_embeddings=6, embedding_dim=4)
optimizer = torch.optim.SGD(table.parameters(), lr=0.1)

token_ids = torch.tensor([[1, 3, 1]], dtype=torch.long)
before = table.weight.detach().clone()

optimizer.zero_grad()
vectors = table(token_ids)
print("Output shape:", tuple(vectors.shape))

# An artificial loss makes the gradient easy to inspect.
loss = vectors.sum()
loss.backward()

print("Gradient for row 1:", table.weight.grad[1].tolist())
print("Gradient for row 3:", table.weight.grad[3].tolist())

optimizer.step()
change = table.weight.detach() - before

assert torch.allclose(change[1], torch.full((4,), -0.2))
assert torch.allclose(change[3], torch.full((4,), -0.1))
assert torch.equal(change[0], torch.zeros(4))
print("Update checks passed")

The output shape is (1, 3, 4): one sequence, three token positions, four coordinates. Both occurrences of identifier 1 initially return identical vectors because they access the same parameters. This table alone cannot distinguish the first occurrence from the third, even if their surrounding text differs. Context-dependent representations require later computation; the Transformer chapter supplies that machinery.

The deliberately artificial vectors.sum() gives every returned coordinate a derivative of 1. Because row 1 appears twice, its accumulated gradient is 2 per coordinate; row 3 receives 1. After optimizer.step(), their coordinates decrease by 0.2 and 0.1 respectively. Row 0 stays unchanged under this plain SGD update because it received no gradient. That last claim needs the condition: optimizer settings such as weight decay or momentum can change the update behavior.

This loss teaches no language. In a useful model, the downstream task supplies the loss, and backpropagation adjusts the selected rows toward better predictions. The identifiers themselves remain integers; gradients update table.weight, not token_ids. Similar usage can encourage useful relationships between vectors, but neither a particular coordinate nor proximity guarantees a human-readable meaning.

¶Two failures worth catching early

With this table, identifier 6 is already out of bounds: valid identifiers run from 0 through 5. On CPU, that lookup raises IndexError: index out of range in self. Check the tokenizer’s vocabulary against the table size rather than clipping identifiers; clipping silently changes the input. Passing floating-point identifiers instead produces a RuntimeError because the lookup requires integer indices. Keep them as torch.long here.

A more dangerous failure is silent: pairing a tokenizer with the wrong table can return perfectly valid vectors for the wrong tokens. Matching dimensions is not enough; the identifier-to-token mapping must match too.

For an interview, explain why this is trainable despite being a lookup: row selection is discrete, but the selected row contains differentiable parameters. Then try replacing the final identifier 1 with 3. Before running the code, which row should now receive twice the gradient?

¶Can a tiny autoregressive model generate text through next-token prediction?

A generator can produce a plausible sentence without storing that sentence as a retrievable record. The revealing step is feeding its own output back into its input. One prediction becomes the condition for the next prediction; mistakes become conditions too. I prefer building this loop before introducing a larger architecture, because a fluent-looking result can otherwise hide what the program actually does.

For the study assistant, start with an intentionally inadequate model: predict the next word using only the current word. This is a bigram model, meaning its predictions depend on one preceding token. It is autoregressive because generation repeatedly conditions on previously generated output, although this particular model ignores everything except the latest token. It cannot answer questions from notes. It can expose the machinery that generates a sequence.

¶Fit the transitions, then sample them

The example uses PyTorch and a deliberately space-separated corpus, including spaces before punctuation. Its 16 tokens produce 15 input–target pairs and a vocabulary of 6 tokens. The first pair asks the model to predict support from notes; another asks it to predict learning from the same input. Neither target is uniquely correct.

The embedding has 8 dimensions. A linear layer converts that vector into 6 scores, giving the whole network just 102 trainable parameters. I use full-batch updates here: batching machinery costs more clarity than it saves on 15 examples. Run this in your existing PyTorch environment; no download or accelerator is required.

import torch
from torch import nn

torch.manual_seed(7)

text = (
    "notes support revision . "
    "notes support learning . "
    "revision supports learning . "
    "learning supports revision ."
)
tokens = text.split()
vocabulary = sorted(set(tokens))
token_to_id = {token: i for i, token in enumerate(vocabulary)}
ids = torch.tensor([token_to_id[token] for token in tokens],
                   dtype=torch.long)
inputs = ids[:-1]
targets = ids[1:]

model = nn.Sequential(
    nn.Embedding(len(vocabulary), 8),
    nn.Linear(8, len(vocabulary)),
)
optimizer = torch.optim.Adam(model.parameters(), lr=0.05)
criterion = nn.CrossEntropyLoss()

with torch.no_grad():
    initial_loss = criterion(model(inputs), targets).item()

model.train()
for step in range(400):
    optimizer.zero_grad()
    logits = model(inputs)
    loss = criterion(logits, targets)
    loss.backward()
    optimizer.step()

model.eval()
with torch.inference_mode():
    final_loss = criterion(model(inputs), targets).item()
    support_input = torch.tensor([token_to_id["support"]])
    probabilities = torch.softmax(model(support_input)[0], dim=-1)

    print(f"Training loss: {initial_loss:.3f} -> {final_loss:.3f}")
    print("Next-token probabilities after 'support':")
    for token, probability in zip(vocabulary, probabilities.tolist()):
        print(f"  {token}: {probability:.3f}")

    generated = ["notes"]
    for step in range(20):
        current = torch.tensor([token_to_id[generated[-1]]])
        probabilities = torch.softmax(model(current)[0], dim=-1)
        next_id = torch.multinomial(probabilities, num_samples=1).item()
        next_token = vocabulary[next_id]
        generated.append(next_token)
        if next_token == ".":
            break

print("Generated:", " ".join(generated))

CrossEntropyLoss receives raw scores, not probabilities; applying softmax before that loss would change the intended calculation. During generation, however, torch.multinomial needs nonnegative sampling weights, so the code converts scores into probabilities. The sampled identifier becomes the next input. Generation stops at a period or after 20 predictions, whichever comes first.

Inspect the probabilities after support. The training pairs split equally between revision and learning, so a successful fit should put roughly half the probability on each, not drive training loss to zero. Exact printed decimals can vary across environments. A fixed seed helps reproducibility; it does not promise identical arithmetic everywhere.

¶What this experiment does not prove

A decreasing training loss proves only that this model fits these transitions better. There is no held-out evaluation here because this experiment demonstrates a mechanism, not language quality. I would reject it as evidence that the study assistant understands course material. Even perfectly learned transitions can assemble a sequence absent from the corpus.

Two failures are worth provoking. Starting with exam raises KeyError because that token has no identifier; this vocabulary has no unknown-token fallback. Changing the input tensor to floating point makes nn.Embedding raise RuntimeError because embedding indices must be integer tensors. Neither failure is fixed by adding training steps.

In an interview, “Why not always choose the highest-scoring token?” deserves a mechanical answer: choosing the maximum discards alternatives, whereas sampling lets multiple continuations occur in proportion to their assigned probabilities. Neither method repairs missing context.

For an exercise, add both “exam notes support revision .” and “lecture notes support learning .” to the corpus and retrain. Inspect predictions after notes in both cases. They must be identical because the network receives the same identifier. What would have to change for the model to distinguish those two histories?

¶Why do temperature, context limits, and sampling prevent guaranteed answers?

Your study assistant can give the same wrong exam date on every run. Making its output repeatable has not made it correct. The next-token experiment exposes the reason: generation selects continuations from a learned distribution; it does not check each continuation against a course administrator’s calendar. I would start debugging an incorrect answer by checking what information the model received, not by adjusting its randomness.

¶Temperature changes confidence, not evidence

Temperature rescales the logits before the softmax calculation already covered. For a positive temperature temperature, divide each logit by that value. Values below 1 sharpen the distribution; values above 1 flatten it. If the highest-scoring token is wrong, sharpening makes that wrong choice more likely. No new evidence enters the calculation.

This standalone experiment holds 3 candidate logits fixed. It prints probabilities, then draws 12 samples at each temperature. The labels represent possible continuations, not actual predictions from a trained model.

import math
import random

labels = ["Monday", "Tuesday", "unknown"]
logits = [2.0, 1.0, 0.0]

def probabilities(logits, temperature):
    if temperature <= 0:
        raise ValueError("temperature must be positive")
    scaled = [value / temperature for value in logits]
    peak = max(scaled)
    weights = [math.exp(value - peak) for value in scaled]
    total = sum(weights)
    return [weight / total for weight in weights]

for temperature in (0.5, 1.0, 2.0):
    weights = probabilities(logits, temperature)
    rng = random.Random(7)
    draws = rng.choices(labels, weights=weights, k=12)
    print(temperature, [round(p, 3) for p in weights])
    print(draws)

The probability of Monday falls from approximately 0.867 to 0.665 to 0.506. Subtracting peak leaves the probabilities unchanged while avoiding unnecessarily large exponentials. The fixed random.Random seed makes this local experiment repeatable; it does not establish reproducibility across arbitrary model servers or software versions.

Try passing 0 to probabilities: it raises ValueError with temperature must be positive. Greedy decoding handles the zero-temperature idea separately by selecting a highest-scoring token rather than dividing by zero. I prefer greedy decoding for an initial comparison because it removes sampling variation, unless variation itself is what the experiment is testing.

¶Sampling removes possibilities, not mistakes

Top-k sampling keeps only the k highest-scoring candidates. Top-p sampling keeps a smallest high-probability set whose cumulative probability reaches a chosen threshold. Both renormalize the retained probabilities before sampling. These restrictions can exclude useful alternatives; neither knows which candidate is factual. Even greedy decoding chooses locally: taking the best-scoring token now does not guarantee the highest-probability complete sentence, much less a correct answer.

A practical exercise: swap the first and last logits, leaving the labels unchanged. Lowering temperature now favors “unknown.” The mechanism has no preference for honesty; your assigned scores determined the result.

¶A context window is a hard boundary, not perfect recall

Suppose an illustrative model permits 4,096 tokens across input and generated continuation. An input occupying 3,600 tokens leaves at most 496 for generation, provided that model uses this shared accounting rule. Actual input and output limits depend on the implementation. Exceeding a limit can cause request rejection; automatic truncation, if enabled, can instead discard the sentence containing the exam date. Reaching the generation limit can leave an answer unfinished.

Even when every note fits, inclusion does not guarantee that the model uses the relevant sentence correctly. A larger window increases what can be supplied, not what is guaranteed to be understood.

Interview question: “Does temperature zero guarantee a correct answer?” My answer is no: it changes token selection, not the evidence available or the learned scores. Distinguish correctness from repeatability before discussing settings.

The next chapter needs this distinction: token distributions describe the choices available, while the Transformer architecture explains how the preceding context shapes those choices.