Turn the Study Assistant into Evidence You Can Defend in an Interview

¶What must a reviewer be able to reproduce from the finished repository?

A polished study-assistant demo proves little if a reviewer cannot rebuild its index, rerun its tests, or trace an answer to its source. This chapter turns the application into defensible evidence: a repository that supports your claims even when you are not there to explain it.

I would publish a smaller, reproducible system before adding another agent tool. A reviewer needs to establish what works, under which conditions, and what still fails—not admire a feature list. The repository should answer 3 questions: can I run it, can I check its answers, and can I verify the trade-offs?

¶Make the first run independent of your laptop

Start README.md with the supported Python version, installation commands, and the shortest path to one cited answer. State whether that path requires network access, credentials, or a GPU. “Runs locally” is misleading if document processing quietly calls a paid service.

I prefer 2 explicitly separated execution paths: an offline fixture run and a live-model run. The fixture path should exercise retrieval, context assembly, response validation, and citation rendering without credentials. It proves application wiring, not model quality. The live path tests the external dependency, provided the reviewer has access to the configured model. Never silently substitute fixtures after an authentication failure.

Commit .env.example with empty credential fields, not working secrets. Record dependency pins, model identifiers or checkpoint revisions, embedding configuration, and index-building instructions. A missing credential should produce an actionable configuration error before document processing starts; a download timeout should identify the failed dependency. Neither should appear as an empty answer.

¶Give every claim an evidence trail

Include a small, redistributable corpus in data/sample/. Self-authored notes are sufficient; copyrighted lecture PDFs are not a portfolio requirement. Preserve source identifiers so a reviewer can follow a citation back to the exact passage. Unless you have redistribution permission, publish synthetic substitutes and explain which original-data results cannot be reproduced.

Keep evaluation questions in eval/questions.jsonl, with expected evidence and answerability labels. For a compact review pack, choose 12 cases: 6 answerable questions, 3 unanswerable questions, and 3 adversarial or malformed-input cases. These are suggested coverage counts, not evidence that 12 cases establish production quality.

Store a run manifest in results/run.json: commit identifier, corpus digest, evaluation-set version, prompt revision, retrieval settings, model configuration, and execution date. Link it to per-question outputs. Otherwise, an aggregate score can survive long after the code that produced it has disappeared.

Publish the numerator and denominator behind each quality claim. “9 of 12 passed this rubric” is inspectable; “highly accurate” is not. Include unsupported citations and incorrect abstentions among the failures. If a source disappears during indexing, report the ingestion failure rather than calculating retrieval quality against a silently reduced corpus.

¶Separate replay from a fresh measurement

Saved responses let a reviewer replay scoring; fresh requests test current behavior. They are different evidence. Even with fixed sampling settings, I would not promise identical hosted-model answers. Record observed variation, and label latency and cost measurements with hardware, concurrency, cache state, retry policy, and pricing date.

An interviewer may ask, “Does your offline test prove the assistant is grounded?” My answer would be: “No. It proves the plumbing handles known responses correctly. Grounding needs source-support evaluation on actual generated answers, with inspectable failures.”

Before publishing, clone into a clean directory and follow only README.md. Then deliberately remove one credential and one source file. Does the repository explain both failures without your help?

¶How do architecture decision records explain why you rejected more complex options?

A repository can reproduce an answer without explaining why its architecture deserves to exist. In an interview, “I used a vector database because it scales” invites the obvious follow-up: scales to what? If your study assistant searches 2,000 chunks on one machine, an extra service may demonstrate deployment work rather than sound judgement.

I prefer a small architecture decision record, or ADR, for each consequential choice. An ADR preserves the constraints, alternatives, evidence, and consequences behind a decision. It is not a diary of every library installation. Put records in docs/decisions/, using names such as 0001-local-retrieval.md, and link them from README.md.

¶Write the rejected option fairly

Consider this illustrative record for the study assistant. The figures below are proposed requirements, not benchmark results. Your published record must distinguish requirements from measurements just as explicitly.

Notice what the record does not claim: local search is universally faster, or vector databases are unnecessary. Exact search still performs work proportional to corpus size and embedding dimension. A remote service can fail with a timeout; a local index can fail by exhausting memory. You have selected a failure profile, not abolished failure.

I would reject the separate service here unless shared access or measured capacity pressure justifies it. “Simpler” is defensible only when you name the obligations removed and accept those retained. Local storage still needs recovery, version checks, and protection against loading an index built with a different embedding model.

¶Attach evidence without laundering assumptions

Add links to eval/retrieval_cases.jsonl, reports/retrieval_baseline.json, and data/corpus_manifest.json once those artifacts actually exist. Record the commit, machine, corpus size, embedding revision, query count, and warm-up policy. A report over 40 queries can describe that run; it cannot establish behavior for every course. If no comparison was run, write not evaluated, not “slower.”

Include a Consequences heading. For this choice, a stale manifest can produce incorrect citations even while search succeeds. If a release check detects a mismatch, document its actual error message and recovery procedure. Do not invent diagnostic output for an unimplemented safeguard.

¶Defend the boundary, not the technology

Interview question: “Why didn’t you use an agent to answer questions?”

Model answer: “The required path is retrieval followed by grounded generation. I chose a fixed workflow because each stage has an independently testable output. An agent would add tool-selection and stopping failures without a demonstrated task requirement. I would reconsider if evaluated questions required choosing among several tools or recovering through alternative actions.”

For practice, write a second record, 0002-no-fine-tuning.md, using your existing evaluations. Separate a retrieval miss from an answer-format failure before proposing adaptation. Which failing cases would actually change your decision?

¶Can you answer foundation questions by tracing tokens, attention, training, and inference?

“The model uses Transformers” will not survive a follow-up about why your study assistant cited the wrong paragraph. A defensible answer traces the question through tokenization, contextual representations, next-token scores, and decoding, then identifies where evidence could have been lost. I would practise that trace before memorizing another architecture diagram. Your repository already supplies the concrete system; use it to make foundational answers testable.

¶“What happens between my question and the first generated token?”

Model answer: “The application assembles instructions, retrieved passages, and the question using the checkpoint’s expected chat format. The tokenizer produces token IDs, not one ID per word. The decoder processes those IDs with positional information and causal attention. The final position’s output produces vocabulary scores; the decoding policy selects the next token. Generation then continues until a stopping condition applies.”

Follow-up: “What if the input contains 3,000 tokens and you reserve 500 output tokens?” Your answer must check the model and serving limits, including special tokens. A nominal 4,096-token combined limit leaves 596 tokens of headroom under those assumptions, not permission to append unlimited retrieval results. Unless the endpoint documents different accounting, budget the complete serialized input. An oversized request may produce a context-length error; automatic truncation can instead discard the passage needed for the answer.

¶“Why does causal masking matter during training?”

Model answer: “At each position, the decoder predicts the following token without attending to later input positions. Training can evaluate many positions in parallel because their targets are already available. That does not let generation produce an entire dependent answer in one forward pass.”

This small check uses illustrative IDs, not a real tokenizer. Run it with Python 3:

token_ids = [11, 24, 37, 52]
input_ids = token_ids[:-1]
labels = token_ids[1:]

allowed_attention = [
    [column <= row for column in range(len(input_ids))]
    for row in range(len(input_ids))
]

assert list(zip(input_ids, labels)) == [(11, 24), (24, 37), (37, 52)]
assert allowed_attention[0] == [True, False, False]
assert allowed_attention[2] == [True, True, True]
print(list(zip(input_ids, labels)))
print(allowed_attention)

Explain why input_ids and labels differ by one position, and why allowed_attention permits the diagonal. In an actual training library, check whether the model shifts labels internally before preparing shifted targets yourself. Shifting twice silently trains the wrong prediction task. Removing the causal mask leaks future information; a reassuring validation loss then measures a task unavailable at inference.

Follow-up: “Does attention explain the answer?” I would not present an attention heatmap as proof of factual grounding. Attention weights describe a computation, not a verified citation. For ordinary dense attention, doubling sequence length from 1,024 to 2,048 creates 4 times as many query–key pairs per head. That is not a guaranteed 4× latency increase: kernels, batching, and hardware change the measured result.

¶“What changes during training, and what changes during inference?”

Model answer: “Training updates the selected parameters through an optimization objective. During ordinary inference, weights stay fixed while activations and the key–value cache change. Retrieved notes change the input, not the weights. Supervised instruction tuning teaches desired responses; preference optimization changes behavior toward preferred outputs, without guaranteeing truth.”

Expect implementation follow-ups. model.eval() changes training-dependent behavior such as dropout; torch.no_grad() disables gradient recording. Neither substitutes for the other. generate() orchestrates decoding, but its name does not establish your stopping policy. If local inference throws CUDA out of memory, inspect input length, batch size, weights, and cache allocation before blaming model quality.

For practice, answer each question aloud in 90 seconds, then change one assumption: remove retrieval, double the context, or enable adapter training. Which recorded test would distinguish your predicted behavior from a plausible but incorrect explanation?

¶How would you debug a failing RAG or agent system during a design interview?

“I would improve the prompt” is a weak answer when the study assistant cites the wrong semester’s exam policy. The model may have followed its instructions perfectly while reading an obsolete document. I start with the first observable divergence from expected behavior, not the most fashionable component to replace.

In an interview, ask for one failing question, its expected evidence, and the request trace. Clarify whether “failing” means an incorrect answer, an unauthorized action, or a timeout. Those failures require different investigations. If production traces contain student information, request redacted identifiers and stage metadata rather than copying raw notes into another debugging service.

¶Work backward from the answer, then test forward

Consider this illustrative failure: the assistant answers “14 days,” but the current course policy says “7 days.” The trace contains request_id, index_version, retrieved_ids, context_ids, and cited_ids. These are application-defined fields, not a tracing library’s required interface.

Model answer: “I would verify the expected policy first, including its effective date and course. Then I would inspect whether its chunk survived ingestion, retrieval, filtering, and context assembly. If the correct evidence reached the model, I would investigate instruction conflicts and answer grounding. I would change one stage at a time and replay the failure before running the regression set.”

Suppose the correct chunk appears among 8 retrieved candidates but disappears from the final 3 context chunks. That localizes the first demonstrated defect to filtering, reranking, or context assembly—not necessarily embedding quality. A minimal fixture makes the distinction executable:

trace = {
    "expected_id": "policy-current-7",
    "retrieved_ids": ["policy-old-14", "policy-current-7", "calendar"],
    "context_ids": ["policy-old-14", "calendar"],
}

expected = trace["expected_id"]
if expected not in trace["retrieved_ids"]:
    print("Inspect ingestion, filters, and retrieval.")
elif expected not in trace["context_ids"]:
    print("Inspect reranking and context assembly.")
else:
    print("Inspect evidence sufficiency and generation.")

The fixture prints the second diagnosis. It does not prove reranking is broken; a legitimate token budget could have removed the chunk. Inspect scores, document versions, and the actual assembled messages. Unless you inspect those messages, “retrieval succeeded” can hide a context-packing failure.

Follow-up: “Would increasing the context window fix it?” My answer is no—not without evidence. More space helps only when truncation removed useful evidence; it can also preserve conflicting policies. I prefer correcting version selection before paying for more input tokens. If the service instead returns context_length_exceeded, inspect the provider’s actual error contract and token accounting; that string is an illustrative label, not a universal API error.

¶Separate agent decisions from execution failures

Now suppose the assistant creates the same study reminder twice after a timeout. Ask whether the tool failed or merely failed to return a response. If the remote write succeeded before the connection dropped, retrying can duplicate the action. I would inspect tool_call_id, the application’s idempotency_key, execution records, and persisted state before authorizing another write.

A validation failure such as invalid_tool_arguments belongs before execution. An authorization denial belongs at the tool boundary. Neither becomes safe because the model retries 3 times. When the execution outcome is unknown, reconcile with the remote system if supported; otherwise stop and expose the uncertainty rather than claiming success.

Follow-up: “What proves your fix worked?” Replay the failing case, then test adjacent cases: an obsolete policy, missing evidence, a denied tool call, and a response lost after a successful write. Compare answer support, duplicate actions, latency, and cost against the unchanged baseline. One repaired answer is evidence of a repair, not evidence of system-wide quality.

Exercise: Give yourself 90 seconds to explain the duplicate-reminder failure without saying “better prompt.” Which observation would distinguish a model choosing the tool twice from one tool execution being retried twice?

¶Which portfolio extensions and learning milestones should guide your next ninety days?

A portfolio can become less convincing as you add features. A study assistant with voice input, three models, and an autonomous workflow gives an interviewer more buttons to press, but not necessarily more evidence that you can make engineering decisions. I would spend the next ninety days improving one demonstrated weakness, then making the improvement easy to inspect. Feature count is a poor substitute for a defensible result.

¶Days 1–30: choose a weakness you can measure

Allocate a realistic budget first: for example, 6 hours each week and a spending ceiling you can afford without relying on free credits. These are planning constraints, not estimates of how long mastery takes. Put your chosen question in ROADMAP.md: “Can the assistant answer questions from scanned lecture notes without losing source traceability?” is narrow enough to finish. “Build a multimodal learning platform” is not.

Choose the extension only after inspecting failures. If relevant pages never enter the retrieved context, investigate retrieval rather than fine-tuning. If scanned pages produce empty text, document extraction is the immediate problem. If answers cite the wrong page despite receiving the right evidence, improving search alone will not repair citation handling.

Create a development set of 30 permitted questions spanning ordinary cases, difficult cases, and questions the notes cannot answer. Keep a separate held-out set for the final comparison; unless it stays untouched during development, it cannot support an honest final check. Record the question categories in evaluation_cases.jsonl and your initial observations in baseline_results.csv. This is a small diagnostic sample, not proof of general reliability.

¶Days 31–60: build one extension, including its failure path

I would choose scanned-note support when extraction failures dominate, a retrieval comparison when relevant passages are missed, or a read-only practice-quiz tool when grounded answering already works. Avoid adding an agent merely because job descriptions mention agents. A bounded workflow is easier to defend unless the task genuinely requires choosing actions from intermediate results.

For the scanned-note extension, follow one document through ingestion, recognition, retrieval, and citation display. Save page-level observations in extraction_report.json. Recognition may turn a minus sign into a dash, causing a mathematically wrong answer even though the retrieved page is correct. An unreadable page should produce an explicit extraction failure, not silently disappear. If processing reaches your configured timeout, preserve the document’s failure status rather than presenting partial ingestion as success.

Write the acceptance rule before inspecting final results in acceptance.md. One illustrative rule is to improve supported answers on scanned questions without introducing new unsupported answers on the ordinary-text cases, while keeping median processing time below your stated budget. Report raw counts alongside percentages: moving from 6 to 8 successes among 10 scanned questions is useful evidence, but too little evidence for a broad accuracy claim.

¶Days 61–90: turn the result into a learning decision

Run the held-out comparison, record limitations in extension_report.md, and ask another developer to challenge one claim. If the extension fails its acceptance rule, keep the negative result. I value an explained rejection more than a polished demonstration that hides regressions.

Practice this interview question: “Why did you postpone fine-tuning?” A defensible answer is: “The failing questions lacked readable source text. I tested extraction first because changing model weights would not restore missing evidence.” The follow-up is harder: “What result would change your decision?” Identify a remaining behavior failure that persists when correct evidence is supplied, then describe the controlled comparison you would run.

Your next learning milestone should follow that unresolved failure, not a tool popularity ranking. The next chapter of your work needs one bounded question, a runnable starting point, and evidence clear enough that someone else can disagree with your conclusion.