¶How do a Python service, container, health check, and release gate make deployment repeatable?
An assistant that passes evaluation on your laptop can still fail after deployment because the container loads an old retrieval index, lacks credentials, or accepts traffic before initialization finishes. This chapter turns the study assistant into a service whose release depends on observable behavior, including what each answer costs.
I would start with one Python service and one container image, not a collection of independently deployed agents. Separate services are worth their operational cost only when you need independent scaling, isolation, or ownership. For this assistant, fewer deployment boundaries mean fewer places for configuration to disagree.
¶Package the process, identify the artifacts
Expose the existing assistant through a small HTTP application. I would use FastAPI here: request validation and explicit response schemas fit the contracts already established for questions, citations, and errors. Keep retrieval and generation outside route handlers so the evaluated implementation remains the deployed implementation.
Run the application with an ASGI server such as Uvicorn, binding to 0.0.0.0 inside the container. Binding to 127.0.0.1 can leave the process running while connections through the published container port fail. Start with 1 worker unless load testing justifies more: 4 workers can load 4 separate copies of an in-memory retrieval index.
Build the image from Dockerfile using your pinned dependencies, a fixed Python minor version, and a base-image digest. Exclude credentials, local environments, and private PDFs through .dockerignore. Run as a non-root user. Supply secrets at runtime; deleting a credential in a later image layer does not remove it from earlier layers.
A container alone does not make generation reproducible. Record the image digest, prompt revision, model identifier, and index version together. Unless the provider offers a pinned model revision, identical application artifacts still cannot guarantee identical model behavior. I would promote the same built image between environments rather than rebuild it after approval.
¶Ask two different health questions
Give /live and /ready different jobs. The first returns HTTP 200 when the process can handle a lightweight request. The second returns HTTP 503 until required local initialization succeeds: configuration is validated, the expected index is loaded, and its version matches the release record.
Do not generate an answer on every health probe. A probe every 10 seconds creates 8,640 checks per day per replica; paid model calls turn monitoring into background expenditure. A provider timeout should normally produce a bounded request failure, not a liveness failure that repeatedly restarts an otherwise healthy process.
Readiness is not an evaluation score. A corrupt index should produce a startup error and keep readiness false; an unsupported answer may arrive with HTTP 200. Only an evaluation gate catches the second failure.
¶Make rejection executable
The following runnable Python example checks illustrative release measurements. Its limits are policy examples, not measured results or universal targets. In continuous integration, load the candidate’s evaluation report and apply this function before promotion.
import math
def release_gate(report):
limits = {
"citation_support": (0.95, 1.0),
"p95_seconds": (0.0, 4.0),
"usd_per_answer": (0.0, 0.02),
}
for name, (minimum, maximum) in limits.items():
value = report.get(name)
if (
isinstance(value, bool)
or not isinstance(value, (int, float))
or not math.isfinite(value)
or not minimum <= value <= maximum
):
raise ValueError(f"release blocked: {name}")
if report.get("critical_failures") != []:
raise ValueError("release blocked: critical_failures")
candidate = {
"citation_support": 0.97,
"p95_seconds": 3.2,
"usd_per_answer": 0.012,
"critical_failures": [],
}
release_gate(candidate)
print("Candidate passed the example policy.")
I prefer explicit rejection to a dashboard that somebody must remember to inspect. Missing metrics must block promotion, not silently become zero. Apply the gate only when the report belongs to the candidate’s exact artifact set and uses the agreed evaluation dataset; otherwise an old passing report can authorize new, untested behavior.
For a rehearsal, change p95_seconds to 4.1 and confirm that ValueError stops the release. Then remove critical_failures and verify rejection again. These checks establish the deployment boundary. What measurements should the running assistant emit so that a passing release cannot hide an expensive regression?
¶What would I trace first when an answer is slow, expensive, or unsupported?
A request log saying “completed in 8 seconds” tells you that somebody waited. It does not tell you whether retrieval stalled, the provider throttled a request, or your retry loop paid for another generation. I would instrument the study assistant’s stage boundaries before adding another performance dashboard. A dashboard cannot recover events your service never recorded.
Start with one trace_id for the incoming request. A trace connects its operations; a span records one operation’s start, duration, outcome, and selected attributes. Give retrieval, reranking, generation, and validation separate spans, with child spans for individual provider attempts. Preserve the identifier across background jobs and service calls, unless doing so would cross a trust boundary that requires a new identifier.
¶Record enough to explain the answer
My minimum record includes release_id, prompt_version, model_id, index_version, and retrieved chunk identifiers. Otherwise, a regression after deployment becomes guesswork: did the code change, did the notes change, or did the model change? Store document revisions alongside chunk identifiers, because an identifier alone cannot reconstruct evidence after reindexing. Record retrieval scores only with their scoring method; scores from different retrievers are not automatically comparable.
I avoid logging complete prompts by default. Course notes, student questions, and tool results can contain private material, and an observability vendor is another data recipient. Keep restricted evidence references and access-controlled snapshots only when your permissions and retention policy allow them. Hashing a student’s email does not automatically make the resulting record anonymous.
This standard-library example demonstrates timing and failure recording, not a production tracing backend. Its 2-chunk fixture deliberately fails the support check; the printed duration comes from your run, not a benchmark.
import json
import time
import uuid
from contextlib import contextmanager
trace_id = uuid.uuid4().hex
events = []
@contextmanager
def span(name):
started = time.perf_counter()
event = {"trace_id": trace_id, "stage": name, "status": "ok"}
try:
yield event
except Exception as exc:
event["status"] = "error"
event["error_type"] = type(exc).__name__
raise
finally:
event["duration_ms"] = round(
(time.perf_counter() - started) * 1000, 3
)
events.append(event)
try:
with span("retrieval") as event:
chunks = ["networks:rev3:12", "networks:rev3:13"]
event["chunk_ids"] = chunks
with span("support_validation") as event:
cited_chunk = "networks:rev3:99"
event["citation_present"] = cited_chunk in chunks
if not event["citation_present"]:
raise ValueError("citation_not_retrieved")
except ValueError:
pass
print(json.dumps(events, indent=2))
¶Follow the symptom to the responsible span
Slow: separate queue wait, retrieval, provider time to first token, and stream completion. An illustrative 8-second request might contain 5 seconds of queue wait and 3 seconds of execution; changing the prompt would miss the bottleneck. Parallel spans overlap, so adding their durations can exceed elapsed request time. Use the critical path—the dependent operations determining completion—not their sum. Record TimeoutError at the failing boundary rather than flattening every failure into “generation failed.”
Expensive: attach provider-reported token usage to each attempt, not merely the final answer. A 429 response followed by a successful retry should remain visible as two attempts, although billing depends on what the provider actually processed. Keep usage_missing distinct from zero. A timed-out generation may have consumed billable work even when your client received no usage record. I would reconcile request-level estimates with provider billing before treating them as financial truth.
Unsupported: inspect the evidence path before changing models. Did retrieval find the relevant passage? Did context assembly retain it? Did generation cite it? Did validation establish support, or merely find a well-formed citation? The example’s citation_not_retrieved failure catches only an invalid evidence reference. It cannot prove that a retrieved passage supports a claim. Attach the earlier evaluation rubric’s result and evaluator version; leave the result unknown if evaluation did not run.
For a release rehearsal, inject one retrieval timeout and one invalid citation, then check that both remain distinguishable in the trace. Compare median and 95th-percentile latency only across comparable workloads, reporting request counts and failures alongside them. If you retain detailed traces for only 10% of successful requests, can you still account for every failed attempt and every usage record?
¶When do caching, batching, concurrency limits, and backpressure improve the service?
A faster model call does not rescue a service that accepts work faster than it finishes it. Requests accumulate, students retry, and retries consume the capacity needed to clear the queue. I would first bound admitted work, then optimize repeated work. Otherwise, a successful demonstration with 1 student can become a timeout factory with 30.
¶Cache only what you can safely reuse
For the study assistant, I would start with document embeddings and retrieval results rather than generated answers. Their reuse conditions are easier to specify. An embedding cache key needs the content hash, embedding model revision, and preprocessing version. A retrieval cache also needs the query, index revision, filters, and authorization scope. Unless those inputs match, a hit is not evidence that the cached result is valid.
Answer caching needs a stricter contract: include the conversation context, retrieved evidence revision, prompt version, model configuration, and permission scope. Recheck access before returning cached content. Otherwise, a revoked document can remain readable through an old answer. A 10-minute expiration does not fix that leak; it merely limits its duration. Semantic caching—reusing answers for similar questions—is especially risky when “Can I submit late?” and “Can I submit late without a penalty?” retrieve similar vectors but require different answers.
Provider-side prompt caching is different: it may reuse computation for an eligible prefix, not return a previously generated answer. Billing and eligibility depend on the provider. Track cache hits alongside actual billed tokens, and coalesce simultaneous identical misses so that 20 requests do not trigger 20 computations before the first result is stored. Keep that coordination bounded; a failed producer must not leave followers waiting indefinitely.
¶Batch throughput, not student patience
Batching combines work so that fixed overhead or accelerator capacity is shared. I would batch offline embedding jobs first. Interactive answers are less forgiving: waiting to fill a batch directly increases time to first token. If an illustrative batch takes 400 ms to fill and 600 ms to execute, its earliest request has already waited about 1 second before network overhead.
Use both a maximum batch size and a maximum waiting time. Batch only compatible requests, and preserve the mapping from each input to its result; otherwise, reordered outputs can attach citations to the wrong student. For hosted generation, batching separate requests in your application does not automatically make the provider execute them more efficiently. Self-hosted continuous batching is an inference-server capability, not a benefit supplied by asyncio.gather.
¶Bound the queue as well as the workers
Concurrency limits cap active work; backpressure prevents excess arrivals from accumulating unchecked. Suppose, purely as a sizing example, 4 active calls each occupy a slot for an average of 2 seconds. Their nominal capacity is 2 requests per second. Sustained arrivals at 3 requests per second add roughly 60 queued requests per minute unless requests are rejected or abandoned. Token quotas and variable answer lengths can lower that capacity further.
asyncio.Semaphore bounds active coroutines, but not the number waiting to acquire it. Pair admission control with a bounded queue such as asyncio.Queue(maxsize=8). put_nowait raises asyncio.QueueFull when that queue is full; translate overload into an explicit response, commonly HTTP 503 with Retry-After. Reserve HTTP 429 for rate-limit policy. A process-local limit also multiplies across replicas: 3 replicas permitting 4 calls each can send 12 upstream calls.
Apply one end-to-end deadline across queueing, execution, and retries. Expire queued requests before dispatch; otherwise, disconnected students can still generate charges. Cancellation may not stop already accepted provider work. For your rehearsal, compare fixed workloads at concurrency 1, 2, and 4, recording completed requests, rejections, queue wait, p95 latency, and billed tokens. Then hold arrivals above measured capacity: does the assistant reject promptly, or merely postpone its failure?
¶How do model or embedding migrations fail, and how can shadow tests and rollback contain them?
A migration can return HTTP 200 while making the study assistant worse. A replacement generator may produce valid JSON but attach citations to unsupported claims. A replacement embedding model may accept the same text yet retrieve irrelevant notes. I treat both as behavioral releases, not configuration cleanup: successful API calls prove almost nothing about answer quality.
¶Keep incompatible retrieval systems apart
Changing an embedding dimension from 768 to 1,024 usually causes an obvious index-dimension error. Keeping 768 dimensions can be more dangerous: the index accepts the query vector even though its coordinates belong to a different embedding space. Retrieval silently deteriorates. Equal vector length is not compatibility.
I would build a separate candidate index rather than overwrite the working one. Bind the embedding revision, preprocessing rules, chunk manifest, distance metric, and index identifier into one immutable retrieval configuration. Unless the provider explicitly guarantees compatibility, re-embed the documents when changing embedding models. Preserve source identifiers and authorization metadata; otherwise, a migration can repair retrieval while introducing a tenant-isolation leak.
The generator needs a corresponding release bundle: pin its model revision where available, prompt version, output schema, and tool definitions. A model change can cause schema-validation failures, context-limit errors, or different tool arguments. Reusing the old prompt is a useful experimental control, not proof that it remains suitable.
¶Shadow without letting the candidate act
A shadow test copies an eligible request to the candidate while only the current release answers the student. For an illustrative rehearsal, start with 100 fixed questions, then shadow 5% of eligible traffic. Those are proposed settings, not measured results. Keep the shadow queue bounded and discard excess copies rather than delaying live requests. Count those drops, or your apparent test coverage becomes misleading.
Disable candidate tool execution and external writes. Shadowing a request must not send a second email or modify a study schedule. Apply the same privacy restrictions to copied prompts, and exclude requests whose data cannot enter the candidate provider. Shadow requests still consume tokens and quota; tag their cost separately so an experiment does not masquerade as normal operating expense.
Compare paired requests using the evaluation methods already established: retrieval recall, citation support, abstention, latency, and cost. Record request_id, release_id, embedding_revision, index_id, and prompt_version. Without those fields, an unsupported answer becomes difficult to attribute. Shadow timing includes different queue and cache conditions, so I would not call it a clean latency benchmark unless those conditions are controlled.
¶Rollback the bundle, not one field
This runnable Python example illustrates a compatibility guard, not a deployment controller. Its ValueError message is deliberately application-defined.
from dataclasses import dataclass
@dataclass(frozen=True)
class RetrievalRelease:
embedding_revision: str
index_embedding_revision: str
index_id: str
def validate(self):
if self.embedding_revision != self.index_embedding_revision:
raise ValueError("embedding/index revision mismatch")
stable = RetrievalRelease("encoder-a-r1", "encoder-a-r1", "notes-a")
candidate = RetrievalRelease("encoder-b-r1", "encoder-b-r1", "notes-b")
active = stable
candidate.validate()
active = candidate
active = stable
assert active == stable
In a service, resolve the active bundle once per request, so an in-flight query cannot combine the old encoder with the new index. Promote through an atomic pointer change using your deployment system. Keep the previous bundle, credentials, index, and service capacity available throughout the rollback window. A pointer cannot restore an index you deleted.
Shadow success should precede a small live canary, not replace it: only live exposure exercises the actual response path. Set release gates before promotion—for example, zero authorization violations and an agreed maximum increase in unsupported answers. These are policy thresholds; choose them from the assistant’s risk and evaluation uncertainty, not a universal percentage.
For the rehearsal, inject an embedding/index mismatch, verify that promotion stops, then restore the previous release during traffic. Check that new requests use the restored bundle and that caches cannot return candidate answers under stable keys. If rollback restores configuration but not student-visible behavior, which state did your release bundle leave out?
¶What does a sourced engineering case study actually prove about model choice, errors, latency, and cost?
A smaller model winning a preference test does not prove that your assistant will be cheaper, faster, or correct. That leap turns useful research into bad procurement advice. I read external case studies as evidence for a specific decision under specific conditions—not as permission to skip the release gates established earlier in this chapter.
¶External case study: instruction following with API-sourced prompts
The 2022 InstructGPT paper, Training language models to follow instructions with human feedback, documents an external assistant-development study using labeler-written prompts and prompts submitted through the OpenAI API. This is published research about an API-facing application setting, not my deployment experience and not a complete production operations report.
The researchers report that human evaluators preferred outputs from a 1.3-billion-parameter InstructGPT model to outputs from the 175-billion-parameter GPT-3 model on their prompt distribution. The smaller model had undergone instruction-oriented training with human feedback. The comparison therefore supports a practical conclusion: parameter count alone is a poor model-selection rule. It does not isolate model size as the cause of the preference difference, because the training procedure also changed.
I would use that result to justify evaluating a smaller instruction-tuned candidate for the study assistant. I would not use it to select one without testing course-note questions. Preference on the paper’s prompts does not establish citation support, correct abstention, or resistance to instructions embedded in retrieved PDFs. Those are different obligations.
¶Separate the reported result from the missing measurements
For model choice, record the comparison and its population. For errors, record what was actually evaluated: the paper reports improvements in truthfulness and reductions in toxic output generation, while explicitly acknowledging that the models still make simple mistakes. Neither statement establishes a zero hallucination rate. An answer can sound helpful and still attribute a theorem to the wrong source.
For latency and cost, the cited preference result supplies no transferable service budget. I cannot infer either from parameter count. Hardware, serving configuration, generated length, batching, and retries affect the bill and elapsed time. A smaller model that needs 3 attempts can cost more per accepted answer than a larger model that succeeds once, depending on their prices and token usage.
I keep an evidence ledger with source_url, reported_claim, evaluation_population, and missing_evidence. Local runs add model_revision, prompt_version, p95_latency_ms, and cost_per_accepted_answer. These are proposed record fields, not fields from the paper. That distinction prevents a copied research number from quietly becoming an internal benchmark.
¶Turn the evidence into a bounded experiment
For an illustrative rehearsal—not a reported result—take 200 held-out study questions, including 40 that require abstention. Run both candidates against identical retrieved passages, unless the experiment explicitly tests retrieval changes. Keep output limits and concurrency fixed; otherwise a latency difference may reflect configuration rather than model choice.
Count unsupported answers separately from malformed output rejected by your validator. Also record request timeouts and rate-limit errors: excluding failed requests makes an unreliable service appear fast. If retries are enabled, include every attempt in cost accounting and measure elapsed time through final completion or failure. Report accepted-answer cost only alongside acceptance rate; a candidate that refuses every difficult question can otherwise look economical.
Ask yourself: “Does the published preference win justify replacing the current model?” My answer is no. It justifies a candidate experiment, with replacement conditional on local quality, failure, latency, and cost gates. As a short exercise, write down which missing measurement would most likely reverse your choice.
The next chapter needs those measurements, their sources, and their limitations—not an unsupported claim that you chose the best model. Keep the evidence intact so you can explain what you shipped and why.