¶How do multimodal encoders and projectors connect images or sound to language?
Your study assistant can cite a paragraph correctly and still misread the arrow in a scanned diagram or miss “not” in a recorded lecture. Adding images, audio, and video means preserving both the evidence and its location while admitting that each input can fail differently.
I would start with one licensed diagram and one short lecture excerpt, not an entire semester’s recordings. A successful text-only retrieval test does not establish that the assistant can read small labels or distinguish overlapping speakers.
¶The bridge is learned, not just resized
A multimodal encoder turns an image or sound segment into a sequence of numerical features. An image encoder may process patches; an audio encoder may process waveform samples or time-frequency features. These representations carry information useful for the encoder’s training task, but they are not automatically compatible with a language model’s token embeddings.
A projector is a learned bridge between those representations. In a simple design, it changes each feature vector’s width to the language model’s embedding width. Other designs use multiple layers or compress the sequence through learned queries. Matching dimensions is necessary for this connection, but insufficient for understanding: training must teach the language model how those features relate to words.
Consider an illustrative image path: a 224 × 224 image split into 16 × 16 patches gives 196 patches, before any additional tokens. If the encoder produces 768 features per patch and the language model expects 2,048, a linear projector can map between those widths. In PyTorch, torch.nn.Linear transforms the final dimension; torch.zeros below supplies dummy features, not encoded image content.
import torch
features = torch.zeros(1, 196, 768)
projector = torch.nn.Linear(768, 2048)
projected = projector(features)
assert projected.shape == (1, 196, 2048)
print(tuple(projected.shape))
The batch still contains 196 vectors after projector(features). This untrained layer demonstrates shape compatibility only. Random weights do not create visual understanding. Incorrect input widths can instead produce a RuntimeError containing mat1 and mat2 shapes cannot be multiplied; check the encoder output and projector input dimensions before changing generation settings.
¶Where the language model receives the evidence
Some architectures place projected features into the language model’s input sequence alongside text embeddings. Others expose encoder outputs through the cross-attention you already know. Neither arrangement implies that every checkpoint accepts every modality. A vision-language model can reject audio as unsupported input, and converting an audio file into bytes does not change that contract.
I avoid treating “multimodal” as a capability checklist. Read the checkpoint’s supported inputs and preprocessing requirements before calling generate. Depending on the architecture, increasing image resolution can increase the feature count and memory demand; it helps small text only when preprocessing preserves the relevant detail. Audio compression can likewise discard timing detail, while sparse video frames can miss an event between samples.
Keep provenance outside the learned bridge. Associate image evidence with source_id, page_number, and, where available, a crop region; associate audio with start_ms and end_ms. Features do not guarantee recoverable coordinates. If the assistant transcribes “do submit” instead of “do not submit,” a fluent answer with a timestamp is still wrong.
For a first experiment, ask the same diagram question using the full page and a readable crop. Record label errors separately from explanation errors. If only the crop succeeds, what evidence disappeared before the language model ever had a chance to reason?
¶When should scanned notes use OCR, and when does a diagram need visual interpretation?
A study assistant can cite the correct page and still give the wrong answer. If extraction turns “≤” into “<”, the citation points to genuine evidence while the answer changes its meaning. If a diagram becomes a list of labels, every word may survive while every relationship disappears.
I would start with OCR—optical character recognition—for scanned paragraphs, not send every page through a vision-language model. OCR produces searchable text that fits the existing retrieval pipeline. But it is the wrong endpoint when the answer depends on arrows, spatial grouping, chart scales, or mathematical layout. A page can need both routes.
¶Route regions, not entire documents
First inspect the PDF’s existing text layer using the extraction path already built. Unless that text is missing or demonstrably corrupt, rasterizing and recognizing it again introduces another opportunity for error. For an image-only page, render a readable image; 300 dots per inch is a reasonable initial experiment, not a guaranteed requirement. Tiny annotations may need a higher-resolution crop, provided the source actually contains that detail. Enlarging blurred pixels does not recover missing strokes.
For the study assistant, distinguish three cases:
- Paragraphs: use OCR, retaining reading order and word or line bounding boxes where available. Two columns concatenated across the page produce fluent-looking nonsense.
- Tables and equations: preserve structure, not just characters. A lost superscript turns “x²” into “x2”; a shifted table cell assigns the right number to the wrong experiment.
- Diagrams: use visual interpretation when the question requires relationships. Ask for visible labels, connections, and uncertainties separately rather than requesting an unrestricted explanation.
Consider an illustrative scanned page containing 2 paragraphs and a feedback-loop diagram. OCR can recover “controller,” “plant,” and “sensor.” It cannot establish the feedback direction merely by recovering those words. Give the visual model the diagram crop and, when needed, the surrounding caption. If the arrowhead is unreadable, the answer should say so rather than infer the conventional arrangement.
¶Keep extraction separate from interpretation
I would store recognized text and generated descriptions as different evidence types. Otherwise, retrieval can silently promote a model’s interpretation into a quotation from the notes. The following runnable fixture records an illustrative interpretation, not an actual OCR or vision-model result:
import json
evidence = {
"source_id": "licensed-control-notes",
"page_number": 3,
"region_id": "diagram-1",
"bbox_normalized": [0.10, 0.40, 0.90, 0.85],
"evidence_kind": "visual_interpretation",
"observed_labels": ["controller", "plant", "sensor"],
"description": "Three labeled blocks connected by lines.",
"uncertainties": ["Feedback arrowhead is unreadable."],
}
x0, y0, x1, y1 = evidence["bbox_normalized"]
assert 0 <= x0 < x1 <= 1
assert 0 <= y0 < y1 <= 1
print(json.dumps(evidence, indent=2))
Here, page_number is one-based; bbox_normalized uses the rendered page’s top-left origin. Preserve the rendering orientation so the citation highlights the right region. Keep the original crop alongside description, and retain the extractor or model revision in ingestion metadata. A reviewer needs pixels, not just another generated sentence.
Test a small licensed fixture set: for example, 6 paragraph crops and 6 diagrams. Measure character errors separately from missing connections and reversed arrows. OCR confidence, when exposed, is not a calibrated probability that an answer is correct; its meaning depends on the engine. I would not choose a universal cutoff without labeled examples.
For your next experiment, ask the same diagram question using OCR text alone and then the crop. Which answer claims a relationship that the visible evidence cannot establish?
¶Can speech recognition turn a lecture excerpt into timestamped searchable evidence?
A lecture transcript can retrieve the right sentence and still support the wrong answer. If speech recognition drops “not” from “gradient descent does not guarantee a global minimum,” the study assistant now has plausible, searchable misinformation. I would start with a small, licensed lecture excerpt and human-checked timestamps, not a semester’s recordings. Automatic speech recognition (ASR) converts speech into text; it does not certify what the lecturer said.
¶Keep the recording attached to every passage
Use the existing retrieval pipeline, but preserve the audio source alongside each transcript segment. Store the recording identifier, transcript text, start and end times, model revision, and review status. Hugging Face’s pipeline supports automatic speech recognition, but timestamp availability and granularity depend on the checkpoint and implementation. Check that contract before selecting a model. Segment timestamps locate a passage; they are not necessarily precise word boundaries.
For a 30-second excerpt cut from 90 seconds into a lecture, a segment at 2–5 seconds belongs at 92–95 seconds in the original recording. Losing that offset produces broken citations even when every word is correct. I prefer recording-relative seconds in the index, with the excerpt offset retained for auditing.
This runnable example uses illustrative, normalized recognizer output—not a transcription result or benchmark. Adapt your recognizer’s response into this shape after checking its timestamp convention.
import math
recording_id = "licensed-lecture-01"
clip_offset_s = 90.0
clip_duration_s = 30.0
segments = [
{"text": "Gradient descent does not guarantee a global minimum.",
"start_s": 2.0, "end_s": 5.0}
]
evidence = []
for segment in segments:
start = float(segment["start_s"])
end = float(segment["end_s"])
if not (math.isfinite(start) and math.isfinite(end)
and 0 <= start < end <= clip_duration_s):
raise ValueError("Invalid segment timestamps")
evidence.append({
"source_id": recording_id,
"text": segment["text"],
"start_s": clip_offset_s + start,
"end_s": clip_offset_s + end,
"reviewed": False,
})
assert evidence[0]["start_s"] == 92.0
assert evidence[0]["end_s"] == 95.0
print(evidence)
The validation raises ValueError with Invalid segment timestamps rather than indexing an impossible interval. When retrieval merges adjacent segments, carry their individual intervals forward or retain the encompassing interval explicitly. Do not assign invented word timestamps by dividing the duration equally: pauses and speaking rates make that precision fictitious.
¶Measure the mistakes that change the lesson
Word error rate counts substitutions, deletions, and insertions divided by the reference transcript’s word count. With 20 reference words, 2 substitutions, 1 deletion, and 1 insertion produce 20% word error rate. Fix your punctuation and casing normalization before comparing systems, otherwise formatting differences can obscure recognition errors.
I would not choose a recognizer on that score alone. A deleted negation can matter more than several incorrect articles. Track technical-term accuracy, negation preservation, and whether a citation actually contains the claimed speech. On a human-checked fixture, count how many cited phrases fall inside their proposed intervals; permit a declared tolerance, such as 1 second, only when that matches your playback interface.
Silence can produce hallucinated text; overlapping speakers can merge contradictory statements. Unless you have evaluated speaker attribution separately, do not label a segment as the lecturer’s rather than a student’s. Also, “as this equation shows” remains incomplete evidence when the equation exists only on a slide. Retrieve the slide too, or abstain from explaining its contents.
Exercise: Record 3 permitted excerpts: clean speech, background noise, and a technical explanation containing a negation. Hand-transcribe them, inspect timestamp alignment, and test one retrieval question per excerpt. Keep the original transcript and any reviewed correction separately; silently replacing errors destroys the evidence needed to compare recognizers later.
If asked why timestamps matter, explain that they make a claim inspectable, not automatically trustworthy. Once the lecturer points at a slide, what additional evidence must the assistant preserve?
¶Why does video require both frame selection and audio alignment?
A lecturer says “this edge causes the cycle,” but your assistant retrieves a diagram shown 8 seconds earlier. The transcript is accurate. The image is sharp. The answer is still wrong because the evidence belongs to different moments. Video ingestion must preserve not just what appeared and what was said, but their relationship in time.
I would start with selected frames plus the timestamped transcript from the preceding section, rather than send every frame to a model. A 120-second clip at 30 frames per second contains 3,600 frames; sampling every 2 seconds produces 60 images. That is a manageable baseline, not a guarantee of coverage. A brief pointer movement can fall entirely between samples.
¶Select for evidence, not just visual variety
For lecture slides, keep periodic samples and add frames around slide changes. Deduplicating near-identical frames saves image processing, but aggressive deduplication can erase the single annotation that matters. For a worked derivation, retain intermediate states; the final equation does not show which substitution produced it. I prefer redundant evidence over a compact index that silently loses the explanation.
Scene-change detection alone is a poor default for lectures: a static slide can contain several minutes of meaningful speech. Conversely, camera movement can trigger changes without adding evidence. Unless the question concerns motion, a study assistant usually needs readable states rather than smooth playback. Motion questions require denser, ordered frames or a model explicitly supporting temporal video input; a collection of images does not automatically supply that capability.
¶Put both streams on one clock
Store each frame’s presentation timestamp—the time it should appear during playback—not merely its frame number. Dividing frame number by an assumed frame rate can misalign variable-frame-rate recordings. Keep source_id, frame_time_s, audio_start_s, and audio_end_s in the same source-relative time coordinate. If a clip begins 300 seconds into the recording, add that offset to clip-relative transcript times exactly once. Otherwise citations jump to the wrong passage.
This runnable example aligns illustrative transcript metadata with selected source-relative frames. It does not decode video or establish that a frame supports the spoken claim.
source_id = "licensed-lecture-01"
clip_offset_s = 300.0
audio_start_s = clip_offset_s + 12.0
audio_end_s = clip_offset_s + 16.0
frame_times_s = [310.0, 312.0, 314.0, 316.0, 318.0]
if audio_end_s <= audio_start_s:
raise ValueError("audio interval must have positive duration")
aligned = [
{"source_id": source_id, "frame_time_s": time_s}
for time_s in frame_times_s
if audio_start_s <= time_s < audio_end_s
]
print(aligned)
The half-open interval includes 312 and 314 seconds but excludes 316. If aligned is empty, report missing visual evidence rather than silently borrowing a distant frame. Even a nonempty match proves only temporal overlap. Speech may anticipate the next slide or refer backward. Preserve nearby candidates and their timestamps so an answer can distinguish “the lecturer says” from “the slide shows.”
¶Test the relationship, not only the transcript
On a small licensed fixture, ask “Which arrow did the lecturer add?” Check frame coverage, transcript accuracy, and alignment separately. Shift audio timestamps by 5 seconds deliberately: does the assistant now cite the wrong diagram, or flag insufficient evidence? There is no universal safe sampling interval; it depends on how briefly the relevant event remains visible.
An interviewer asks, “Why not retrieve audio and images independently?” My answer: independent retrieval can find relevant content while destroying temporal correspondence. Which test in your fixture would catch that failure?
¶How do we test transcription errors, visual mistakes, and the cost of extra inputs?
A study assistant can cite the correct lecture and still teach the opposite lesson. Drop “not” from “the cache is not shared,” and the source link looks convincing while the answer becomes wrong. I would test evidence extraction separately from answer generation; otherwise, a fluent answer can hide a damaged transcript or a reversed diagram arrow.
Start with 12 small, licensed fixtures: 4 scanned pages, 4 lecture excerpts, and 4 short videos. This is a debugging set, not evidence of broad accuracy. Include a faint minus sign, a two-column page, background noise, and a slide change during speech. Store the expected evidence in expected_text, page_number, start_seconds, and end_seconds, alongside the source identifier. Human-check these references before evaluating the assistant; an incorrect reference punishes the correct output.
¶Score the mistake, not just the sentence
For transcription, word error rate counts substitutions, deletions, and insertions, divided by the reference word count. The following dependency-free example computes the minimum number of word edits. Its lowercase, whitespace-only normalization is deliberately explicit: punctuation and number normalization can change scores, so freeze those rules before comparing models.
def word_error_rate(reference, hypothesis):
expected = reference.lower().split()
actual = hypothesis.lower().split()
if not expected:
raise ValueError("Reference must contain at least one word")
previous = list(range(len(actual) + 1))
for i, expected_word in enumerate(expected, start=1):
current = [i]
for j, actual_word in enumerate(actual, start=1):
current.append(min(
previous[j] + 1,
current[j - 1] + 1,
previous[j - 1] + (expected_word != actual_word),
))
previous = current
return previous[-1] / len(expected)
score = word_error_rate(
"the cache is not shared",
"the cache is shared",
)
assert score == 0.2
print(f"Word error rate: {score:.0%}")
That deletion produces 20% word error rate, but its educational consequence deserves a separate failure label. Record critical_term_error when negation, a formula, a unit, or an assessed technical term changes meaning. Word error rate can exceed 100% when insertions accumulate; do not clamp it. A low average is not permission to ignore a corrupted equation.
For diagrams, score explicit facts: arrow direction, axis labels, legend mapping, and relationships. “Looks reasonable” is not a rubric. If an arrow runs from the database to the cache but the answer reverses it, mark that relationship wrong even when both labels are recognized. When a label is genuinely unreadable, test whether the assistant abstains rather than inventing it. I prefer these small factual checks to asking another model whether the description sounds accurate.
For video, require both the correct claim and a usable timestamp. As an exercise, choose a 2-second tolerance around a manually marked event, then inspect failures rather than treating that tolerance as universal. A reference that lands after the slide disappears fails source navigation. If widening the tolerance rescues every result, check audio alignment before changing the scoring rule.
¶Make extra inputs earn their cost
Run paired trials on identical questions: extracted text alone, text plus selected images, then text plus images and audio. Record elapsed_seconds, input_tokens where reported, and estimated_cost, including extraction calls and retries. Image billing may depend on resolution or tiles; audio billing may depend on duration. I would use the provider’s current billing units, not pretend every modality has one token conversion.
Keep output limits and model revisions fixed unless the experiment specifically changes them. Compare supported answers and critical failures against cost per question; sending more evidence is worthwhile only when it repairs errors worth paying for. These fixtures give the next chapter concrete source material and failure cases for examining trust boundaries—not merely plausible-looking answers.