¶Why should extraction preserve page numbers, headings, and document identity?
Your study assistant can retrieve the right sentence and still be useless if you cannot find that sentence in the original notes. This chapter builds a searchable collection from permitted course notes and text-based PDFs without losing the document identity, page boundaries, and headings needed to inspect a result.
I would start with inspectable page records, not a vector database. A database can store damaged text perfectly; it cannot reconstruct a missing heading or discover which edition supplied a paragraph. Extraction is the step that reads source content into machine-readable text. Its output needs provenance: enough information to trace that text back to a particular source and location.
¶A page number needs a coordinate system
Suppose an illustrative 24-page PDF has 3 introductory pages before the printed page numbered 1. Its fourth PDF page is therefore printed page 1. Store the physical, 1-based PDF position in page_number, and keep a separate page_label if you can reliably obtain the printed label. Do not silently treat them as interchangeable. Otherwise, “see page 4” sends a student to the wrong explanation even though the extracted sentence is correct.
Keep one extraction record per page initially, including empty pages. If page 7 produces no text, record that outcome rather than removing the page and renumbering everything afterward. An empty result might mean a genuinely blank page, an image-only page, or an extraction failure. It is not proof of any one cause. Image-only documents need a different path, covered with optical character recognition in the multimodal chapter.
¶Names identify documents poorly
Two lecturers can both upload notes.pdf. One lecturer can replace that file while keeping its name. I prefer a stable logical document_id for the course resource and a separate source_sha256 computed from its original bytes. The first identifies the resource; the second distinguishes exact file versions. A changed hash means changed bytes, not necessarily changed meaning: editing PDF metadata alone can change it.
Keep source_name for display, but do not use it as the unique key unless your ingestion contract guarantees uniqueness. Also record the extractor name, version, and settings in an ingestion manifest. When identical source bytes produce different text after a dependency upgrade, those details let you investigate the difference instead of blaming retrieval.
This standard-library example models the contract using an explicit text fixture. The form-feed character separates its 2 pages; it is not a PDF extraction technique.
from hashlib import sha256
source_bytes = (
"Gradient Descent\nThe learning rate controls the step size."
"\f"
"Limitations\nA large learning rate can cause divergence."
).encode("utf-8")
source_hash = sha256(source_bytes).hexdigest()
records = []
for page_number, text in enumerate(
source_bytes.decode("utf-8").split("\f"), start=1
):
records.append({
"document_id": "optimization-notes",
"source_name": "optimization-notes.txt",
"source_sha256": source_hash,
"page_number": page_number,
"heading": text.splitlines()[0],
"text": text,
})
assert len(records) == 2
assert records[1]["page_number"] == 2
print(records[1])
¶Preserve structure without inventing it
The fixture places a heading on the first line deliberately. Real PDFs do not promise that arrangement. Unless the extractor exposes trustworthy structure, identifying headings from font size or line position is a heuristic—a rule that can be wrong. Preserve an unknown heading as None rather than confidently promoting a running header into a section title. A heading such as “Limitations” changes how the following paragraph should be read.
Reading order deserves the same suspicion. In a 2-column handout, extraction can interleave the left column’s explanation with the right column’s example. The output may contain ordinary words without raising an exception. That is a silent failure, and a successful parser return does not make it correct. Inspect a representative sample against the rendered pages, especially tables, equations, and repeated headers. I would retain the raw extracted text separately from cleaned text so that cleanup mistakes remain reversible.
Exercise: take 3 permitted documents and check their first page, one middle page, and final page. Can you locate every extracted passage in the original, distinguish two versions, and explain each empty result? Once that contract holds, the next question is how to divide those pages into searchable passages without cutting away the context that gives a sentence its meaning.
¶How do we detect broken text extraction instead of indexing unreadable PDFs?
Your study assistant can return the correct page and still retrieve nonsense. A PDF viewer draws readable sentences, but the extractor returns alternating lines from two columns, drops a minus sign, or produces an empty string. Indexing succeeds because a string is valid input. That is the trap: successful processing does not establish that the extracted text preserves the lesson.
I would put a quality gate immediately after extraction, before any chunking or embedding. Its job is not to certify meaning automatically. It separates plausible text from pages requiring inspection, while keeping the original extraction available for comparison. A suspicious page belongs in a review queue, not silently in the searchable collection.
¶Check symptoms, not just character counts
Start with three cheap signals: too little text, excessive replacement characters, and an unusually small proportion of letters or digits. The Unicode replacement character, U+FFFD, often signals decoding damage somewhere in the processing chain. An empty extraction can indicate an image-only page, but it can also be a genuinely blank page. Neither signal identifies the cause by itself.
For ordinary prose notes, the example below flags fewer than 40 non-whitespace characters, a replacement-character fraction above 1%, or an alphanumeric fraction below 50%. These are illustrative review thresholds, not measured accuracy claims. They will overflag equations, title pages, and some programming notes unless you adjust them for your documents. I prefer these explicit checks to a single “quality score”: a reason you can investigate is more useful than an unexplained 0.73.
def inspect_page(document_id, page_number, text):
compact = "".join(text.split())
count = len(compact)
replacement_fraction = compact.count("\ufffd") / max(count, 1)
alphanumeric_fraction = sum(
character.isalnum() for character in compact
) / max(count, 1)
reasons = []
if count < 40:
reasons.append("too_little_text")
if replacement_fraction > 0.01:
reasons.append("replacement_characters")
if count and alphanumeric_fraction < 0.50:
reasons.append("low_alphanumeric_fraction")
return {
"document_id": document_id,
"page_number": page_number,
"text": text,
"status": "review" if reasons else "candidate",
"reasons": reasons,
"non_whitespace_characters": count,
"replacement_fraction": replacement_fraction,
"alphanumeric_fraction": alphanumeric_fraction,
}
pages = [
"",
"Gradient \ufffd\ufffd\ufffd descent \ufffd\ufffd\ufffd",
"Gradient descent updates model parameters using the gradient "
"of the loss. The learning rate controls the update size.",
]
reports = [
inspect_page("optimization-notes", number, text)
for number, text in enumerate(pages, start=1)
]
assert [report["status"] for report in reports] == [
"review", "review", "candidate"
]
for report in reports:
print(report["page_number"], report["status"], report["reasons"])
This runs without a PDF library because it tests the gate independently of the extractor. Feed each extracted page into inspect_page at your ingestion boundary. The first fixture reports too_little_text; the second also reports replacement_characters; the third becomes candidate, deliberately not “verified.” Preserve text unchanged. Removing damaged characters before inspection would hide the evidence of failure.
¶The dangerous failures look like normal sentences
Character checks cannot detect reading order. A two-column page might extract “Gradient descent Validation loss updates parameters measures generalization.” Every character is legitimate; the explanation is broken. Compare rendered pages against extracted text for at least one ordinary page, one dense page, and one structurally unusual page per document. This is a starting sample, not a guarantee. Inspect every flagged page, and expand sampling when documents mix layouts.
Tables and equations deserve particular suspicion. Flattened table cells can attach a learning rate of 0.01 to the wrong experiment. A missing minus sign changes an equation without making its text look corrupt. When these elements carry examinable facts, check their relationships manually; passing isalnum checks says nothing about mathematical correctness. A repeated footer is another failure: extraction preserves it perfectly, yet later search can mistake repetition for useful content.
Keep extraction exceptions separate from quality findings. If the PDF parser throws an exception, record its actual type and message with the page or document identity. Do not catch it and substitute "": that converts “parser failed” into “possibly blank page.” Likewise, a timeout is an extraction failure, not evidence that the document contains no text. For image-only material, route the document to a separate conversion workflow; optical character recognition is covered in the multimodal chapter.
An interviewer might ask, “Why not let an LLM repair every extracted page?” My answer: repair can invent a missing symbol or reorder statements incorrectly, making corruption harder to notice. I would first change extraction settings or tools and compare against the rendered source. As an exercise, add a two-column-looking sentence to the fixtures: the gate should miss it. What inspection record would you require before allowing that page into the collection?
¶Which chunk boundaries preserve an explanation without duplicating the whole document?
A search result that contains “this prevents the problem” but omits the preceding explanation is readable, correctly extracted, and almost useless. Chunking can damage evidence without corrupting a single character. For the study assistant, the useful unit is usually an explanation: a claim together with its conditions, example, or exception. A chunk is the bounded passage you store and search independently.
I start with headings and paragraphs, not a fixed character count. Keep a definition beside its example when both fit; keep a warning beside the procedure it qualifies. Unless the notes are unusually repetitive, splitting every 500 characters cuts through too many of these relationships. Conversely, treating a 12-page chapter as one passage makes a specific explanation compete with unrelated material inside the same searchable unit.
¶Use structure first, then enforce a size limit
For an initial experiment, try a target of 300 tokens and a ceiling of 450, provided the embedding model accepts that length with room for any added heading. These are starting settings, not measured optima. Count with that model’s tokenizer before embedding: 300 words are not 300 tokens. If the input exceeds the model’s limit, a library may silently truncate it, leaving stored text that contains evidence its vector never represented.
Pack adjacent paragraphs within a heading until the next paragraph would exceed the ceiling. If one paragraph is oversized, split at sentence boundaries; if one sentence is oversized, use tokenizer offsets as a last resort. Do not cross an unrelated heading merely to fill space. A table also needs special treatment: separating a row from its column labels produces ambiguous evidence, even when both pieces satisfy the size limit.
The following standard-library example demonstrates paragraph packing on one already-validated page. It uses a word budget so it runs without a model download; it is not a token-limit check. pack_page deliberately raises ValueError for an oversized paragraph rather than pretending to handle sentence splitting.
import re
def pack_page(text, document_id, page, max_words=40):
if max_words < 1:
raise ValueError("max_words must be positive")
paragraphs = list(re.finditer(
r"\S(?:.*?\S)?(?=\n\s*\n|\Z)", text, flags=re.S
))
chunks = []
start = end = None
words = 0
def save(left, right):
chunks.append({
"document_id": document_id,
"page": page,
"start_char": left,
"end_char": right,
"text": text[left:right],
})
for paragraph in paragraphs:
count = len(paragraph.group().split())
if count > max_words:
raise ValueError("paragraph exceeds max_words")
if start is not None and words + count > max_words:
save(start, end)
start = end = None
words = 0
if start is None:
start = paragraph.start()
end = paragraph.end()
words += count
if start is not None:
save(start, end)
return chunks
text = (
"A deadlock occurs when transactions wait for resources "
"held by each other.\n\n"
"For example, transaction A holds row X and waits for row Y; "
"transaction B does the reverse.\n\n"
"A timeout ends a wait, but does not prove that the database "
"detected a deadlock."
)
chunks = pack_page(text, "database-notes-v1", page=7)
assert len(chunks) == 2
for chunk in chunks:
assert text[chunk["start_char"]:chunk["end_char"]] == chunk["text"]
print(chunk)
The definition and example occupy the first chunk; the timeout distinction occupies the second. That boundary deserves inspection: a question comparing deadlocks and timeouts may need both. Passing the size check does not prove the split is useful. Preserve document_id, page, start_char, and end_char so you can inspect the exact source span. Store a document revision and chunker configuration alongside the output; offsets become invalid if you later normalize the source text differently.
¶Overlap is a repair, not a default entitlement
I avoid automatic 20% overlap as the first fix. With a 400-token window and an 80-token overlap, the stride is 320 tokens: a long document produces roughly 25% more indexed text than disjoint windows, ignoring its edges. Adjacent results can then repeat the same explanation instead of showing different evidence.
Use overlap only when inspected boundaries repeatedly separate necessary context. Prefer repeating one complete sentence over copying an arbitrary token tail, and never carry text across an unrelated heading. If an explanation crosses a page boundary, allow a multi-page chunk only while retaining separate source spans for each page; assigning everything to the starting page creates false provenance.
Exercise: choose 10 questions from your permitted notes. Mark the smallest passage needed to answer each, then inspect whether your chunks preserve those passages. Record dangling references, detached exceptions, and duplicated examples—not just chunk counts.
Interview question: Why not increase chunk size until every explanation fits? My answer: larger passages preserve more context but mix more topics, and they can exceed the embedding model’s input limit. The decision depends on document structure and the questions being searched. Once the boundaries survive inspection, the next question is whether the searchable representation can distinguish those explanations.
¶How do sentence embeddings, cosine similarity, and lexical search rank passages?
A student asks, “Why does my model perform well on training data but poorly on unseen examples?” Your notes call this overfitting. A search that requires matching words can miss the explanation. Conversely, a semantic search for a course code such as CS401 may return a thematically related passage from the wrong course. I start the study assistant with both lexical and vector rankings available separately, because their disagreements expose different failures.
¶Two ways to score the same passage
Lexical search rewards matching terms. A useful baseline is TF–IDF: term frequency measures a word’s presence within a passage, while inverse document frequency reduces the influence of words appearing throughout the collection. The TfidfVectorizer implementation below also normalizes each passage vector. BM25 is another lexical scoring method; it adds term-frequency saturation and explicit document-length normalization. I prefer TF–IDF for this first experiment because you can inspect its vocabulary directly with get_feature_names_out().
A sentence embedding instead represents a passage as a fixed-length vector trained to support comparisons between texts. Unlike the token embeddings covered earlier, you get one vector for the whole input. “Sentence” does not require exactly one sentence: a paragraph can work, provided it fits the encoder’s input limit. Do not average arbitrary token embeddings from a language model and assume you have a useful retrieval model. Pooling produces a vector; suitable training makes that vector useful.
Cosine similarity divides the vectors’ dot product by their lengths. It ranges from −1 to 1 for nonzero real vectors; higher means more aligned, not more truthful. With unit-length vectors, cosine is just the dot product. A score of 0.8 does not mean an 80% probability of relevance. Unless you calibrate a threshold against your own questions, I would not use a universal cutoff to declare a passage relevant.
¶Compare rankings without a database
Install the dependencies with python -m pip install sentence-transformers scikit-learn numpy in your project environment, then run this script. The first run downloads sentence-transformers/all-MiniLM-L6-v2, so it needs network access. Record the resolved checkpoint revision and dependency versions for reproducibility; a model name alone is not a complete experiment record.
import numpy as np
from sentence_transformers import SentenceTransformer
from sklearn.feature_extraction.text import TfidfVectorizer
passages = [
{
"id": "ml-p12",
"page": 12,
"text": "Overfitting occurs when a model memorizes training examples "
"and fails to generalize to unseen data.",
},
{
"id": "ml-p18",
"page": 18,
"text": "Gradient descent updates model parameters to reduce "
"the training loss.",
},
{
"id": "admin-p2",
"page": 2,
"text": "CS401 requires submission of the final project on Friday.",
},
]
texts = [passage["text"] for passage in passages]
query = "Why does memorization hurt performance on new examples?"
lexical = TfidfVectorizer()
term_vectors = lexical.fit_transform(texts)
query_terms = lexical.transform([query])
lexical_scores = (term_vectors @ query_terms.T).toarray().ravel()
encoder = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
passage_vectors = encoder.encode(texts, normalize_embeddings=True)
query_vector = encoder.encode([query], normalize_embeddings=True)[0]
vector_scores = passage_vectors @ query_vector
def show_ranking(label, scores):
print(label)
order = np.argsort(-scores, kind="stable")
for index in order:
passage = passages[int(index)]
print(
f'{passage["id"]} page={passage["page"]} '
f'score={float(scores[index]):.4f}'
)
show_ranking("TF-IDF", lexical_scores)
show_ranking("Cosine", vector_scores)
The boring detail matters: both score arrays use the original passage order. Sorting scores without carrying the corresponding passage identity creates a provenance failure—plausible text attached to the wrong page. Here, np.argsort returns positions, and those positions select the original records. Stable sorting preserves input order for ties; it does not make tied passages relevant.
This encoder produces 384 dimensions, so the 3 passage vectors contain 1,152 coordinates. At 4 bytes per coordinate, their raw float32 payload occupies 4,608 bytes, excluding metadata and Python overhead. The matrix multiplication scores every passage: this is exact vector search, not approximate search. I keep this baseline before introducing an index because it separates ranking errors from index errors.
¶Inspect failures before trusting scores
Try CS401 as the query, then try “unseen examples.” If every query term is absent from the fitted vocabulary, TF–IDF produces a zero vector and all scores become 0. That is an empty lexical match, not evidence that every passage is equally useful. An empty or entirely filtered corpus can instead raise ValueError with empty vocabulary.
Dense search still produces scores for unfamiliar identifiers, but that does not guarantee it preserves their exact meaning. Also inspect encoder.max_seq_length: inputs beyond the configured limit can be truncated, silently excluding the sentence you needed. For models requiring query and passage prefixes, follow that model’s retrieval contract; the symmetric encoding above is not universal.
Interview question: Why not rank everything with embeddings? My answer: semantic similarity helps with paraphrases, while exact terminology can decide relevance. Neither score proves correctness. Next, increase the collection size: when does scoring every vector become expensive enough to justify an approximate index?
¶When does a vector database earn its operational cost over exact search?
A study assistant can acquire a network dependency, another bill, and a backup problem before it acquires enough notes to justify any of them. I would start this project with exact search over a local embedding matrix. Move beyond it when measurements expose a limit, or when shared storage and concurrent updates justify a database independently of search speed.
Separate two decisions: how to find nearby vectors and how to operate their storage. An approximate index can run inside your Python process; a vector database can offer exact search. Buying a database does not automatically make retrieval approximate, and choosing an approximate algorithm does not require running a server.
¶Make exact search earn its replacement
For normalized embeddings, the matrix multiplication from the previous section scores every chunk. With 10,000 chunks, 384 dimensions, and float32 storage, the vector matrix occupies 15,360,000 bytes, about 14.6 MiB. At 1 million chunks, that becomes about 1.43 GiB, excluding text, metadata, temporary arrays, and index overhead. Those figures describe storage, not latency: processor, memory bandwidth, and numerical-library configuration determine scan time.
This runnable experiment uses synthetic vectors to isolate search work from embedding generation. It measures neither semantic quality nor database performance. Save it as exact_search.py in your existing NumPy environment.
import time
import numpy as np
rng = np.random.default_rng(42)
vectors = rng.standard_normal((10_000, 384)).astype(np.float32)
vectors /= np.linalg.norm(vectors, axis=1, keepdims=True)
query = rng.standard_normal(384).astype(np.float32)
query /= np.linalg.norm(query)
def search(matrix, query_vector, k=5):
scores = matrix @ query_vector
candidates = np.argpartition(-scores, k - 1)[:k]
order = np.argsort(-scores[candidates])
indices = candidates[order]
return indices, scores[indices]
search(vectors, query)
durations = []
for _ in range(30):
start = time.perf_counter()
indices, scores = search(vectors, query)
durations.append(time.perf_counter() - start)
print(f"Vector storage: {vectors.nbytes / 2**20:.2f} MiB")
print(f"Median search: {np.median(durations) * 1000:.2f} ms")
print("Chunk rows:", indices.tolist())
print("Scores:", scores.tolist())
np.argpartition selects candidates without sorting every score; np.argsort orders only those candidates. The function assumes at least 5 rows and nonzero, normalized vectors. For production ingestion, reject nonfinite embeddings: dividing a zero vector by its norm produces NaN, after which rankings become untrustworthy without necessarily raising an exception. Preserve the returned row-to-chunk mapping; sorting metadata separately creates a worse failure—plausible passages attached to the wrong pages.
¶Approximation buys less searching, not better evidence
An approximate nearest-neighbor index avoids comparing the query with every vector. HNSW, a common graph-based approach, follows links between nearby vectors and explores a bounded candidate set. Exploring more candidates generally improves agreement with exhaustive search in exchange for additional work. It can still miss a passage that exact search would return.
I keep exact search as the comparison baseline even after adding an approximate index. On the same frozen collection, run the same queries against both and record identifiers, scores, and elapsed time. Compare their first 5 results, inspecting disagreements rather than declaring either list semantically correct. Exact search guarantees the ranking under your chosen vector metric, not that the embedding model understood the question.
Filtering needs its own test. If you request passages from one course but filter only after taking the nearest 5 across all courses, the result may be empty even though matching passages exist. Check whether the implementation searches within eligible records or discards ineligible candidates afterward. A fast empty result is still a retrieval failure.
¶Pay for operations only when you need them
A database earns consideration when multiple application processes need a consistent collection, updates cannot wait for a local rebuild, or measured search time exceeds your budget at expected load. It also introduces connection failures, timeouts, persistence configuration, and restore testing. I would not trade a working local index for those obligations merely because a tutorial uses a hosted service.
Before accepting a backend, verify that returned records retain chunk_id, document_id, page_number, and embedding_revision. A dimension mismatch may produce an explicit error; mixing two embedding revisions with identical dimensions can silently produce meaningless scores. Require one embedding revision per searchable collection unless compatibility has been demonstrated.
As an exercise, increase the experiment’s row count and write down the smallest collection that violates your own search-time or memory budget. That measured boundary—not a universal chunk-count rule—is your interview answer to “Why did you choose this index?”
The next chapter needs a retriever that returns passages with stable source references, not just similarity scores. Keep that contract unchanged while it becomes the evidence supplier for generated answers.