LEANN Vector Index: Smaller RAG Storage, Slower Retrieval

The LEANN vector index trades stored embeddings for query-time computation: the project reports shrinking a 60-million-chunk RAG index from 201 GB to 6 GB by keeping a pruned graph and recomputing embeddings along the search path. My interest is not just the smaller number. It is whether that storage saving is worth making retrieval take seconds rather than milliseconds.

LEANN comes from UC Berkeley's Sky Computing Lab and is open source under the MIT license. I am working from ByteMuni's write-up of the project and its reported benchmarks, not an independent reproduction. That distinction matters before treating any of these results as a capacity plan.

Why I'd consider recomputing embeddings

I normally think of embeddings as something to compute during ingestion and keep ready for retrieval. LEANN challenges the second half of that arrangement. A graph search visits only part of the collection, so storing every full-precision embedding may be an expensive convenience when storage is the limiting resource.

The source illustrates the size problem with 60 million chunks, 768 dimensions and four bytes per float32 value. That comes to 184,320,000,000 bytes before graph overhead. The important distinction for me is between the documents I need to retain and the extra representation I keep to search them.

LEANN does not remove the need for the underlying text. It replaces the stored full-precision vectors with a pruned graph and compact product quantization, or PQ, codes. When a query needs a precise comparison, LEANN recomputes the relevant embedding.

For a personal document assistant, I'd consider that exchange. For the intelligent search and AI-assisted editorial workflows I work on, I'd first ask whether retrieval sits in an interactive path or a background task. I would not give those two paths the same latency budget.

How LEANN makes recomputation less expensive

I see three parts of the design as important. Simply deleting the vectors would leave too much query-time work; LEANN also changes how it chooses and processes candidates.

Approximate scores narrow the expensive work

LEANN first scores neighboring candidates using their compact PQ codes. It then recomputes exact embeddings for the more promising candidates. A global queue lets the search revisit candidates whose approximate scores become relevant later.

My reading is that compression is being used as a filter, not asked to carry the entire accuracy requirement. That distinction matters because the source reports that PQ-only retrieval could not reach the same recall target at LEANN's storage budget.

Batching groups embedding work

LEANN collects candidates across search hops and embeds them together rather than processing every small group immediately. According to the reported latency breakdown, embedding recomputation still accounts for about 76% of query time.

That is where I'd expect hardware and embedding-model choices to deserve attention. I would measure those before spending time optimizing the surrounding application code.

Pruning preserves highly connected nodes

The graph also consumes storage. LEANN prunes it while allowing the top 2% of nodes by degree to retain more connections. The intent is to preserve frequently used routes rather than cut every node equally.

I would not treat ordinary graph pruning as equivalent. The source's synthetic FAISS experiment illustrates why reducing neighbor counts can damage recall, but it is not a full-scale LEANN reproduction. I would keep that experiment separate from the project's benchmark claims.

Is LEANN accurate enough to replace HNSW?

In the team's reported NQ result, LEANN and HNSW both achieved 25.5% downstream accuracy at 90% Recall@3. I would read that as evidence for a particular operating point, not a promise that every collection or question will behave identically.

The same comparison reports 2.48 seconds of retrieval for LEANN versus 0.05 seconds for HNSW. End-to-end RAG time was 23.34 seconds versus 20.95 seconds because generation took about 20 seconds in that setup.

I find the end-to-end comparison useful, but I would not borrow its reassurance for another product. If my generation step is shorter, retrieval could occupy a much larger share of the wait. If my interface shows retrieved passages before generating an answer, the retrieval delay matters on its own.

There are also two storage comparisons in the source: 201 GB to 6 GB from the repository benchmark, and 188 GB to 4 GB from the paper comparison. I would keep those pairs intact rather than combine their endpoints into a stronger-looking claim. Neither tells me the total disk requirement for my application, including documents, models and build artifacts.

Trying LEANN against a real document collection

I'd start with a bounded collection of documents I already understand. My first goal would be to find unacceptable retrieval behavior cheaply, not reproduce the largest benchmark.

Build a small, separate index

The documented installation command is uv pip install leann. The first run downloads an embedding model, so I'd separate that setup from query measurements. The following example uses the builder and searcher API shown in the source, without adding an LLM to the test.

from pathlib import Path
from leann import LeannBuilder, LeannSearcher

index_path = str(Path("editorial-demo.leann").resolve())

builder = LeannBuilder(backend_name="hnsw")
builder.add_text(
    "Draft articles must be reviewed before publication."
)
builder.add_text(
    "Video entries include a headline and an editorial description."
)
builder.build_index(index_path)

searcher = LeannSearcher(index_path)
results = searcher.search(
    "What happens before an article is published?",
    top_k=1,
)
print(results)

Those are sample documents, not a retrieval-quality test. I'd replace them with representative chunks and keep a separate set of questions whose relevant passages I can identify manually.

Measure retrieval before adding generation

I'd record retrieval latency, returned passages and index size first. Then I'd add generation and measure the complete response. This keeps an acceptable answer from hiding weak retrieval, or a slow generator from hiding expensive search.

My question set would include semantic questions, exact identifiers and ambiguous wording. In a newsroom CMS, I'd want to distinguish finding background coverage from finding a specific item. I would evaluate those needs separately rather than assume semantic retrieval should handle both.

Budget for construction, not just the finished index

The source says index construction still requires computing embeddings for all passages before building the graph. A small finished index therefore does not imply an equally small peak build footprint.

I'd measure peak storage during ingestion, retain the source documents and record the embedding-model version alongside the index. Because LEANN recomputes embeddings during search, I would treat a model change as a rebuild decision, not a casual configuration edit.

Test contention and the fallback path

Before connecting LEANN to an application, I'd run overlapping searches on the intended hardware. I would check how query-time embedding work affects the rest of the process, not just whether an isolated query finishes.

I'd also define the failure behavior: what the interface shows if retrieval misses its deadline, whether a keyword path remains available, and whether a user can retry without losing their work. For my first deployment, I'd prefer an optional research tool over a dependency in the publish flow.

What LEANN does not solve

I would not adopt LEANN merely because the index is smaller. It moves work into retrieval, and the source reports seconds-level search latency. If I needed consistently millisecond-oriented production retrieval and could accommodate the storage, I'd keep HNSW or evaluate DiskANN instead.

I would also keep exact-match requirements separate. The source recommends BM25 or hybrid search for identifiers and exact phrases. My preference would be to preserve an explicit exact-search route rather than make every lookup depend on embeddings.

Local storage is not a complete privacy decision either. The source lists local generation options and hosted API integrations. If my requirement were that documents stay on the machine, I'd check the embedding and generation configuration before calling the application private.

CPU performance is another unresolved point for me. The source mentions a CPU-only Linux installation but says the team has not published CPU latency numbers. I would not translate GPU results into a laptop performance promise.

The benchmark evidence is self-reported, and the source says it has not been independently reproduced at full scale. I'd use it to justify an experiment, not a replacement decision. I'd also ask how updates, deletions and rebuilds fit my workflow; the supplied material does not establish those operational details.

My first trial would be a local research assistant over a fixed document collection, with retrieval timing visible during testing. I'd leave latency-sensitive production search alone until the smaller index proves useful on my hardware, with my documents, under the wait my users will actually accept.