The retrieval question, revisited

Two years ago the argument was that growing context windows would make retrieval pipelines obsolete. I made a version of that argument and was partly wrong.

Here is where the line actually is now, with the reasoning rather than the conclusion, because the line keeps moving and the reasoning does not.

the four variables#

Whether to retrieve or to stuff the context depends on four things:

Corpus size. If everything fits, stuff it. "Fits" now means a large document set — a codebase, a product's full documentation, a year of one team's tickets. It does not mean an enterprise's entire document store.

Query pattern. If you ask many questions against the same corpus, caching the prefix makes stuffing cheap. If every query touches a different slice of a large corpus, retrieval wins.

Latency budget. Prefill on a very large context takes real time even when cached. If you need sub-second responses, a small retrieved context is faster.

Citation requirement. If you must show which source supported a claim, retrieval gives you that structurally. Stuffing requires trusting the model's attribution, which is less reliable than people assume.

the decision, concretely#

Stuff the context when: the corpus is under a few hundred thousand tokens, you query it repeatedly (so caching applies), and you want the model to see relationships between distant parts.

The clearest example remains a codebase. Retrieval over code performs worse than whole-repository context because code's meaning lives in the relationships between files, and chunking destroys exactly that.

Retrieve when: the corpus is large, queries are diverse, latency matters, or you need citations.

Do both when: you have a large corpus with a hot subset. Stuff the hot subset — the style guide, the schema, the core documents — and retrieve from the tail.

what actually changed: retrieval became a tool#

The important architectural shift is not about size. It is that retrieval moved from a preprocessing step to a tool the model calls.

Old shape:

query → embed → search → rerank → stuff top-k → generate

One retrieval, before generation, with k fixed by you.

New shape:

query → model reasons → calls search tool → reads results
      → reasons → searches again with a better query → reads
      → generates answer with citations

The model decides what to search for, evaluates whether the results answered the question, and searches again with a refined query if not.

This is dramatically better and it is better for a specific reason: the first query is usually not the right query. A user asks about "the timeout issue." The right search is for the specific component's retry configuration, which you only know to search for after reading something else.

A fixed one-shot retrieval cannot do that. An agent with a search tool can.

what this means for your pipeline#

Embeddings matter less. When the model can iterate, a mediocre first retrieval is recoverable. When you had one shot, embedding quality was everything.

Keyword search came back. Hybrid search — BM25 plus vectors — consistently outperforms pure vector search, and for a model that can iterate, plain keyword search is often sufficient. Exact terms matter: error codes, function names, product names. Embeddings are bad at exact match and always were.

If you built a pure-vector pipeline in 2023, adding keyword search is probably your biggest available quality improvement.

Chunking matters less, and differently. Instead of chunking for retrieval, store documents whole and retrieve whole documents when they fit. Chunk only what is too large, and chunk on structural boundaries — sections, functions, headings — rather than by token count.

Metadata filtering matters more. The model can specify constraints: this project, this date range, this author. Filtering is cheap, precise, and it is frequently what the query actually needed. Make sure your index supports it.

the practical setup#

For most applications:

1. Postgres with pgvector plus full-text search. One system, hybrid search, metadata filters, and joins to your relational data.
2. Expose search as a tool, not as a preprocessing step. Let the model iterate.
3. Return whole documents where they fit; chunk on structure where they do not.
4. Cache the stable prefix — the instructions, the schema, the core reference material.
5. Measure retrieval quality separately from answer quality. If the answer is wrong, you need to know whether the right document was retrieved. Most teams cannot answer that and debug blind.

That last one is the single most useful piece of instrumentation in a retrieval system and almost nobody has it.

the thing I got wrong#

I said the RAG infrastructure category was solving a temporary problem. The capability got absorbed, as predicted. The infrastructure repositioned rather than disappearing, which I did not predict.

That is the third time I have watched this exact pattern and failed to apply the lesson. Infrastructure around a model limitation rarely dies. It moves to whatever the model still cannot do, which is usually one layer out.