Retrieval recall is the RAG metric that actually decides quality

The failure everyone misdiagnoses

A RAG system gives a wrong answer. The team's first instinct is to rewrite the prompt. Then to try a bigger model. Then to add a second agent that checks the first one.

None of that helps if the source passage was never retrieved. The model was asked a question with the answer absent from its context, and it did what models do - produced the most plausible text available. The bug is upstream, and every hour spent on the prompt is an hour spent on the wrong component.

So before touching generation, answer one question with a number: when the answer exists in the corpus, how often does it make it into the context window?

How to measure recall@k on your own corpus

You do not need a labelling team. You need fifty to two hundred real questions with a known source, which most organisations can assemble from support tickets, internal wikis, or the questions users have already asked.

For each question, record the document and passage that genuinely answers it. Then run your retriever and check whether that passage appears in the top k results. The percentage that do is recall@k.

• Build the evaluation set from real questions, not invented ones - invented questions accidentally use corpus vocabulary and inflate the score
• Record the ground-truth passage identifier, not just the document, or you will not notice chunking failures
• Measure at the k you actually pass to the model, then again at a much larger k - the gap tells you whether the problem is retrieval or ranking
• Re-run the whole set on every ingestion change; it takes seconds and catches regressions that user reports would surface weeks later

Reading the two numbers

Recall@k at your production k and recall at a large k (say 100) form a diagnosis pair.

If both are low, the passage is not being found at all - an ingestion or embedding problem. Chunking is splitting the answer across boundaries, the document never made it into the index, or the embedding model is a poor fit for the domain vocabulary.

If large-k recall is high but production-k recall is low, retrieval works and ranking does not. That is the easier problem: add a reranking pass over the top candidates and most of the gap closes.

The fixes, in the order they pay

Chunking first. Structure-aware splitting that respects headings, table boundaries, and list items beats fixed-token windows on almost every real corpus, because the answer and the heading that gives it meaning stay together.

Then hybrid retrieval. Dense vectors handle paraphrase; lexical search handles exact identifiers, part numbers, error codes, and names - the things users actually search for and embeddings routinely miss. Running both and merging is a larger single improvement than any embedding-model upgrade we have measured.

Then reranking. A cross-encoder over the top fifty candidates, keeping ten. It costs latency, so measure whether your recall gap justifies it.

Query rewriting last. It helps with conversational follow-ups where the question is unintelligible standalone, and helps very little otherwise.

Why we keep this in Postgres

We default to pgvector on PostgreSQL rather than a dedicated vector store. Not because it is faster - for very large corpora it is not - but because the embedding rows live in the same transaction as the source rows.

When a document is updated, its chunks and vectors update in the same commit. There is no sync job, no eventual-consistency window where the system confidently cites a paragraph that was deleted yesterday. That class of bug is expensive to debug and invisible in every dashboard.

Below roughly ten million chunks with a properly tuned HNSW index, this holds comfortably. Above that, measure before moving.

Then, and only then, generation

Once recall is where it needs to be, generation quality becomes a real target rather than a proxy for retrieval failure. Citations become verifiable, because the cited passage is genuinely in context.

This is also the point at which an evaluation harness starts paying for itself: golden questions, expected sources, a regression threshold, and a CI gate. It is the only thing that lets a team change a prompt on a Friday.