Your RAG Problem Is Recall, and You Are Fixing Precision

The demo answered every question. In production it confidently answers the wrong thing, and the standard response is to add a reranker. For the most common failure, that cannot possibly work — not because rerankers are bad, but because reordering a list cannot add something that was never on it.

RAG has two distinct retrieval failures. They produce an identical symptom, a wrong answer delivered confidently, and they have opposite fixes. Most teams never distinguish them, which is why so much RAG tuning feels like guesswork.

Recall@k is a ceiling on the entire pipeline

Everything downstream of retrieval operates on the documents retrieval returned. The reranker reorders them. The prompt frames them. The model reasons over them. None of those stages can reach back into the corpus for something that was not fetched.

So if recall@10 is 0.70, then for three questions in ten the answer is absent from the context before the model has seen anything. The ceiling on end-to-end accuracy is 70%, and no amount of work downstream moves it. That is worth internalising because it inverts the usual debugging order: the model is the last place to look, not the first.

corpus ──► retrieve k docs ──► rerank ──► prompt ──► model ──► answer
             ^                  ^
             |                  |
      recall decided here       can only reorder what
      (a hard ceiling)          arrived; cannot raise
                                the ceiling

  recall@10 = 0.70  ──►  30% of questions are unanswerable
                          before the model is invoked at all

Why a reranker cannot fix a recall failure

A cross-encoder reranker takes the candidate set and scores each document against the query properly, rather than through the lossy proxy of embedding distance. It is genuinely good at that. It is also, by construction, a function whose domain is the retrieved set.

If the correct document sits at rank 400 and you retrieved 20, the reranker never sees it. Recall@20 is unchanged after reranking because reranking cannot change which documents were fetched. What it changes is which of those 20 arrives first, and that is precision.

What the published numbers actually show

The clearest public figures come from a controlled comparison on T2-RAGBench, a corpus of 23,088 queries over 7,318 financial documents containing mixed text and tables. Ten retrieval strategies were run over the same corpus, which is the part that matters — most published comparisons vary the corpus as well as the method, so the numbers are not comparable.

Two things stand out. Moving from dense-only to hybrid fusion lifts Recall@5 from 0.587 to 0.695 — that is a genuine recall improvement, because it changes what gets fetched. And the jump from 0.308 to 0.472 at Recall@1 after reranking is the precision effect: the same candidates, reordered so the right one arrives first.

The honest caveat is that this is one benchmark on one domain, and the domain matters enormously. Financial documents are dense with exact terminology — instrument names, ticker symbols, line-item labels — and exact terms are precisely what lexical matching is good at and what embeddings blur. Do not read this as "BM25 beats embeddings"; read it as "your corpus decides, and the default assumption is untested."

Why hybrid fusion uses ranks and not scores

Having decided to run both retrievers, you have to combine two result lists. The instinct is a weighted sum of the scores. That does not work, for a reason worth understanding rather than working around.

BM25 produces an unbounded score whose magnitude depends on term frequencies, document length, and corpus statistics. Cosine similarity produces a bounded value in a completely different range with a completely different distribution. There is no principled conversion between them, so any weight you choose is arbitrary, and the arbitrary weight that worked on your test set will drift as the corpus grows.

Reciprocal Rank Fusion sidesteps this by discarding the scores and using only positions. A document's contribution is a function of where it ranked, not what it scored, so the two lists become comparable by construction.

def reciprocal_rank_fusion(
    result_lists: list[list[str]], k: int = 60
) -> list[tuple[str, float]]:
    """Fuse ranked lists by position, never by score.

    BM25 scores are unbounded and corpus-dependent; cosine similarity is
    bounded and differently distributed. No weighted sum of the two is
    principled, so RRF uses rank alone and the lists become comparable.

    k dampens the weight of the very top positions. The value 60 is the
    convention from the original paper and is a starting point, not a
    tuned constant — raise it to flatten the curve, lower it to let rank 1
    dominate.
    """
    scores: dict[str, float] = {}

    for results in result_lists:
        for position, doc_id in enumerate(results, start=1):
            scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + position)

    return sorted(scores.items(), key=lambda pair: pair[1], reverse=True)

Documents surfaced by both retrievers accumulate contributions from both lists and rise. Documents found by only one still appear, which is what preserves the complementary strengths — the exact-term match BM25 catches and the paraphrase match embeddings catch.

Diagnosing which failure you actually have

This is the step teams skip, and it costs weeks. You need a set of questions with the document that answers each one labelled. A hundred is enough to be decisive; it does not need to be thousands, and building it by hand is a day of work that saves considerably more.

def diagnose(cases, retrieve, k: int = 20):
    """Separate recall failures from ranking failures.

    cases: [(query, expected_doc_id), ...]
    retrieve: (query, k) -> ordered list of doc_ids
    """
    missing, buried, top = 0, 0, 0

    for query, expected in cases:
        results = retrieve(query, k)

        if expected not in results:
            missing += 1          # recall failure — fix the fetch stage
        elif results[0] != expected:
            buried += 1           # precision failure — reranking helps here
        else:
            top += 1

    n = len(cases)
    print(f"recall@{k}    {(n - missing) / n:.3f}")
    print(f"not retrieved {missing / n:.3f}  <- more hybrid/chunking work")
    print(f"retrieved, not first {buried / n:.3f}  <- reranker earns its cost")
    print(f"already first {top / n:.3f}")

The output tells you where to spend. A large not-retrieved fraction means work on the fetch stage: add lexical search alongside vector search, revisit chunking, check whether the query and the corpus even share vocabulary. A large retrieved-but-not-first fraction is exactly what a reranker is for, and it will pay for itself.

Most recall failures trace back to chunking

When documents are missing from the candidate set entirely, the cause is usually upstream of retrieval. Fixed-size chunking at 500 or 1,000 tokens works on a curated demo corpus because that corpus is short, well-structured, and semantically coherent. Production corpora are none of those.

The specific damage is predictable, and it is catalogued well in Your Chunks Failed Your RAG in Production. A table split across two chunks leaves both halves meaningless — the headers are in one and the figures in the other. A sentence whose subject was named a paragraph earlier becomes an unresolvable pronoun. A definition and the passage that depends on it land in different chunks and never co-occur in a result set. None of these are retrieval bugs; retrieval is faithfully returning chunks that no longer contain a retrievable idea.

Common mistakes

1. Adding a reranker to fix a recall problem. It cannot, by construction, and the effort spent proves nothing except that the problem was elsewhere.
2. Assuming dense retrieval is the strong baseline. On at least one careful benchmark it loses to BM25 on nearly every metric, and nobody tested it on your corpus.
3. Combining BM25 and vector scores with a weighted sum. The ranges are incompatible, the weight is arbitrary, and it drifts as the corpus grows. Fuse on rank.
4. Evaluating end to end only. An answer-quality score cannot tell you whether the failure was fetch, rank, or generation, so it cannot tell you what to fix.
5. Chunking at a fixed token count without looking at what the documents are. Tables and cross-referential prose break in ways that are invisible until recall is measured.
6. Increasing k instead of improving recall. Retrieving 50 documents rather than 10 raises recall and buries the answer deeper in context, converting a recall problem into a precision problem and a cost problem at once.

Security and performance considerations

Retrieval is an access-control surface and is frequently treated as though it is not. If the index contains documents some users may not read, filtering after retrieval is too late — the content has already entered a context window that may be logged, cached, or summarised into a response. Permission filters belong in the query, enforced by the store, not applied to the result set afterwards.

There is also an injection surface. Retrieved documents are untrusted content placed directly into the model's context, which is the same structural weakness behind tool description attacks: the model cannot distinguish retrieved data from instruction. A document containing plausible-looking directives will be read as directives. Anything indexed from a source users can write to needs to be treated accordingly.

On cost, the two-stage pipeline adds a reranking call per query, and raising k raises both retrieval latency and the tokens sent to the model on every request. Both scale with traffic rather than with corpus size, which makes them easy to underestimate at design time — the compounding effect is the same one behind runaway agent spend, where per-call costs look reasonable and the per-run total does not.

Troubleshooting

1. Answers are confidently wrong rather than hedged — usually a recall failure. The model had material and it was the wrong material, which produces confidence rather than uncertainty.
2. Works for questions phrased like the docs, fails on paraphrases — vocabulary mismatch. Lexical-only retrieval, or embeddings that do not fit the domain.
3. Works for paraphrases, fails on exact identifiers and codes — the reverse. Embeddings blur exact terms; add lexical retrieval.
4. Recall is fine and answers are still wrong — the answer is arriving in context but too late or too diluted. This is where a reranker earns its cost.
5. Quality degraded over months with no code change — corpus drift. New documents entered the index whose vocabulary the embedding model was not chosen for.
6. Tables answered incorrectly with confident numbers — near-certain chunking damage. Check whether headers and rows survive in the same chunk.

The discipline that makes RAG tractable is refusing to tune blind. Measure recall against labelled cases first, decide from the split which stage is failing, and only then choose a fix. It is a day of unglamorous work and it replaces the alternative, which is changing embedding models for a fortnight and hoping.

If you are building a retrieval-grounded assistant and want it evaluated properly rather than demoed convincingly, that is what our generative AI engineering work covers. If something is already live and answering badly, a short diagnostic engagement starts by measuring recall before anything gets rebuilt.