RAG in Production: Four Stages, Four Ceilings, and How to Tell Which One Failed

The demo answered everything. Production answers confidently and wrongly, and the team's response is to change something — a different embedding model, a bigger chunk size, a reranker, a stronger generation model. Sometimes it helps. Nobody can say why, because nothing measured which part was broken.

This is not a discipline problem. It follows from the architecture: a RAG pipeline has four stages, each one caps the next, and all four produce an identical symptom when they fail. Without stage-level measurement there is no way to tell them apart, and an end-to-end quality score reports only that the answer was wrong — which was the starting observation.

Why the ceiling framing matters more than a score

Consider a pipeline where chunking split a financial table so headers sit in one chunk and figures in another. Neither chunk answers the question. Retrieval is working perfectly — it returns the most similar chunks available — and the correct answer was destroyed before retrieval ever ran.

Now add a reranker. It reorders chunks that do not contain the answer. Swap the embedding model — even for one of the strongest available, which the T2-RAGBench comparison found lost to BM25 on nearly every metric for text-and-table documents. It embeds chunks that do not contain the answer. Upgrade the generation model. It reasons more capably over chunks that do not contain the answer. Three real improvements, three unchanged outcomes, and a growing suspicion that RAG simply does not work well.

corpus
  |
  v  INGESTION      what survives chunking is all that can exist
  |                 ceiling: is the answer intact in some chunk?
  v  RETRIEVAL      fetches k candidates from what exists
  |                 ceiling: context recall
  v  RANKING        orders and truncates to what fits usefully
  |                 ceiling: context precision
  v  GENERATION     reasons over what arrived
  |                 ceiling: faithfulness
  v
answer

  the lowest ceiling decides the system;
  effort spent above it produces no measurable change

That last line is the practical value of the model. It predicts, before you do the work, that a change at a stage above the binding constraint will not move the numbers — which is the single most common wasted fortnight in RAG projects.

Four metrics, four stages

The standard evaluation vocabulary maps onto the stages more cleanly than it is usually presented. Taking the Ragas definitions as the reference point, each metric isolates one stage and says nothing about the others.

Ingestion has no metric of its own, which is exactly why it is the stage teams overlook. It is measured indirectly: persistently low context recall that does not improve when retrieval is changed is an ingestion problem wearing a retrieval costume.

Reading the pattern to localise the failure

Individually these numbers are mildly interesting. Together they are diagnostic, because each combination has essentially one explanation.

The second row is worth dwelling on because it is the most expensive mistake in the table when read backwards. Adding a reranker to a low-recall pipeline cannot help — reordering a candidate set cannot introduce a document that was never fetched — and that argument, with the measured numbers behind it, is the subject of recall versus precision in retrieval.

You need about a hundred labelled cases

None of this works without a set of questions paired with the document that answers each. That is the piece teams skip, and it is a day of unglamorous work that replaces weeks of substitution and hope.

A hundred cases is enough to be decisive. The goal is not statistical rigour but a clear split — how often the answer was never retrieved, how often it was retrieved and buried, how often it arrived and was ignored. Those three proportions point at a stage, and the pointing is unambiguous long before the sample is large.

def stage_report(cases, retrieve, generate, k: int = 20):
    """Attribute failures to a stage rather than scoring the system.

    cases: [(query, expected_doc_id, expected_answer_substring), ...]
    """
    never_retrieved = buried = ignored = ok = 0

    for query, expected_doc, expected_text in cases:
        results = retrieve(query, k)

        if expected_doc not in results:
            never_retrieved += 1          # ingestion or retrieval
            continue

        answer = generate(query, results)

        if expected_text.lower() not in answer.lower():
            # The document was present. Position decides which stage.
            rank = results.index(expected_doc)
            if rank > 4:
                buried += 1               # ranking
            else:
                ignored += 1              # generation
        else:
            ok += 1

    n = len(cases)
    print(f"never retrieved  {never_retrieved / n:.2f}  -> ingestion / retrieval")
    print(f"retrieved, buried {buried / n:.2f}  -> ranking")
    print(f"present, ignored  {ignored / n:.2f}  -> generation")
    print(f"correct           {ok / n:.2f}")

This is deliberately cruder than a full evaluation framework. It answers one question — which stage — and answering that correctly is worth more than a dashboard of scores nobody can act on.

The stage with no metric is the one that fails most

Fixed-size chunking at five hundred or a thousand tokens works on a demo corpus because that corpus is short, well-structured, and semantically coherent. Production corpora are none of those, and the damage is predictable enough to enumerate.

• A table split across chunks leaves headers in one and figures in the other. Both halves are individually meaningless and neither answers the question.
• A sentence whose subject was named a paragraph earlier becomes an unresolvable pronoun with no referent in its chunk.
• A definition and the passage depending on it land in separate chunks and never co-occur in a result set.
• Boilerplate — headers, footers, navigation — dominates short chunks and drags their embeddings toward each other, so everything looks moderately similar to everything.

None of these are retrieval bugs. Retrieval is faithfully returning the most similar chunks available, and the chunks no longer contain retrievable ideas. That is why changing embedding models does not fix it, and why persistent low recall that survives a retrieval change should send you upstream rather than sideways.

Where storage enters the picture

Storage choice sits underneath retrieval and is usually made first, on the wrong criteria. It is worth noting that it constrains the stages above it in ways that are easy to miss — a vector column too wide to index means no index at all, and an index that does not fit in memory turns a fast retrieval stage into the system's latency budget. That decision is arithmetic rather than preference, and is covered in the pgvector sizing analysis.

The relevant point for this model is that storage is not a stage. It is a constraint on the retrieval stage, and no storage decision improves a pipeline whose binding ceiling is ingestion or generation. Teams migrate databases to fix problems that a chunking change would have solved for nothing.

Common mistakes

1. Evaluating end to end only. It reports that answers are bad without indicating which stage produced them, so every subsequent change is a guess.
2. Adding a reranker to a low-recall pipeline. Reordering cannot introduce a document that was never retrieved.
3. Increasing k to raise recall. It works, and it buries the answer deeper in a longer context while billing every additional document on every request.
4. Increasing chunk size to fix splitting. It converts a recall problem into a precision and cost problem. Chunk on structure first.
5. Changing embedding models to fix an ingestion problem. The new model embeds the same damaged chunks.
6. Migrating to a dedicated vector database to fix generation quality. Storage constrains retrieval only.
7. Treating faithfulness as a retrieval metric. It measures whether the model used the context it was given, which is a generation property.
8. Building the labelled set after the first three fixes have failed. It is a day of work and it should precede them.

Security and performance considerations

Retrieval is an access-control surface. If the index holds documents some users may not read, the permission filter belongs in the query and must be enforced by the store — filtering after retrieval is too late, because the content has already entered a context window that may be logged, cached, or summarised into a response the user does see.

Retrieved chunks are also untrusted content placed directly into the model's context, which is the same structural weakness behind instruction-carrying tool descriptions: the model cannot distinguish retrieved data from instruction. Anything indexed from a source users can write to should be treated as an injection vector rather than as reference material.

On cost, the two levers that improve quality — larger k and a reranking pass — both scale with request volume rather than with corpus size, which makes them easy to underestimate when the corpus is the thing you are thinking about. Every additional retrieved document is billed on every query, which compounds in the way described in why agent bills explode: individually reasonable requests, an unreasonable total.

Troubleshooting by symptom

1. Confidently wrong rather than hedged — the model had material and it was the wrong material. Check context recall before anything else.
2. Correct for questions phrased like the documents, wrong for paraphrases — vocabulary mismatch at retrieval. Consider hybrid search.
3. Correct for paraphrases, wrong for exact identifiers and codes — the reverse. Embeddings blur exact terms; lexical retrieval catches them.
4. Tables answered with confident wrong numbers — near-certain chunking damage. Check whether headers and rows survive together.
5. Recall and precision both good, answers still wrong — generation. The context arrived and was not used; look at the prompt before the retriever.
6. Quality degraded over months with no deploy — corpus drift. New documents entered with vocabulary the embedding model was not chosen for.
7. One document type always answered badly — an ingestion path that handles that format poorly. Inspect its chunks directly rather than its retrieval scores.

The discipline the model asks for is small: measure each stage separately, find the lowest ceiling, and work only there until it moves. What it replaces is the alternative — changing plausible things in sequence and hoping — which is expensive precisely because several of those changes are genuine improvements to stages that were never the constraint.

If you are building a retrieval-grounded product and want it evaluated rather than demonstrated, that is what our generative AI practice does. Where something is already live and answering badly, an assessment engagement starts by localising the failing stage before recommending anything be rebuilt.