Retrieval-augmented generation has become the default architecture for "ask questions about my data" features. It's also become the default reason support tickets pile up six weeks after launch. The pattern is consistent: the demo works, the pilot works, then the system meets real users and real corpora and the answers start drifting from "useful" to "hallucinated with citations."
We've been called in to fix RAG in production for everything from healthcare knowledge bases to code search to legal discovery to internal HR. The same seven mistakes show up almost every time. None of them are exotic. All of them are preventable. If you're shipping a RAG system in 2026, this is the checklist.
Mistake 1: chunking by character count, not by meaning
The textbook tutorial says "split documents into 512-token chunks with 50 tokens of overlap." Almost every starter RAG project does this, and almost every starter RAG project hits the same wall: chunks slice through the middle of clauses, separate questions from their answers, and split table rows from their headers.
The fix. Chunk semantically. For structured documents, chunk by section and subsection. For policy and contract content, chunk by clause. For tables, keep the entire row plus its column headers in a single chunk. For markdown and HTML, parse the document tree and use it. For PDFs, run layout detection first.
A semantic chunker beats a character-count chunker on retrieval quality by 15–40% in our internal evals — across every domain we've tested. The cost is one extra preprocessing step. The benefit shows up the day you ship.
Mistake 2: treating retrieval as a one-shot
The naive RAG pipeline is: embed the query, find top-k similar chunks, stuff them into the prompt. This works for direct factual lookups and breaks for everything else. Multi-hop questions ("which of our customers in Europe missed an SLA last quarter and what did we promise them?") cannot be answered from a single nearest-neighbor pass over the corpus.
The fix. Treat retrieval as a plan, not a call. Use query decomposition: break the question into sub-queries the model can answer with separate retrievals, then synthesize. Use HyDE (hypothetical document embeddings) — have the model write the document it's looking for, then retrieve against that. For complex questions, plan-then-retrieve outperforms retrieve-then-generate by a wide margin.
The cost is more model calls and more latency. The benefit is that the system actually answers the question.
Mistake 3: relying on dense embeddings alone
Dense semantic search is great at concept matching and bad at keyword fidelity. If a user searches for "ABC-123 error code" and the document says "error code ABC-123," dense search will find it. If the user searches for "section 4.2.7" or a specific function name or an SKU, dense search will quietly miss it because semantic similarity isn't the right notion of relevance.
The fix. Hybrid retrieval — combine dense (vector) and sparse (BM25 or equivalent keyword search), then merge with reciprocal rank fusion or a learned re-ranker. Hybrid retrieval is, in our experience, the single highest-leverage RAG upgrade. It's also the one that costs almost nothing to implement.
Add a cross-encoder re-ranker on the top 50–100 hybrid candidates and you'll find another 5–15 points of accuracy. The re-ranker pass costs a few hundred milliseconds and is worth every one of them.
Mistake 4: stuffing the context window because you can
When the model supports a 200,000-token context, it's tempting to throw fifty chunks at it and let it sort things out. Don't. Long-context performance degrades non-uniformly: the model attends well to the start and end of the context and loses signal in the middle. This is the "lost in the middle" effect, and it doesn't go away just because the model card says "1M tokens."
The fix. Retrieve aggressively, then prune. A small re-ranker plus a strict top-k (typically 5–15 chunks for most question types) outperforms throwing the whole haystack into the prompt. Order chunks by relevance, not by document order, and put the most important content at the start and end of the context.
You'll save tokens, you'll cut latency, and your accuracy will go up. All three at once.
Mistake 5: no evaluation set, no faithfulness check
Production RAG systems silently degrade. The corpus updates. The embedding model gets swapped. The chunking strategy gets tweaked. None of this triggers an alert because nobody set up a way to detect it. By the time users complain, the regression is weeks old and it's unclear which change caused it.
The fix. Build two eval sets and run them on every change.
The first is a retrieval eval — questions paired with the chunks that should be retrieved. Score retrieval@k. Track it over time.
The second is a faithfulness eval — questions paired with the answer the model should generate, with assertions about both correctness and grounding. Use a strong LLM as a judge to score whether the answer is supported by the retrieved chunks. Track hallucination rate as a first-class metric.
Without these, you are flying blind. With them, every prompt change, every model swap, every chunking tweak gets a number, and you can pick the change that actually helps.
Mistake 6: treating metadata as optional
A retrieval system without metadata is a retrieval system with one hand tied. When a user asks "what does our 2025 employee handbook say about parental leave?" — the 2025 and the employee handbook are filters, not semantic features. A pure embedding search will dutifully return passages from the 2022 handbook and the 2024 contractor policy, all semantically similar, all wrong.
The fix. Capture every piece of structured metadata you can at ingestion: document type, source, owner, last-modified date, jurisdiction, language, sensitivity, version, parent document. Apply them as hard filters before similarity search whenever the query implies them. Use the LLM (or a small classifier) to extract filter values from the query.
A metadata-filtered hybrid search is the most boringly effective RAG architecture there is. We use it as a default and only escalate from there.
Mistake 7: ignoring the human's actual question
The deepest mistake is also the most subtle. RAG systems get judged on whether they retrieved the right document. Users judge them on whether they answered the question. These are not the same thing.
A user asking "can I expense client dinners on this trip?" doesn't want the policy passage; they want a yes-or-no answer with the relevant rule cited. A user asking "what was the change in revenue from Q3 to Q4?" doesn't want the 10-Q; they want the number, with the source. A user asking "is this contract clause unusual?" doesn't want a similar contract; they want an opinion grounded in similar contracts.
The fix. Build the answer-shaping logic explicitly. Classify the question type. Pick a generation template per type. Require citations as structured outputs (not free-text "(Source: ...)" strings the model invents). Render the answer in the shape the user expected, with the evidence one click away.
This is the difference between a system that "uses RAG" and a system that answers questions. The former gets graded by engineers on retrieval@5. The latter gets graded by users on whether they keep coming back.
The compounding fix
If your RAG system is underperforming in production, you almost certainly have three or more of these mistakes simultaneously. Don't try to fix them all at once — instrument first.
Run the retrieval eval. Run the faithfulness eval. Look at the questions where the retrieval was right but the answer was wrong (generation problem) versus the ones where the retrieval missed (chunking, hybrid search, or metadata problem). Fix in that order. Re-run the evals after each change. Keep the changes that move the numbers. Roll back the ones that don't.
This is unglamorous work. It is also the only kind that ships. Most "RAG isn't working" stories end the same way: when someone finally builds the eval set, the path forward becomes obvious in a week.
Have a RAG system that's behaving worse in production than it did in the demo? Diffco runs RAG audits in five working days — eval set, retrieval analysis, prioritized fixes. Book a call and we'll tell you which of these seven you're hitting.
