You shipped your RAG pipeline. It passed your internal tests. Your demo looked clean. Then production happened — and suddenly your AI is confidently telling users the wrong thing, hallucinating details that aren't in your documents, or returning completely irrelevant chunks that make the whole system look broken.
Welcome to the gap between "RAG that works in a notebook" and "RAG that works when real users hit it."
This post is for builders who are past the tutorial phase. You've got a retrieval-augmented generation system running, but something is wrong and you're not entirely sure what. I'm going to walk you through the five most common production RAG failures I see in 2026, exactly why they happen, and how to fix them with real tools and real code.
Let's get into it.
---
Mistake #1: Chunk Size Disasters
This is the silent killer of RAG quality. Most builders pick a chunk size once — usually whatever the tutorial used — and never revisit it. Then they wonder why their retrieval returns context that's either too narrow to be useful or so bloated it drowns the signal.
Why it breaks things:
Chunks that are too small (say, 128 tokens) lose surrounding context. Your embedding captures a fragment of a sentence, and when retrieved, the LLM doesn't have enough information to answer correctly. Chunks that are too large (1024+ tokens) embed multiple concepts into a single vector, making similarity search imprecise. You retrieve a chunk that's partially relevant, and the irrelevant parts actively mislead the model.
The real-world failure mode:
Imagine a legal document RAG system. A user asks about termination clauses. Your 128-token chunks split a single clause across three separate chunks. The retriever grabs one of them — the one that mentions "30 days notice" — but misses the adjacent chunk that says "unless the contract is for a fixed term." Your AI answers with incomplete information and creates legal liability.
The fix:
Stop treating chunk size as a static configuration. Use LangChain's RecursiveCharacterTextSplitter with overlap, and test multiple chunk sizes against your actual query distribution.
```python
from langchain.text_splitter import RecursiveCharacterTextSplitter
splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=64,
separators=["\n\n", "\n", ".", " "]
)
chunks = splitter.split_documents(docs)
```
The `chunk_overlap` parameter is critical — it ensures context bleeds across chunk boundaries. Start at 10-15% of your chunk size.
More importantly, evaluate chunk quality with Ragas. Run your retrieval pipeline against a test set and measure `context_recall` — if it's below 0.7, your chunks are losing information. Adjust chunk size, reindex, and retest. This is not a one-time task. As your document corpus evolves, your optimal chunk size may shift.
For document types with natural structure (markdown, HTML, code), use structure-aware splitting rather than character-based splitting. Respect the document's own hierarchy.
---
Mistake #2: Embedding Model Mismatches
You built your index with one embedding model. Then you switched models for cost reasons, or upgraded to a newer version, and forgot to reindex. Or worse — you're querying with a different model than you indexed with. Your similarity scores become meaningless, and retrieval quality collapses.
Why it breaks things:
Embedding models don't share vector spaces. A vector produced by `text-embedding-ada-002` is not comparable to one produced by `text-embedding-3-large` or Cohere's `embed-english-v3.0`. If your stored vectors came from model A and your query vector comes from model B, cosine similarity will return garbage. The math is valid but the semantic comparison is not.
This also happens subtly when you use different embedding models for different document types — PDFs embedded with one model, web-scraped content with another — all stored in the same Pinecone or Chroma index.
The fix:
Enforce model consistency at the infrastructure level, not the application level.
```python
import hashlib
import json
def get_index_metadata(model_name: str, model_version: str) -> dict:
return {
"embedding_model": model_name,
"embedding_version": model_version,
"index_fingerprint": hashlib.md5(
f"{model_name}:{model_version}".encode()
).hexdigest()
}
def validate_query_model(query_model: str, index_metadata: dict) -> bool:
return query_model == index_metadata["embedding_model"]
```
Store your embedding model name and version as index metadata. On every query, validate that the query embedding model matches the index embedding model. If they don't match, raise an error rather than returning bad results silently.
When you upgrade embedding models, you must reindex everything. There's no shortcut. With Pinecone, create a new index with the new model, run your ingestion pipeline against it, validate quality with Ragas, then swap the index pointer in your application config. Blue-green deployment for vector indexes.
For OpenAI embeddings specifically, note that `text-embedding-3-large` with `dimensions=256` can outperform `ada-002` at a fraction of the cost — but only if your entire pipeline uses the same dimensionality setting consistently.
---
Mistake #3: Retrieval Scoring Bugs
Your retriever returns the top-k chunks by cosine similarity. Sounds reasonable. In practice, this is often the wrong signal, and builders don't realize it until users start complaining.
Why it breaks things:
Cosine similarity measures vector direction, not semantic relevance to the specific question being asked. A chunk about "machine learning model training" and a query about "how do I train my dog" can have surprisingly high cosine similarity because both involve "training" in a general sense. Dense retrieval alone is brittle for out-of-distribution queries.
The other failure mode: you're retrieving top-5 chunks, but chunks 3, 4, and 5 are marginally relevant at best. You're padding your context with noise, and the LLM's attention gets diluted. Studies consistently show that LLMs perform worse when irrelevant context is present than when no context is provided at all.
The fix:
Implement a two-stage retrieval pipeline with Cohere Rerank.
```python
import cohere
from typing import List
co = cohere.Client("your-api-key")
def rerank_chunks(query: str, chunks: List[str], top_n: int = 3) -> List[dict]:
response = co.rerank(
query=query,
documents=chunks,
top_n=top_n,
model="rerank-english-v3.0"
)
# Filter by relevance score threshold
filtered = [
r for r in response.results
if r.relevance_score > 0.4
]
return filtered
```
Stage 1: Dense retrieval from your vector store (retrieve top-20 candidates).
Stage 2: Cohere Rerank scores each candidate against the query using a cross-encoder model, then returns only the top-3 that actually matter.
The relevance score threshold is important — don't just take top-n blindly. If your best chunk scores 0.2, that's a signal that nothing in your index is relevant to this query, and you should tell the user that rather than hallucinating an answer.
Also implement hybrid search if your vector store supports it. Pinecone supports sparse-dense hybrid retrieval, combining BM25 keyword matching with dense semantic search. This dramatically improves retrieval for queries that contain specific terms, product names, or technical identifiers that semantic search alone handles poorly.
Track your retrieval metrics in LangSmith. Set up traces on every retrieval call and monitor the distribution of your top-1 similarity scores over time. A sudden drop in average similarity scores often indicates a data quality issue before it becomes a user-facing problem.
---
Mistake #4: Context Window Overflow
You're retrieving 10 chunks, each 512 tokens, plus your system prompt, plus the conversation history, plus the user's question. You've just blown past your context window limit, and the LLM is either throwing an error, silently truncating your context, or — most dangerously — truncating the most relevant parts because they happened to be at the end of your context string.
Why it breaks things:
LLMs have finite context windows. Even with 128K token models, you can hit limits faster than you think when you factor in system prompts, conversation history, and retrieved chunks. More critically, research on "lost in the middle" effects shows that LLMs pay less attention to information in the middle of long contexts. If you're stuffing 10 chunks in, chunks 4-7 may effectively be invisible to the model.
The fix:
Implement context budget management as a first-class concern in your pipeline.
```python
import tiktoken
def count_tokens(text: str, model: str = "gpt-4o") -> int:
enc = tiktoken.encoding_for_model(model)
return len(enc.encode(text))
def build_context_within_budget(
chunks: List[str],
system_prompt: str,
conversation_history: str,
query: str,
max_tokens: int = 8000,
model: str = "gpt-4o"
) -> str:
# Reserve tokens for fixed components
fixed_tokens = (
count_tokens(system_prompt, model) +
count_tokens(conversation_history, model) +
count_tokens(query, model) +
500 # buffer for response
)
available_for_context = max_tokens - fixed_tokens
# Add chunks until budget is exhausted
selected_chunks = []
used_tokens = 0
for chunk in chunks: # chunks should already be ranked by relevance
chunk_tokens = count_tokens(chunk, model)
if used_tokens + chunk_tokens <= available_for_context:
selected_chunks.append(chunk)
used_tokens += chunk_tokens
else:
break
return "\n\n---\n\n".join(selected_chunks)
```
This approach ensures you never overflow the context window and always include your highest-relevance chunks first (assuming your chunks are already reranked).
For long-running conversations, implement a sliding window on conversation history — summarize older turns rather than including them verbatim. LangChain has `ConversationSummaryBufferMemory` for this exact use case.
If you're building complex multi-step agent systems and want a deeper framework for managing production AI behavior at scale, the GUARDIAN Framework covers context management, cost control, and production monitoring in detail — it's the systematic approach I'd recommend for anyone running RAG in a commercial environment.
---
Mistake #5: Eval Blindness
This is the most expensive mistake on this list, because it compounds every other mistake silently. You have no systematic way to measure whether your RAG system is actually answering correctly. You're flying blind, and you won't know something broke until a user tells you — or worse, until they stop using your product.
Why it breaks things:
RAG quality is not binary. A response can be factually correct but miss the nuance the user needed. It can cite the right source but misinterpret it. It can answer the literal question while ignoring the actual intent. Without quantitative evaluation, you can't distinguish between these failure modes, you can't measure the impact of changes to your pipeline, and you can't prioritize which problems to fix first.
Most builders do ad-hoc testing — they ask the system a few questions, it looks okay, they ship. This is not evaluation. This is vibes.
The fix:
Build a proper evaluation harness using Ragas and LangSmith.
```python
from ragas import evaluate
from ragas.metrics import (
faithfulness,
answer_relevancy,
context_recall,
context_precision
)
from datasets import Dataset
test_data = {
"question": [...], # user queries
"answer": [...], # your RAG system's answers
"contexts": [...], # retrieved chunks used
"ground_truth": [...] # correct answers (human-labeled)
}
dataset = Dataset.from_dict(test_data)
results = evaluate(
dataset,
metrics=[
faithfulness, # is the answer grounded in the context?
answer_relevancy, # does the answer address the question?
context_recall, # did retrieval find the right chunks?
context_precision # are retrieved chunks actually relevant?
]
)
print(results)
```
The four Ragas metrics tell you different things:
Each metric points to a different part of your pipeline to fix. Low faithfulness is a prompting problem. Low context recall is a chunking or embedding problem. Low context precision is a retrieval scoring problem.
Connect your Ragas evaluation to LangSmith for continuous monitoring:
```python
from langsmith import Client
ls_client = Client()
ls_client.create_run(
name="rag_eval_weekly",
run_type="chain",
inputs={"test_set_size": len(test_data["question"])},
outputs={
"faithfulness": results["faithfulness"],
"answer_relevancy": results["answer_relevancy"],
"context_recall": results["context_recall"],
"context_precision": results["context_precision"]
}
)
```
Run this evaluation weekly, or on every significant change to your pipeline. Set alert thresholds — if faithfulness drops below 0.75, you get notified before users do.
Build your test set from real user queries, not synthetic ones. The queries that break your system in production are never the ones you thought to test in development.
---
The Deeper Pattern: These Mistakes Are Connected
Here's what I want you to notice: these five mistakes aren't independent. They compound.
Bad chunk sizes → poor embedding quality → retrieval scoring becomes unreliable → you compensate by retrieving more chunks → context window overflows → your eval doesn't catch any of it because you're not measuring systematically.
Fixing one without fixing the others gives you marginal improvement. The builders who ship RAG systems that actually work in production treat this as a system — they have a methodology for diagnosing which layer of the stack is failing, and they fix layers in the right order.
That's the difference between debugging by intuition and debugging by framework. If you want the full systematic approach — including how to structure your RAG pipeline for observability from day one, how to build evaluation datasets efficiently, and how to handle the edge cases that will absolutely hit you in production — that's what the ORACLE Framework PDF covers in depth. I'll keep this post focused on the five mistakes, but know that there's a structured methodology behind all of this that goes considerably further.
---
Quick Wins You Can Implement Today
Before you go, here's your action list:
1. Audit your chunk sizes — pull 20 random chunks from your index and read them. Do they make sense in isolation? Do they contain complete thoughts? If not, adjust and reindex.
2. Verify embedding model consistency — check that your ingestion pipeline and query pipeline use identical model names and versions. Add an assertion that throws if they don't match.
3. Add Cohere Rerank to your retrieval — even if you only use it for your most important query types to start. Measure the difference in answer quality.
4. Implement token counting — add `tiktoken` to your pipeline and log the token count of every context you send to the LLM. You'll immediately see where you're overflowing.
5. Build 50 evaluation examples — take real user queries from your logs, label the correct answers, and run Ragas against them. You now have a baseline. Everything you change can be measured against it.
If you're earlier in your