Most RAG hallucinations are a retrieval problem, not an LLM problem. The model hallucinates because you handed it the wrong context, or not enough of the right one. Fix the retrieval layer and you fix most of the hallucinations.

This is what we’ve learned deploying RAG systems across legal research, enterprise knowledge bases, and customer support platforms. None of these findings are theoretical.

The Root Cause

When an LLM hallucinates in a RAG system, the instinct is to blame the model. Swap GPT-4o for Claude, fine-tune on domain data, engineer a better prompt. Sometimes that helps. Usually it doesn’t.

The actual failure modes, in order of frequency:

  1. The relevant chunk wasn’t retrieved. The right document exists in your corpus, but your retrieval didn’t surface it.
  2. The retrieved chunks had conflicting information. The model tried to reconcile contradictions and invented a middle ground.
  3. The chunks were technically correct but missing necessary context. The model filled in the gaps with plausible-but-wrong assumptions.
  4. There was no relevant document. The model was asked to answer from context that didn’t exist, and it tried anyway.

Number four is actually the easiest to solve (tell the model to say “I don’t know”). Numbers one through three are where the engineering work lives.

Hybrid Retrieval

Pure vector search is not good enough for production RAG. Dense embeddings are excellent at semantic similarity but they have a known weakness: they miss exact matches.

Consider a legal research system. A user searches for “Section 12(g) exemption threshold”. A dense retrieval system might return documents about securities exemptions generally, which are semantically close but miss the specific statutory language. BM25 (sparse retrieval) will find the exact phrase.

The solution is to run both and merge the results:

from rank_bm25 import BM25Okapi
import numpy as np

def hybrid_retrieve(query: str, corpus: list[str], embeddings: np.ndarray,
                    query_embedding: np.ndarray, alpha: float = 0.5,
                    top_k: int = 20) -> list[int]:
    # Dense scores
    dense_scores = (query_embedding @ embeddings.T).flatten()
    dense_scores = (dense_scores - dense_scores.min()) / (dense_scores.max() - dense_scores.min() + 1e-8)

    # Sparse scores (BM25)
    tokenized = [doc.split() for doc in corpus]
    bm25 = BM25Okapi(tokenized)
    sparse_scores = np.array(bm25.get_scores(query.split()))
    sparse_scores = (sparse_scores - sparse_scores.min()) / (sparse_scores.max() - sparse_scores.min() + 1e-8)

    # Weighted combination
    combined = alpha * dense_scores + (1 - alpha) * sparse_scores
    return combined.argsort()[::-1][:top_k].tolist()

The alpha parameter needs tuning per domain. For technical documentation with precise terminology, lean toward BM25 (alpha = 0.3). For conversational queries against prose documents, lean toward dense (alpha = 0.7). Run an offline evaluation with labeled queries to find your optimal value.

Reciprocal Rank Fusion (RRF) is an alternative to weighted scoring. It’s more robust when the two score distributions have very different shapes:

def rrf_merge(dense_ranks: list[int], sparse_ranks: list[int], k: int = 60) -> list[int]:
    scores: dict[int, float] = {}
    for rank, doc_id in enumerate(dense_ranks):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    for rank, doc_id in enumerate(sparse_ranks):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

Cross-Encoder Reranking

Bi-encoder retrieval (query and document embedded separately) is fast but approximate. Cross-encoders (query and document processed together) are slower but dramatically more accurate.

The production pattern: retrieve 20–50 candidates with hybrid search, then rerank the top 20 with a cross-encoder, then pass the top 5 to the LLM.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

def rerank(query: str, candidates: list[str], top_n: int = 5) -> list[tuple[str, float]]:
    pairs = [(query, doc) for doc in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return ranked[:top_n]

The ms-marco-MiniLM-L-6-v2 model is a good default: fast enough to run synchronously (under 100ms for 20 candidates on a T4) and meaningfully better than bi-encoder ranking alone. For high-stakes domains like legal or medical, consider cross-encoder/ms-marco-electra-base, which trades some latency for higher accuracy.

Chunking Strategy

Chunking decisions have more impact on retrieval quality than model choice. Two principles:

Sentence-aware chunking. Never split mid-sentence. Use a library like nltk.sent_tokenize or spacy to detect sentence boundaries before chunking. A 512-token chunk that cuts a sentence in half will produce a retrieval hit that confuses the LLM.

Overlapping windows. Use a 10–20% overlap between adjacent chunks. This ensures that information at chunk boundaries isn’t lost. If your chunk size is 400 tokens, your overlap should be 40–80 tokens.

def chunk_document(text: str, chunk_size: int = 400,
                   overlap: int = 60) -> list[str]:
    import nltk
    sentences = nltk.sent_tokenize(text)
    chunks, current_chunk, current_len = [], [], 0

    for sentence in sentences:
        tokens = sentence.split()  # approximate token count
        if current_len + len(tokens) > chunk_size and current_chunk:
            chunks.append(' '.join(current_chunk))
            # Keep overlap tokens from end of current chunk
            overlap_words = ' '.join(current_chunk).split()[-overlap:]
            current_chunk = overlap_words + tokens
            current_len = len(current_chunk)
        else:
            current_chunk.extend(tokens)
            current_len += len(tokens)

    if current_chunk:
        chunks.append(' '.join(current_chunk))
    return chunks

Include metadata in your chunk embeddings. Document title, section header, date, and author all improve retrieval accuracy. Embed them alongside the content text, not as a separate field.

Citation Grounding

The most reliable way to prevent hallucination in the LLM response is to make citation mandatory in the prompt, then verify those citations programmatically.

Assign an ID to every chunk you pass in context:

def build_context(chunks: list[tuple[str, dict]]) -> str:
    lines = []
    for i, (text, metadata) in enumerate(chunks):
        source = metadata.get('source', 'Unknown')
        lines.append(f"[{i+1}] SOURCE: {source}\n{text}")
    return "\n\n---\n\n".join(lines)

Then instruct the model to cite by index:

You are a research assistant. Answer the question using ONLY the provided sources.
For every factual claim, cite the source using [1], [2], etc.
If the sources do not contain sufficient information, say "I cannot find this in the provided documents."
Do not guess or infer information not present in the sources.

Post-process the response to verify every citation index exists in your context, and flag responses with uncited claims. This gives you a measurable hallucination rate you can track over time.

Production Monitoring

Track these metrics in your RAG pipeline:

  • Retrieval recall@k: For a labeled evaluation set, what percentage of the time does the correct document appear in the top-k retrieved results?
  • Reranker MRR: Mean Reciprocal Rank after reranking (higher is better)
  • Citation coverage: What percentage of factual claims in LLM responses have a valid citation?
  • User correction rate: If your UI has thumbs up/down, what fraction of responses get corrected?

Build an offline evaluation harness before you go to production. A set of 100–200 hand-labeled query/answer pairs is enough to catch major regressions during development.

The difference between a RAG system that erodes user trust and one that earns it is almost always in the retrieval layer. Get that right, and the LLM will mostly take care of itself.