Chapter 02 — Building a RAG Pipeline That Doesn't Fall Over in Production

Every RAG tutorial demos the same happy path: embed some documents, embed a query, cosine-similarity your way to the top-k chunks, stuff them in a prompt, done. That pipeline works great for a 20-document demo and starts misbehaving somewhere around document 2,000 — not because the LLM gets worse, but because retrieval quietly starts feeding it the wrong context.

Chunking is a retrieval decision, not a preprocessing detail

The first failure mode I hit wasn't the model at all — it was that fixed-size chunking (say, 512 tokens with 50-token overlap) routinely split a table, a code block, or the one sentence that actually answered the question, across a chunk boundary. The fix was to chunk along document structure first and only fall back to size-based splitting inside a section that's still too big:

def chunk_document(sections: list[Section], max_tokens: int = 400):
    chunks = []
    for section in sections:
        if token_count(section.text) <= max_tokens:
            chunks.append(Chunk(text=section.text, heading_path=section.path))
            continue
        # only split oversized sections, and keep the heading path attached
        # to every sub-chunk so retrieval still knows what it belongs to
        for piece in split_by_tokens(section.text, max_tokens, overlap=60):
            chunks.append(Chunk(text=piece, heading_path=section.path))
    return chunks

Attaching heading_path to every chunk and prepending it to the embedded text ("Section: Billing > Refunds\n\n<chunk text>") gave the retriever context it otherwise didn't have, and made a visible dent in "technically similar but contextually wrong" retrievals.

Hybrid retrieval, because pure dense search has a blind spot

Dense embeddings are excellent at semantic similarity and bad at exact matches — a product SKU, an error code, an API method name. Those get smeared across embedding space just like everything else, and a pure cosine-similarity search will happily return a plausible-sounding paragraph that doesn't mention the SKU at all. The fix was combining dense retrieval with a sparse lexical pass (BM25) and merging with reciprocal rank fusion:

def hybrid_search(query: str, k: int = 8):
    dense_hits = vector_store.search(embed(query), top_k=20)
    sparse_hits = bm25_index.search(query, top_k=20)
 
    scores: dict[str, float] = {}
    for rank, hit in enumerate(dense_hits):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (60 + rank)
    for rank, hit in enumerate(sparse_hits):
        scores[hit.id] = scores.get(hit.id, 0) + 1 / (60 + rank)
 
    ranked = sorted(scores.items(), key=lambda kv: kv[1], reverse=True)
    return [chunk_store[cid] for cid, _ in ranked[:k]]

RRF is deliberately dumb — it only cares about rank position, not raw score, which sidesteps the problem of dense and sparse scores living on incomparable scales. It's a small function that fixed a class of "why didn't it find the exact term I typed" bug reports overnight.

The failure mode nobody's demo shows: confident retrieval, wrong answer

The scariest failures were the ones where retrieval returned plausible chunks that weren't actually sufficient to answer the question, and the LLM filled the gap with something fluent and wrong. The mitigation that helped most was making the model justify itself against what it was actually given, as a cheap self-check before the answer goes out:

def answer_with_citations(query: str, chunks: list[Chunk]) -> Answer:
    context = "\n\n".join(f"[{i}] {c.text}" for i, c in enumerate(chunks))
    response = llm.generate(
        system=(
            "Answer only from the numbered context below. "
            "Cite the chunk index for every claim. "
            "If the context doesn't contain the answer, say so explicitly."
        ),
        prompt=f"Context:\n{context}\n\nQuestion: {query}",
    )
    # a claim with no citation, or a citation index that doesn't exist,
    # is a strong signal the model is filling gaps rather than reporting
    return validate_citations(response, num_chunks=len(chunks))

Forcing citations doesn't eliminate hallucination, but it turns an invisible failure into a detectable one: an uncited claim or a citation pointing at nothing is cheap to flag and route to a fallback ("I don't have enough information") instead of shipping a fluent guess.

None of these three fixes are exotic — structure-aware chunking, hybrid retrieval, and citation-checked generation are all things you'll find mentioned somewhere in the RAG literature. The lesson was less "which technique" and more that the demo-to-production gap in RAG is almost entirely a retrieval-evaluation gap — you don't find these failures by reading about them, you find them by logging what got retrieved for real queries and actually reading the misses.