TL;DR: A deep-dive guide to building production RAG systems: chunking strategies, embedding model selection, vector databases, hybrid search, re-ranking, and evaluating retrieval quality separately from generation quality.
Retrieval-Augmented Generation (RAG) is the default architecture for production LLM applications that need to work with private, current, or domain-specific data. The pattern is simple in concept — chunk your documents, embed each chunk, store vectors, retrieve the most relevant chunks at query time, and inject them into the LLM prompt. But every step has traps that silently degrade quality. This guide covers each stage in depth and explains why most RAG failures come from the retrieval step, not the generation step.
RAG is not a database feature. It is a full pipeline, and each stage is a potential failure point. If the right information is not in the retrieved chunks, no LLM prompt engineering can fix it.
The Full RAG Pipeline: End to End. The pipeline has six stages: chunk — split source documents into discrete pieces; embed — convert each chunk into a vector embedding using an embedding model; store — save vectors in a vector database with the original text as metadata; retrieve — at query time, embed the user question and find the nearest chunks by vector similarity; inject — prepend retrieved chunks to the LLM prompt as context; generate — the LLM answers using the provided context plus its own knowledge. Each stage has tuning knobs that interact: chunk size affects embedding quality, embedding model choice affects retrieval accuracy, number of retrieved chunks affects prompt context budget and LLM attention.
Key insight: The retrieval step and the generation step should be evaluated separately. If the right chunk was retrieved but the LLM still answered wrong, the problem is in generation (prompt engineering, model choice). If the right chunk was never retrieved, the problem is in retrieval (chunking, embeddings, search strategy) — and no amount of prompt tweaking will fix it.
Chunking Strategies — The Most Important Decision You Will Make. Chunk boundaries determine what can ever be retrieved. If a fact is split across two chunks, no query will retrieve it whole — the model will see only half the context and may hallucinate the rest. Three approaches:
Fixed-size chunking: Split every N characters/tokens with optional overlap. Simplest to implement and most predictable resource usage. Fails when chunk boundaries cut through sentences, paragraphs, or code blocks. Best for: uniform documents (log files, consistent-formatted records) where semantic boundaries do not matter. Overlap (10-20%) reduces but does not eliminate boundary cuts.
Semantic chunking: Split at natural boundaries: paragraphs, sections, markdown headings, code function boundaries. Requires detecting these boundaries (regex for markdown, AST for code, sentence boundary detection for prose) and produces chunks of variable size. Higher retrieval quality because each chunk is a coherent unit. Best for: documentation, articles, books, codebases — any content with natural structure.
Recursive chunking: Apply semantic chunking first, then recursively split chunks that exceed a maximum token limit using a smaller boundary (sentence, then phrase, then fixed-size). Combines the coherence of semantic with the size guarantees of fixed-size. Best for: mixed-content documents where some sections are short and well-structured while others are long and unstructured.
Rule of thumb: Start with markdown heading + paragraph boundaries (semantic), set max chunk size to 500-1000 tokens, and validate that no single fact is split across chunk boundaries by inspecting a sample of actual chunks.
Choosing an Embedding Model. Embedding quality affects retrieval accuracy more than the LLM you use for generation. A great embedding model with a mediocre LLM outperforms a mediocre embedding model with GPT-4. Key factors: dimensionality (smaller = cheaper storage, faster search, but may lose nuance), domain fit (general-purpose models like text-embedding-3-small work well for most content, but code-specific models like code-search-babbage are better for code), and multilingual support (some models support 100+ languages, others only English). Open-source options (BGE, E5, GTE) can be self-hosted for privacy and cost savings. Cloud options (OpenAI text-embedding-3-, Cohere embed-) trade cost for zero operational overhead. Benchmark on your actual data — embedding leaderboards on public benchmarks do not predict performance on your domain.
Bad practice: Picking the embedding model that ranks highest on the MTEB leaderboard without testing it on your specific documents and queries. Good practice: Build a small eval set of 20-50 query/expected-chunk pairs and compare recall@k across 2-3 candidate embedding models before committing.
Vector Database Options — A Honest Comparison. Four options cover 90% of RAG deployments:
Pinecone: Fully managed, serverless, no ops. Best for: teams that do not want to manage infrastructure, need to scale to millions of vectors quickly, or want built-in hybrid search and re-ranking. Downside: vendor lock-in, cost at scale.
Chroma: Open-source, embedded (runs in your process), simple API. Best for: prototyping, small-scale applications, local development. Downside: not designed for multi-node production deployments or very large datasets.
pgvector (PostgreSQL extension): Add vector search to an existing Postgres database — no new infrastructure. Best for: teams already running Postgres, applications where vectors are alongside relational data, moderate scale (up to millions of vectors). Downside: query performance degrades at very large scale compared to specialized vector databases, fewer built-in features (hybrid search requires additional setup).
Weaviate: Open-source, self-hosted or cloud, with built-in hybrid search and modules for automatic embedding. Best for: teams that want open-source flexibility with managed options, need hybrid search out of the box, or want to avoid managing a separate embedding pipeline. Downside: operational complexity of self-hosting, smaller ecosystem than Pinecone.
Rule of thumb: Start with Chroma for prototyping. Move to pgvector if already on Postgres. Move to Pinecone for production scale with no ops. Consider Weaviate if you need open-source with hybrid search built-in.
Hybrid Search — When Vector-Only Fails. Vector search finds chunks that are semantically similar to the query. But semantic similarity fails on exact-match queries: product SKUs ("SKU-404-BLACK"), error codes ("HTTP 503"), specific names ("Jane Doe"), and numeric identifiers ("order #12345"). These are keyword lookups, not semantic searches, and an embedding will return chunks about "black products" for "SKU-404-BLACK" or chunks about HTTP in general for "HTTP 503." Hybrid search combines vector similarity with keyword search (BM25) and merges results using reciprocal rank fusion (RRF) or a weighted sum. This catches exact-match queries that pure vector search misses while preserving semantic understanding.
Bad practice: "Our vector database handles semantic search, so we do not need keyword search." Reality: Every production RAG system needs hybrid search. Build it in from the start — retrofitting hybrid search after users report that searching for error codes or names returns irrelevant results is harder than including it from day one.
Re-Ranking — Fixing Noisy Top-k Retrieval. The top-k chunks from initial retrieval are noisy — the semantically "closest" chunks are not always the most useful ones for answering the question. Re-ranking passes the top 20-50 retrieved chunks through a cross-encoder model that scores each one against the query more accurately than embedding similarity alone. The re-ranker looks at the actual text content of query and chunk together, not just their vector proximity. This reranks the results so the most useful chunks are at the top, and cuts chunks that are similar in embedding space but irrelevant in content. Re-ranking adds latency (10-100ms per chunk) and cost (requires a second model call), but dramatically improves the final answer quality by ensuring the LLM receives the most relevant context within its limited window.
Key metric: Re-ranking typically improves retrieval precision by 10-20 percentage points at the cost of 200-500ms additional latency. Worth it for any application where answer quality matters more than sub-second response time.
Evaluating Retrieval Quality in Isolation. This is the single most important and most skipped RAG practice. Most teams build a RAG system, test it with 3 demo queries, and ship it — then wonder why users report bad answers. An evaluation set for retrieval is a collection of query/expected-chunk pairs. For each query, check: did the retriever return the expected chunk in the top-k results? If not, diagnose the retrieval stage separately from generation. Build an eval set of 30-100 queries covering: typical user queries, edge cases (very short queries, very long queries, queries with typos), exact-match queries (SKUs, error codes, names), and adversarial queries (queries that try to retrieve the wrong thing). For each query, the expected chunk is the chunk that contains the correct answer. Run your retrieval pipeline against this eval set and measure recall@k (is the right chunk in the top k?) and precision@k (are the top k chunks relevant?). Track these metrics over time — they will degrade silently as your document corpus changes, chunking parameters shift, or embedding models get updated.
Bad practice: "Users will tell us if the answers are wrong." Reality: Users stop using your app and you never know why. Build the eval set before you ship. It takes one day and saves months of firefighting.
Common Failure Modes.
Chunks splitting facts across boundaries: A product description is split mid-sentence by a fixed-size chunker — the first chunk has "The price is $", the second has "49.99 and includes free shipping." No query will retrieve both halves together, and the LLM sees an incomplete sentence. Fix: Use semantic chunking and validate chunk boundaries on a sample of actual documents.
Stale embeddings after document updates: You update a product page with new pricing, but the embedding was generated from the old version and stored in the vector DB. Queries about the new price retrieve the old version's embedding (still semantically similar to "pricing") and the old text in the metadata. Fix: Implement a webhook or scheduled job that re-embeds documents when they change. Track version hashes on embeddings.
Retrieval returns technically-relevant but practically-useless chunks: The user asks "What is the refund policy for damaged items?" and the retriever returns chunks about "general refund policy" (90% match) and "item condition guidelines" (85% match) but not the specific "damaged item refunds" section (75% match). The top chunks are semantically similar but do not answer the question. Fix: Re-ranking catches this — the cross-encoder correctly scores the specific section higher. Without re-ranking, the LLM sees generic policies and gives a generic answer.
Metadata pollution — chunks look alike in embedding space: A knowledge base has 500 chunks, all starting with "According to Section 12.4 of the Employee Handbook..." — the embeddings are nearly identical because the preamble dominates the vector, and retrieval returns essentially random chunks within that section. Fix: Strip repeated boilerplate before embedding, or use a weighted embedding that discounts common prefixes.
Further Reading.
- Pinecone RAG Learning Center
- LangChain RAG From Scratch (GitHub)
- Anthropic Contextual Retrieval Guide
- OpenAI Production RAG Best Practices
Key takeawayA deep-dive guide to building production RAG systems: chunking strategies, embedding model selection, vector databases, hybrid search, re-ranking, and evaluating retrieval quality separately from generation quality.