Retrieval-Augmented Generation has become the standard pattern for building LLM applications over private data. But the gap between a working demo and production system is larger than most teams expect. This guide covers the practical decisions that determine RAG success.
RAG Architecture Overview
At its core, RAG has three stages:
Each stage has multiple decision points. Let's walk through them.
Stage 1: Document Ingestion
Text Extraction
Before you can embed, you need clean text. Document type determines approach:
- • PDF (native) — PyMuPDF or pdfplumber for text extraction
- • PDF (scanned) — OCR required (Tesseract, Azure Document Intelligence)
- • DOCX/PPTX — python-docx, python-pptx for structured extraction
- • HTML — BeautifulSoup with boilerplate removal
- • Tables — Specialized extractors (Camelot, Tabula) or vision models
Chunking Strategy
Chunking is where most RAG implementations go wrong. Key considerations:
Chunk Size
256-512 tokens typical. Smaller = more precise retrieval. Larger = more context per chunk.
Overlap
10-20% overlap prevents cutting sentences. Too much = redundant retrieval.
Semantic Chunking
Split on topic boundaries, not just character count. Better coherence, harder to implement.
Hierarchical
Parent-child chunks. Retrieve child, include parent for context. Best of both worlds.
Embedding Models
2026 embedding landscape:
- • OpenAI text-embedding-3-large — Best general performance, 3072 dimensions
- • Cohere embed-v3 — Strong multilingual, compression support
- • Voyage-3 — Domain-specific options (code, legal, finance)
- • Open source — BGE-M3, E5-mistral for self-hosted needs
Tip: Match query and document embedding models. Never mix embeddings from different models in the same index.
Stage 2: Retrieval
Vector Database Selection
Choose based on scale and requirements:
| Database | Best For | Scale |
|---|---|---|
| Pinecone | Managed simplicity | 1M-1B vectors |
| Weaviate | Hybrid search, self-hosted | 10M-100M vectors |
| Qdrant | Performance, filtering | 10M-1B vectors |
| pgvector | PostgreSQL integration | 1M-10M vectors |
| Chroma | Prototyping, local dev | <1M vectors |
Hybrid Search
Pure vector search misses keyword matches. Combine with BM25:
Hybrid search improves recall by 15-30% in our benchmarks, especially for technical queries with specific terms.
Reranking
Vector search retrieves candidates; reranking refines them:
- Retrieve top 20-50 candidates via vector search
- Rerank with cross-encoder model (Cohere Rerank, BGE-reranker)
- Take top 5-10 for generation context
Reranking adds latency (100-200ms) but significantly improves relevance. Worth it for quality-critical applications.
Stage 3: Generation
Context Window Management
Retrieved chunks compete for limited context window space:
- • Reserve space — System prompt, query, and output need room
- • Order matters — Most relevant chunks first (LLMs pay more attention to start)
- • Deduplicate — Remove redundant chunks before stuffing context
- • Summarize — For large contexts, summarize less-relevant chunks
Prompt Engineering for RAG
Key elements of effective RAG prompts:
- • Instruct to only use provided context
- • Request citations/source references
- • Define behavior when context is insufficient
- • Specify output format explicitly
Production Checklist
Before going live:
- ✓ Evaluation pipeline — Automated testing with ground truth Q&A pairs
- ✓ Monitoring — Track retrieval quality, latency, cost per query
- ✓ Feedback loop — Capture user thumbs up/down for continuous improvement
- ✓ Incremental updates — Strategy for adding new documents without full reindex
- ✓ Fallback handling — What happens when retrieval returns nothing relevant?
- ✓ Access control — Filter results based on user permissions
Need help building RAG?
I build production RAG systems tailored to your documents and use case.
Contact me