Vector Databases and Chunking Strategies: The Foundation of Production RAG

RAG Needs a Search Engine. Vector Databases Are That Engine.

You have a million documents. A user asks a question. You need to find the 5 most relevant passages in under 50 milliseconds. Traditional keyword search fails here because users rarely phrase their questions using the exact words in your documents. Someone searching for 'return policy' needs to find passages about 'refund procedures' and 'money-back guarantees.' The words are different but the meaning is the same.

This is where vector databases come in. Instead of storing text as words and searching by keyword matching, vector databases store text as dense numerical vectors (embeddings) and search by semantic similarity. Two passages that mean the same thing will have similar vectors, even if they use completely different words. This is the foundation of every modern RAG system.

How Embeddings Work

An embedding model converts text into a fixed-size vector of floating point numbers. A typical embedding might have 768 or 1536 dimensions. Each dimension captures some aspect of the text's meaning, though the dimensions are not individually interpretable.

The key property: texts with similar meanings produce vectors that are close together in this high-dimensional space. 'The cat sat on the mat' and 'A kitten rested on the rug' will have very similar vectors despite sharing almost no words. 'The cat sat on the mat' and 'Stock prices rose 3% today' will have very different vectors.

Popular embedding models include OpenAI's text-embedding-3 (1536 or 3072 dimensions), Cohere's embed-v3, and open-source options like BGE, E5, and GTE from HuggingFace. The choice of embedding model affects retrieval quality more than most people realize. A better embedding model can improve RAG answer quality by 10-20% without changing anything else in the pipeline.

How Vector Search Works

Once you have embeddings, you need to search through them efficiently. The naive approach of comparing your query vector against every stored vector works fine for 10,000 documents but falls apart at a million. You need specialized data structures.

1. Embed: convert each text passage into a vector using your chosen embedding model
2. Index: store vectors in a specialized data structure optimized for approximate nearest neighbor (ANN) search
3. Query: embed the search query using the same model, then find the K closest vectors using cosine similarity or dot product

The most popular indexing algorithm is HNSW (Hierarchical Navigable Small World graphs). Think of it like a skip list for high-dimensional space. It builds a multi-layer graph where each layer has fewer nodes. Search starts at the top layer (few nodes, big jumps) and drills down to lower layers (more nodes, precise results). HNSW can search 100 million vectors in under 10 milliseconds with 95%+ recall.

The Vector Database Landscape

A new database category emerged almost overnight once RAG became mainstream. Each option has different strengths:

• Pinecone: fully managed, zero operational overhead, scales automatically. Best for teams that do not want to manage infrastructure. Can get expensive at scale.
• Weaviate: open-source with a managed cloud option. Supports hybrid search natively and handles multi-modal data (text, images, audio). Strong community.
• Qdrant: built in Rust for raw performance. Excellent for latency-sensitive applications. Open-source with a cloud option.
• Chroma: developer-friendly, great for prototyping. Easy to embed in Python applications. Less battle-tested for large-scale production.
• pgvector: a PostgreSQL extension that adds vector similarity search. If you already use PostgreSQL, pgvector means no new infrastructure. Sufficient for most applications under 1 million vectors.
• Milvus: designed for billion-scale vector search. GPU-accelerated indexing. Heavier operationally but handles the largest workloads.

For most applications starting out, pgvector is the pragmatic choice. You get vector search without adding a new database to your stack. When you outgrow it (typically above 5 million vectors or when you need sub-10ms latency), migrate to a purpose-built solution.

Chunking: The Most Underrated Decision in RAG

Before you can embed documents, you need to split them into chunks. This sounds simple. It is not. Chunking strategy has more impact on RAG quality than model selection, and most teams spend too little time on it.

The core tension: smaller chunks are more specific (better precision) but lose context (worse recall). Larger chunks preserve context but include irrelevant information that dilutes the signal. Finding the right balance for your specific data and queries is an iterative process.

Chunking Strategies Compared

• Fixed-size chunks (256 or 512 tokens): the simplest approach. Just split at token boundaries. Fast to implement but often cuts sentences in half, breaking the meaning. A sentence split mid-thought produces two chunks that are individually useless.
• Sentence-based splitting: split at sentence boundaries to preserve complete thoughts. Better coherence but produces inconsistent chunk sizes. A short sentence becomes a tiny chunk; a long paragraph becomes a huge one.
• Recursive character splitting: try to split at paragraph boundaries first. If the result is too large, split at sentence boundaries. If still too large, split at word boundaries. LangChain's default approach. Good balance for most text.
• Semantic chunking: use embeddings to detect topic shifts within a document. Start a new chunk when the topic changes. The most accurate approach but also the slowest, since you need to embed sentences during chunking.
• Parent-child chunking: index small chunks (128 tokens) for precise search but return the larger parent chunk (512 tokens) to the LLM for more context. You get the precision of small chunks with the context of large ones.

The sweet spot for most use cases is 256 to 512 tokens with 10-20% overlap between consecutive chunks. The overlap prevents information loss at chunk boundaries. A key detail mentioned at the end of one chunk also appears at the beginning of the next.

Chunking is like slicing a pizza. Too many slices and each piece is too small to satisfy. Too few and each piece is too big to handle. The best size depends on your data and your queries. There is no universal right answer.

Practical Chunking Tips

After working with dozens of RAG systems, certain patterns emerge:

• Start with recursive character splitting at 512 tokens with 50-token overlap. It is a solid baseline that works for most text.
• Test with your actual queries. Run 50 representative questions and check whether the retriever returns the right chunks. If relevant information is consistently split across chunks, your chunks are too small.
• Respect document structure. If your documents have clear sections (headings, chapters), use those boundaries as natural split points.
• Enrich chunks with metadata. Attach the document title, section heading, and page number to each chunk. This metadata is invaluable for filtering and citation.
• Watch out for tables and structured data. Table rows split across chunks become meaningless. Treat tables as atomic units or convert them to text descriptions.

Try It in the Lab

The pipeline lab's Chunker component lets you set chunk_size and overlap. Try different values and observe how they affect your configuration quality score. Start at 512, then experiment with 256 and 1024. The scoring system reflects the real trade-offs you will face when tuning production RAG systems.

Further Reading

• Pinecone's guide to chunking strategies
• LangChain documentation on text splitters
• Johnson et al. (2019): 'Billion-scale similarity search with GPUs' (FAISS paper)
• Malkov and Yashunin (2018): 'Efficient and robust approximate nearest neighbor using HNSW'
• MTEB Benchmark for comparing embedding models