RAG and Vector Databases: The Complete System Design Guide

RAG and vector databases help AI applications find relevant information from private data sources. You can ask ChatGPT a question about your company's internal documents.. It confidently gives you an answer that sounds great. One problem: it's completely wrong. The model has never seen your data.

This is the core limitation of every Large Language Model. They only know what they were trained on, and that training data has a cutoff date. They can not access your private databases, your updated documentation, or anything that happened last week.

RAG (Retrieval-Augmented Generation) solves this. Instead of hoping the LLM memorized the right answer, you retrieve the actual documents and hand them to the model alongside the user's question. The model generates its answer grounded in real data, not guesswork.

This guide takes you from zero to production-ready understanding. No AI/ML background needed. If you understand how a database index works, you already have the mental model for 80% of this.

Part 1: What Are LLMs and Why Do They Hallucinate?

Before we fix the problem, let's understand it. A Large Language Model (LLM) is essentially a massive autocomplete engine. It was trained on billions of pages of text from the internet, books, and code repositories. During training, it learned patterns: given these words, what word most likely comes next?

Think of it like a really well-read friend who has read every book in the library. If you ask them about World War 2, they'll give you a solid answer because they read about it. But if you ask them about your company's Q2 revenue report, they'll make something up that sounds plausible, because they never saw that document. That's hallucination.

The Three Limitations of Raw LLMs

• Knowledge cutoff: Training data stops at a fixed date. GPT-4 doesn't know about events from last month.

• No private data: The model has never seen your internal docs, Notion pages, Confluence wikis, or Slack messages.

• No source citations: Even when the model is correct, it can not tell you where that information came from. You just have to trust it.

Interview insight: When an interviewer says 'Design a chatbot that can answer questions about our company's documentation,' they're testing whether you know RAG. A raw LLM is the wrong answer. RAG is the architecture they're looking for.

Part 2: RAG - The Architecture That Fixed Everything

RAG stands for Retrieval-Augmented Generation. The name tells you exactly what it does: before generating an answer, it retrieves relevant information first.

Here's the simplest way to think about it. Imagine you're a customer support agent. A customer asks about the refund policy. You don't try to remember the policy from memory. You open the company handbook, find the refund section, read it, and then explain it to the customer in your own words. That's RAG. The handbook is the retrieval step. Your explanation is the generation step.

The RAG Pipeline Step by Step

There are two phases that happen at completely different times:

Phase 1: Ingestion (Offline)

This happens once per document, not per user query. You're preparing your knowledge base.

1. Collect your documents: PDFs, web pages, Markdown files, database records, whatever your knowledge lives in.

2. Split them into chunks: Break each document into smaller pieces (typically 500-1000 tokens each). Why? Because LLMs have a limited context window, and smaller chunks mean more precise retrieval.

3. Generate embeddings: Run each chunk through an embedding model (like OpenAI's text-embedding-3-small). This converts text into a vector, a list of numbers that captures the meaning of that text.

4. Store in a vector database: Save each (vector, chunk_text, metadata) tuple in a database optimized for similarity search.

Phase 2: Retrieval + Generation (Online)

This happens every time a user asks a question. Speed matters here.

1. Embed the user's query: Convert the question into a vector using the same embedding model.

2. Search the vector database: Find the top-K most similar chunks (typically K=5 to K=20). This takes 1-10 milliseconds.

3. Re-rank (optional but recommended): Use a cross-encoder model to re-score the results. The initial vector search is fast but approximate. Re-ranking is slower but more accurate.

4. Assemble the prompt: Stuff the retrieved chunks into the system prompt alongside the user's question.

5. Generate the answer: Send the assembled prompt to the LLM. It now has the context it needs to answer accurately.

# What the LLM actually sees (simplified)

prompt = f"""
System: You are a helpful support agent.
Answer ONLY based on the provided context.
If the context doesn't contain the answer, say "I don't know."

Context:
---
{chunk_1_text}
---
{chunk_2_text}
---
{chunk_3_text}

User Question: {user_query}
"""

The most common RAG mistake: stuffing too many chunks into the context. More context is not better. Irrelevant chunks dilute the signal and confuse the model. Start with top-5, measure answer quality, then adjust.

Part 3: Embeddings - Turning Words Into Math

This is where most beginners get confused. Let's make it simple.

Computers don't understand words. They understand numbers. An embedding model takes a piece of text and converts it into a list of floating-point numbers (a vector). The clever part: texts with similar meaning end up with similar numbers.

Think of it like GPS coordinates for meaning. "Pizza recipe" and "pasta cooking" are at nearby coordinates because they're about similar topics. "Python code" is far away because it's about a completely different subject. When you search, you're asking: which stored coordinates are closest to my query's coordinates?

Embedding Models You Should Know

• OpenAI text-embedding-3-small: 1536 dimensions, $0.02/1M tokens. The default choice for most projects.

• OpenAI text-embedding-3-large: 3072 dimensions, $0.13/1M tokens. Better accuracy, higher cost.

• Cohere embed-v3: Strong multilingual support. Good if your docs are in multiple languages.

• Open-source (BAAI/bge-large, nomic-embed): Free, run on your own GPU. Great for privacy-sensitive applications.

Similarity Metrics

Once you have two vectors, you need a way to measure how similar they are. The three most common metrics:

# Cosine Similarity (most common for text)
# Measures the angle between two vectors
# Range: -1 (opposite) to 1 (identical)
# Ignores magnitude, focuses on direction
cos_sim = dot(A, B) / (norm(A) * norm(B))

# Euclidean Distance (L2)
# Measures straight-line distance between points
# Lower = more similar
# Sensitive to vector magnitude
l2_dist = sqrt(sum((a - b)^2 for a, b in zip(A, B)))

# Dot Product
# Fast to compute, used when vectors are normalized
# Higher = more similar
dot_prod = sum(a * b for a, b in zip(A, B))

For 99% of RAG applications, use cosine similarity. It's the default in Pinecone, pgvector, and most vector databases. Only switch if you have a specific reason.

Part 4: Vector Databases - Not Just a Fancy Index

Could you store vectors in PostgreSQL and do a brute-force comparison against every single one? Technically yes. With 1,000 vectors, it would work fine. With 10 million? You'd be waiting seconds per query, and your database server would be on fire.

Vector databases solve this with Approximate Nearest Neighbor (ANN) algorithms. The key word is approximate. Instead of checking every single vector (O(N) time), they use clever data structures to find the "close enough" nearest neighbors in O(log N) time. You sacrifice a tiny bit of accuracy (maybe 95-99% recall instead of 100%) but gain orders of magnitude in speed.

HNSW: The Most Popular Index

HNSW (Hierarchical Navigable Small World) works like a skip list for vectors. It builds a multi-layered graph where the top layer has few nodes (long-range connections) and the bottom layer has all nodes (short-range connections). To search, you start at the top and navigate downward, getting closer to your target at each level.

• Time complexity: O(log N) per query

• Space: Needs all vectors in memory (expensive for billions of vectors)

• Accuracy: Very high recall (95-99%)

• Used by: Pinecone, Weaviate, Qdrant, pgvector (default)

IVF: The Partitioned Approach

IVF (Inverted File Index) works like database sharding. It clusters all vectors into groups using k-means clustering. To search, you first find which cluster(s) your query is closest to, then only search within those clusters. The nprobe parameter controls how many clusters to check. Higher nprobe means better recall but slower queries.

When to Use What

• Under 100K vectors: pgvector with HNSW. Keep it simple. Use your existing PostgreSQL.

• 100K to 10M vectors: Pinecone (managed) or Weaviate/Qdrant (self-hosted). Dedicated vector DB worth the complexity.

• 10M+ vectors: Milvus or Pinecone Serverless. Need distributed indexing, sharding, and disk-based storage.

• Privacy requirements: Self-hosted Qdrant or Weaviate. Your vectors never leave your infrastructure.

-- pgvector: Vector search in PostgreSQL (simplest setup)

-- 1. Enable the extension
CREATE EXTENSION vector;

-- 2. Create table with vector column
CREATE TABLE documents (
    id         BIGSERIAL PRIMARY KEY,
    content    TEXT NOT NULL,
    metadata   JSONB DEFAULT '{}',
    embedding  vector(1536)  -- OpenAI ada-002 dimensions
);

-- 3. Create HNSW index for fast search
CREATE INDEX ON documents
    USING hnsw (embedding vector_cosine_ops)
    WITH (m = 16, ef_construction = 200);

-- 4. Query: find 5 most similar documents
SELECT id, content, 1 - (embedding <=> $1) AS similarity
FROM documents
ORDER BY embedding <=> $1  -- <=> is cosine distance
LIMIT 5;

Part 5: Chunking - The Decision That Makes or Breaks Your RAG

Here's a truth most tutorials skip over: the single biggest factor in RAG quality is not which LLM you use or which vector database you pick. It's how you chunk your documents.

Chunk too small and each piece lacks context. The model retrieves fragments that don't make sense alone. Chunk too large and you waste precious context window space on irrelevant text, plus your similarity search becomes less precise because each vector represents too many different ideas.

The Chunking Playbook

1. Start with recursive character splitting (LangChain's RecursiveCharacterTextSplitter). It tries to split on paragraphs first, then sentences, then words. Respects natural boundaries.

2. Set chunk size to 500-1000 tokens. This matches the sweet spot for most embedding models.

3. Add 10-20% overlap between chunks. If a concept spans two chunks, overlap ensures at least one chunk captures the full idea.

4. Attach metadata to every chunk: source document, page number, section heading, date. This enables filtering during retrieval (e.g., 'only search docs from last 30 days').

5. Test with real queries before optimizing. Run 20-30 real user questions through your pipeline and check if the right chunks are being retrieved.

# Python: Recursive chunking with LangChain
from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=800,          # tokens per chunk
    chunk_overlap=150,       # overlap between chunks
    separators=[             # try splitting in this order:
        "\n\n",              # 1. paragraph breaks
        "\n",                # 2. line breaks
        ". ",                # 3. sentence endings
        " ",                 # 4. words (last resort)
    ]
)

chunks = splitter.split_text(document_text)
# Each chunk is self-contained and ~800 tokens

Part 6: Production Patterns That Separate Prototypes From Real Systems

Getting a RAG demo working takes an afternoon. Getting it production-ready takes weeks. Here are the patterns that matter.

1. Hybrid Search (Vector + Keyword)

Pure vector search sometimes misses exact matches. If a user searches for error code 'ERR_CONNECTION_REFUSED', vector search might return docs about general connection issues instead of the exact error code. Hybrid search combines vector similarity with traditional keyword matching (BM25) and merges the results.

# Hybrid search: combine vector + keyword results
vector_results = vector_db.search(query_embedding, top_k=20)
keyword_results = elasticsearch.search(query_text, top_k=20)

# Reciprocal Rank Fusion (RRF) - simple but effective
def rrf_merge(vector_hits, keyword_hits, k=60):
    scores = {}
    for rank, doc in enumerate(vector_hits):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
    for rank, doc in enumerate(keyword_hits):
        scores[doc.id] = scores.get(doc.id, 0) + 1 / (k + rank)
    return sorted(scores.items(), key=lambda x: x[1], reverse=True)

2. Query Transformation

Users ask terrible questions. 'It doesn't work' is not a good search query. Query transformation uses a lightweight LLM call to rewrite the user's question into a better search query before hitting the vector database.

# Query transformation examples

# Original: "it broke after the update"
# Rewritten: "application errors after software update deployment"

# Original: "how much does it cost"
# Rewritten: "pricing plans and subscription costs"

# HyDE (Hypothetical Document Embeddings):
# Generate a hypothetical answer, embed THAT, search with it
hypothetical = llm("Write a short answer to: " + user_query)
hyde_embedding = embed(hypothetical)
results = vector_db.search(hyde_embedding, top_k=5)

3. Metadata Filtering

Not all chunks are equal. A refund policy from 2024 should outrank one from 2020. Metadata filtering lets you narrow the search space before vector comparison happens, making it both faster and more relevant.

4. Guardrails and Safety

• Always include 'If the context doesn't contain the answer, say I don't know' in your system prompt. This prevents the model from falling back to its training data.

• Log every (query, retrieved_chunks, generated_answer) triple. This is your debugging goldmine when answers go wrong.

• Set a confidence threshold on retrieval similarity. If the best match is below 0.7 cosine similarity, don't even try to generate an answer.

• Rate limit LLM calls. Each call costs money. A bug in your frontend could generate thousands of calls in seconds.

Part 7: Designing a RAG System in an Interview

If you get asked 'Design a customer support chatbot' or 'Design a documentation search system' in a system design interview, this is your answer. Here's the architecture an interviewer expects.

Key Talking Points for the Interview

1. Separate ingestion from serving. Ingestion is async, batch-oriented, and can be slow. Serving is synchronous and latency-sensitive. Never let document processing block user queries.

2. Cache aggressively. Embed the same query once (embedding cache). Cache popular question-answer pairs (result cache with TTL). LLM calls are your biggest cost.

3. Circuit breaker on LLM calls. If OpenAI is down, fall back to Claude or a self-hosted model. Never leave users hanging.

4. Multi-tenancy via namespaces. Each customer's data lives in a separate namespace in the vector DB. Metadata filtering ensures tenant isolation.

5. Observability from day one. Track retrieval latency, relevance scores, LLM token usage, cost per query, and user satisfaction (thumbs up/down on answers).

Back-of-Envelope Estimation

# RAG system estimation for interview

# Assumptions:
# - 1M documents, avg 2000 tokens each
# - Chunk size: 800 tokens with 20% overlap
# - 1M docs * ~3 chunks/doc = 3M chunks
# - Embedding: 1536 dimensions (float32)

# Storage:
#   3M vectors * 1536 dims * 4 bytes = ~18 GB (vectors only)
#   + metadata + raw text = ~30-50 GB total
#   Fits in a single Pinecone index or pgvector instance

# Latency per query:
#   Embed query:     5-15ms
#   Vector search:   1-10ms
#   Re-ranking:      50-200ms (optional)
#   LLM generation:  500-3000ms (dominates)
#   Total:           ~600-3200ms (acceptable for chat)

# Cost per query (OpenAI pricing):
#   Embedding:   ~$0.0001 (negligible)
#   LLM (GPT-4o): ~$0.01-0.03 per query
#   At 100K queries/day: $1,000-3,000/day
#   With caching (30% hit rate): $700-2,100/day

Part 8: The Seven Mistakes Everyone Makes (and How to Avoid Them)

1. Using different embedding models for ingestion and queries. If you embed documents with model A and queries with model B, the vectors live in different spaces. Similarity scores become meaningless. Always use the same model for both.

2. Ignoring chunking quality. Most teams spend weeks tuning the LLM prompt and zero time evaluating chunk quality. Run your top 50 user questions through retrieval and check if the right chunks come back. This matters more than prompt engineering.

3. No evaluation framework. How do you know if your RAG is getting better or worse? Build a test set of (question, expected_answer, expected_source_doc) triples. Run it after every change. Measure retrieval recall and answer accuracy.

4. Stuffing too much context. More retrieved chunks is not always better. After about 5-10 chunks, additional context starts hurting answer quality because the model gets confused by irrelevant information.

5. Not handling stale data. If your source docs update, you need to re-chunk and re-embed them. Build an incremental update pipeline, not a full re-index every time.

6. Skipping the re-ranker. The initial vector search uses bi-encoders (fast, but imprecise). A cross-encoder re-ranker is 10x slower but dramatically improves relevance for the final top-K. Worth it for production.

7. No fallback for retrieval failures. If the vector DB returns no results above the similarity threshold, don't send an empty context to the LLM. Return a graceful 'I don't have information about that' response with suggested alternative queries.

Part 9: Where Is RAG Heading?

RAG is evolving fast. Here are the patterns gaining traction in 2025-2026:

• Agentic RAG: Instead of a single retrieve-then-generate step, an AI agent decides what to search for, evaluates the results, and iterates until it has enough context. Think of it as RAG with a reasoning loop.

• GraphRAG: Microsoft's approach that builds a knowledge graph from your documents first, then traverses the graph during retrieval. Better at answering questions that span multiple documents.

• Corrective RAG (CRAG): The system evaluates whether the retrieved documents actually answer the question. If not, it triggers a web search or asks for clarification instead of generating a bad answer.

• Multi-modal RAG: Retrieve not just text, but images, tables, and code snippets. Embed everything into the same vector space.

• Smaller, fine-tuned models: Instead of using GPT-4 for everything, teams are fine-tuning smaller models (Llama 3, Mistral) specifically for their domain. Lower latency, lower cost, and often better accuracy for narrow use cases.

The pattern is clear: RAG is becoming the default architecture for any AI application that needs to work with private or recent data. If you're building anything with LLMs in production, you're building some variant of RAG.

Start Building

RAG is fundamentally a system design problem. Chunking is partitioning. Vector search is indexing. The embedding pipeline is a data pipeline. If you understand distributed systems, you already understand most of RAG.

Practice designing these systems hands-on in ScaleDojo's Architecture Lab. Our interactive levels walk you through building real system designs, from URL shorteners to planet-scale architectures, with AI-powered feedback on every attempt.

Explore the full system design roadmap at scaledojo.dev/roadmap, covering 22 topics from networking fundamentals to AI infrastructure at scale.