Every AI Product Is a Pipeline. Most People Just Don't Know It Yet.
Open ChatGPT and type a question. You get an answer in seconds. It feels like one thing is happening: you ask, the AI answers. But behind that simple interaction, your question passes through a document loader, a text chunker, an embedding model, a vector database, a retrieval step, a prompt template, an LLM, an output parser, a content filter, and a cost tracker. That is ten components, connected in a specific order, each with configuration choices that affect quality, cost, and latency. That chain of components is called a pipeline. And designing it well is the difference between an AI product that works and one that hallucinates, costs $50,000/month, or takes 30 seconds to respond.
ScaleDojo's GenAI Pipeline Lab lets you design these architectures visually - drag components onto a canvas, wire them together, configure each one, and submit for instant scoring. This guide walks you through everything: what GenAI pipelines are, why architecture matters, and exactly how to use the lab to go from zero to production-ready.
Why GenAI Architecture Matters (More Than Model Selection)
Most conversations about AI focus on which model to use: GPT-4o vs Claude vs Gemini. But model selection is just one decision out of dozens. The architecture around the model - how you feed it data, guard its outputs, cache results, control costs - determines whether your AI product actually works in production.
The $50K/month Mistake
A startup builds a customer support chatbot. They connect user questions directly to GPT-4o with no caching, no semantic deduplication, no cheaper model routing. Every question - even 'what are your hours?' which gets asked 200 times per day - triggers a full GPT-4o call. Their monthly bill hits $50,000. A single SemanticCache component would have cut it to $8,000. Architecture, not model choice, was the problem.
The Hallucination Factory
A legal tech company builds a contract review tool. They send entire 200-page contracts to an LLM in one chunk. The model hallucinates clause numbers, invents legal precedents, and misses critical sections because the context window overflows. The fix is architectural: add a Chunker to split the document, an EmbeddingModel and VectorDB for semantic retrieval, and a Reranker to surface the most relevant chunks. The LLM only sees what it needs. Hallucinations drop by 90%.
The 30-Second Response
An e-commerce company builds a product recommendation chatbot. Every request goes through: document loading, chunking, embedding, vector search, reranking, prompt construction, LLM generation, and output parsing. Total latency: 28 seconds. Users leave after 5 seconds. The fix: add a SemanticCache (80% of queries are repeated), use a faster embedding model, and run the ContentFilter in parallel instead of sequentially. Latency drops to 3 seconds.
The model is the engine. The pipeline is the entire car. You can have a Ferrari engine, but if the transmission is broken, the tires are flat, and there are no brakes, you are not going anywhere safely.
Anatomy of a GenAI Pipeline: The Six Stages
Every GenAI pipeline follows a predictable pattern. Data enters, gets processed, feeds an AI model, and the output gets checked and delivered. Here are the six stages:
Stage 1: Ingestion - Getting Data In
Before an AI can answer questions about your data, it needs to access that data. Ingestion components bring data into the pipeline from various sources.
FileLoader: reads PDFs, Word docs, CSVs, markdown files
WebScraper: pulls content from web pages and sitemaps
APIConnector: fetches data from REST APIs and webhooks
DatabaseReader: queries SQL/NoSQL databases for structured data
The key decision: what data does your AI need access to? A customer support bot needs your FAQ and product docs. A code reviewer needs your codebase. A legal assistant needs contract templates and case law. Start with ingestion.
Stage 2: Processing - Preparing Data for AI
Raw documents are too large for LLMs to process effectively. Processing components break documents into searchable pieces.
Chunker: splits large documents into smaller segments (typically 256-1024 tokens each)
EmbeddingModel: converts text chunks into numerical vectors that capture semantic meaning
VectorDB: stores and indexes these vectors for fast similarity search
Reranker: re-scores retrieved chunks by relevance to improve retrieval quality
This stage is where RAG (Retrieval-Augmented Generation) lives. Instead of cramming an entire document into the LLM's context window, you embed and store chunks, then retrieve only the relevant ones at query time. This reduces cost, improves accuracy, and enables working with datasets that far exceed any model's context limit.
Stage 3: Generation - The AI Core
This is where the actual AI inference happens. An LLM receives context (retrieved chunks + user query) and generates a response.
LLM: the language model itself (GPT-4o, GPT-4o-mini, Claude, Gemini, Llama, etc.)
PromptTemplate: structures the input to the LLM with system instructions, context, and query
OutputParser: extracts structured data from the LLM's text response (JSON, lists, etc.)
Router: directs queries to different LLMs or pipelines based on complexity or topic
Key architecture decision: which LLM model to use. GPT-4o is powerful but expensive ($5/1M input tokens). GPT-4o-mini is 30x cheaper and handles 80% of queries just as well. A Router component can send simple queries to the cheap model and complex ones to the expensive model - cutting costs by 60-70% with minimal quality loss.
Stage 4: Safety and Guardrails
Production AI needs guardrails. Without them, models can generate harmful content, leak PII, or produce outputs that violate compliance requirements.
ContentFilter: blocks toxic, harmful, or off-topic outputs before they reach users
PIIRedactor: strips personally identifiable information (names, emails, SSNs) from inputs and outputs
InputValidator: checks user queries for prompt injection attacks
Guardrails: enforces output format, length, and content constraints
Stage 5: Optimization - Cost and Performance
AI APIs are expensive. Optimization components reduce cost and latency without sacrificing quality.
SemanticCache: caches responses for semantically similar queries (e.g., 'What are your hours?' and 'When are you open?' get the same cached answer)
CostTracker: monitors spend per query, per user, per day - triggers alerts when budgets are exceeded
RateLimiter: prevents API abuse and controls request throughput
LoadBalancer: distributes requests across multiple LLM providers for redundancy and cost optimization
Stage 6: Observability - Watching It All
You cannot improve what you cannot measure. Observability components give you visibility into pipeline health.
Logger: records every request, response, latency, and cost for debugging and analytics
Evaluator: runs automated quality checks on LLM outputs (faithfulness, relevance, coherence)
A/BTestRouter: splits traffic between pipeline variants to test improvements
FeedbackCollector: captures user thumbs-up/down signals to build evaluation datasets
The RAG Pattern - The Foundation of Modern GenAI
RAG stands for Retrieval-Augmented Generation. It is the single most important pattern in GenAI architecture, and it appears in nearly every level of the pipeline lab. Here is how it works:
User asks a question: 'What is our refund policy for enterprise customers?'
The question is converted into a vector (numerical representation) by the EmbeddingModel
The VectorDB searches for document chunks whose vectors are most similar to the question vector
Top-K chunks are retrieved (e.g., the 5 most relevant paragraphs from your policy documents)
A PromptTemplate combines the retrieved chunks with the original question into a structured prompt
The LLM generates an answer grounded in the retrieved context - not from its training data
The OutputParser extracts the answer in the desired format
Why RAG matters: LLMs are trained on public internet data that is months or years old. They do not know your company's policies, products, or internal documentation. RAG bridges this gap by feeding relevant, current information to the model at query time. Without RAG, the model would hallucinate answers based on its training data. With RAG, it answers based on your actual documents.
RAG is like giving the AI an open-book exam instead of asking it to answer from memory. It dramatically reduces hallucination because the answer is right there in the retrieved context.
How the GenAI Pipeline Lab Works
The GenAI Pipeline Lab is a visual pipeline builder with 25 levels across 5 acts. Each level presents a real-world client scenario: a startup needs a customer support bot, a law firm wants a contract reviewer, a healthcare company needs a HIPAA-compliant medical assistant. Your job is to design the pipeline architecture.
Step 1: Read the Case Brief
The left panel shows the client's problem statement, budget per 1,000 requests, latency SLA (maximum response time), and any special constraints. Read this carefully - it tells you exactly what you need to build and what limits you must stay within.
Step 2: Check the Missions
Below the case brief, you will find missions - specific requirements your pipeline must satisfy. These might include 'Include a VectorDB,' 'Add a ContentFilter,' or 'Use chunk_size between 256 and 512.' Missions have live checkmarks that update as you build, so you can see your progress in real time.
Step 3: Add Components
The right panel shows available components grouped by category. Click the + button next to any component to add it to your canvas. Components appear as draggable nodes that you can position anywhere on the canvas.
Step 4: Connect Components
Drag from a node's output handle (right side) to another node's input handle (left side) to create connections. Data flows left to right: ingestion components feed into processing, processing feeds into generation, and generation feeds into safety/output. Invalid connections (e.g., VectorDB to Chunker) are rejected automatically.
Step 5: Configure Each Node
Click any node on the canvas, then switch to the Config tab on the right panel. Each component has configuration options that affect cost, latency, and quality. For example, an LLM node lets you choose the model (gpt-4o vs gpt-4o-mini), temperature (creativity vs precision), and max tokens. A Chunker lets you set chunk_size and overlap. These choices matter - configuration quality is 20% of your score.
Step 6: Watch the Gauges
The top bar shows real-time cost and latency estimates that update as you add and configure components. Green means within budget. Red means over budget. These gauges help you iterate before submitting - you can see the impact of swapping GPT-4o for GPT-4o-mini or adding a SemanticCache in real time.
Step 7: Submit for Evaluation
When you are satisfied with your pipeline, click Submit. Your design is evaluated across 6 scoring dimensions (100 points total) and you receive a letter grade from S to F. Detailed feedback tells you exactly what worked and what to improve.
How Scoring Works: 6 Dimensions, 100 Points
Every submission is evaluated across six dimensions. Understanding these dimensions helps you design better pipelines from the start.
1. Pipeline Completeness (20 points)
Did you satisfy all the missions? If the case brief requires a VectorDB, ContentFilter, and specific chunk sizes, each unsatisfied mission costs you points. This dimension rewards careful reading of the requirements.
2. Configuration Quality (20 points)
Are your components properly configured? An LLM with no model selected, a Chunker with chunk_size set to 10,000 (way too large), or an EmbeddingModel with no model specified will lose points here. The evaluator checks that configurations are reasonable and complete.
3. Architecture Fitness (20 points)
Does your pipeline follow good architectural patterns? This dimension checks for proper connections (data flows logically from ingestion through processing to generation), no orphan nodes (every component is connected), and no impossible connections (e.g., output feeding back into input without a Router).
4. Cost Efficiency (15 points)
Is your estimated cost within the client's budget? The evaluator calculates cost per 1,000 requests based on your component choices and configurations. Using GPT-4o when GPT-4o-mini would suffice, or missing a SemanticCache opportunity, hurts this score.
5. Latency Awareness (15 points)
Is your estimated latency within the client's SLA? Sequential processing of too many components can push latency over limits. Optimization components like SemanticCache and parallelizable paths help keep latency low.
6. Safety and Guardrails (10 points)
Did you include appropriate safety components? Production AI pipelines need content filtering, PII redaction, and input validation. Higher-level scenarios require HIPAA compliance, SOC2 controls, or financial regulatory guardrails. Missing safety components is a common mistake that costs easy points.
The most common mistake is optimizing for one dimension at the expense of others. A pipeline that is cheap but has no safety guardrails will score poorly overall. Balance all six dimensions.
Grade Thresholds and Star Rewards
Your total score maps to a letter grade that determines your star reward:
S grade (95-100 points): 3 stars - exceptional architecture with optimization and safety
A grade (85-94 points): 3 stars - strong pipeline that meets all requirements efficiently
B grade (70-84 points): 2 stars - good pipeline with minor issues in cost or architecture
C grade (55-69 points): 1 star - functional but missing optimization or safety components
D grade (40-54 points): 0 stars - significant gaps in the pipeline design
F grade (0-39 points): 0 stars - fundamental architecture issues or hard gate violations
Hard Gates: Instant F
Some mistakes are so severe they result in an automatic F grade regardless of other scores:
Empty pipeline (no components at all)
Missing an LLM when one is available in the component palette
Using a forbidden connection (e.g., connecting incompatible components)
Exceeding the budget by more than 3x
Exceeding the latency SLA by more than 5x
The 5 Acts: From Foundations to Enterprise
The lab is structured as 5 acts with 5 levels each, progressively introducing more complex architecture patterns:
Act 1: Foundations (Levels 1-5)
Build your first RAG pipelines. Start with a simple document Q&A system and progress to multi-source retrieval. You will learn the core pattern: FileLoader -> Chunker -> EmbeddingModel -> VectorDB -> PromptTemplate -> LLM -> OutputParser. These five levels teach the fundamentals that everything else builds on.
Act 2: Advanced Retrieval (Levels 6-10)
Level up your retrieval game. Add Rerankers for better accuracy, implement hybrid search (combining keyword and semantic search), handle multi-modal data (text + images), and optimize chunk sizes for different document types. These levels teach you that retrieval quality is often more important than model quality.
Act 3: Agents and Multi-Step (Levels 11-15)
Build agent-based systems that can plan, use tools, and chain multiple LLM calls. Design multi-agent architectures where specialized agents collaborate on complex tasks. These levels introduce Routers, tool-calling patterns, and orchestration components.
Act 4: Production Scale (Levels 16-20)
Take your pipelines to production. Add SemanticCache for cost reduction, implement A/B testing for continuous improvement, add monitoring and observability, and handle high-throughput scenarios. These levels teach you that a demo and a production system are very different things.
Act 5: Enterprise and Compliance (Levels 21-25)
Design enterprise-grade AI systems. Handle HIPAA compliance for healthcare, SOC2 requirements for SaaS, multi-tenant isolation, data residency requirements, and audit logging. These levels represent the real-world challenges that enterprise AI teams face daily.
10 Pro Tips for High Scores
Read the case brief twice. The budget, latency SLA, and constraints tell you exactly what is expected. Missing a constraint is the fastest way to lose points.
Start with the core RAG pattern. FileLoader -> Chunker -> EmbeddingModel -> VectorDB -> PromptTemplate -> LLM -> OutputParser. Add safety and optimization on top.
Use GPT-4o-mini by default. Switch to GPT-4o only when the case brief explicitly requires it or the task is genuinely complex (multi-step reasoning, code generation).
Always add a ContentFilter. Safety points are the easiest 10 points to earn. Every production pipeline needs content filtering.
Add SemanticCache for any user-facing application. It cuts cost by 60-80% for common queries and dramatically reduces latency.
Configure every node. Unconfigured components lose you Config Quality points. Set chunk_size, model, temperature, and other params for every component.
Check the cost gauge before submitting. If it is red, swap expensive models for cheaper ones or add caching. Budget overrun is a hard gate.
No orphan nodes. Every component should be connected to at least one other component. Orphans hurt your Architecture Fitness score.
Use hints strategically. The first hint is cheapest (5 credits). Hints are progressive - each one reveals more about what you are missing.
Save frequently. Saving is free. If you are experimenting, save your current state before making big changes so you can go back.
Start Building
The best way to learn GenAI architecture is to build. Open the GenAI Pipeline Lab, start with Act 1 Level 1, and design your first pipeline. You will make mistakes - that is the point. Every failed submission teaches you something about cost, latency, safety, or architecture that you would never learn from reading documentation alone.
By the time you finish Act 1, you will understand RAG better than most engineers who have only read about it. By the time you finish Act 5, you will be designing production-grade GenAI architectures that would pass a senior engineering review. That is not an exaggeration - the levels are designed by engineers who build these systems professionally.
Go build something. The canvas is waiting.