AI and ML Infrastructure at Scale: Serving Models to Billions of Users

ChatGPT: 200 Million Users, 100ms Per Token

When ChatGPT launched, it reached 100 million users in 2 months - the fastest-growing consumer app in history. Behind every response is a 175-billion parameter model that must run inference in real time, streaming tokens at ~30 per second. Training that model cost $100M in GPU compute. Serving it to 200 million users costs even more. ML infrastructure at scale is an entirely different engineering discipline from traditional web applications.

ML Platform Architecture

ML Infrastructure Stack:

  TRAINING SIDE (offline, batch)        SERVING SIDE (online, real-time)
  ================================     ================================
  [Data Pipeline]                       [Client Request]
  Raw data -> Feature engineering            |
       |                                     v
  [Feature Store]  <-- shared -->       [Feature Store]
  (Feast/Tecton)                        Same transformations as training!
  Ensures training/serving parity            |
       |                                     v
  [Training Cluster]                    [Model Server]
  GPU cluster (A100s/H100s)             (Triton/TorchServe/vLLM)
  Distributed training (FSDP/DeepSpeed) Batched inference (GPU)
       |                                     |
  [Model Registry]                      [A/B Router]
  (MLflow/Weights & Biases)             v1 -> 90% traffic
  Version, metrics, lineage             v2 -> 10% traffic (canary)
       |                                     |
  [Model Artifact]  ---- deploy ---->   [Inference Result]
  (S3/GCS)                              p99 < 100ms target

  Key insight: training and serving MUST use
  identical feature transformations.
  Training-serving skew = silent model degradation.

Inference Optimization

Making Models Fast Enough for Production:

Technique         Speed Gain    Quality Loss    How It Works
-----------       ----------    ------------    ----------------------
Batching          10-50x        None            Accumulate N requests,
                  throughput                    run as single GPU batch

Quantization      2-4x          <1% accuracy    FP32 -> INT8 weights
(INT8/INT4)       faster                        4x smaller model size

Distillation      5-10x         1-3% accuracy   Train small 'student'
                  faster                        to mimic large 'teacher'

Speculative       2-3x          None            Small model drafts tokens,
Decoding          faster                        large model verifies batch

KV Cache          >10x          None            Cache attention keys/values
                  for LLMs                      for previously seen tokens

Model             N GPUs for    None            Split model layers across
Parallelism       1 model                       multiple GPUs (tensor/pipeline)

Real numbers (GPT-3 scale, 175B params):
  Naive serving: 1 request/sec per GPU
  With all optimizations: ~100 requests/sec per GPU
  100x improvement for the same hardware

The Feature Store: Preventing the #1 ML Bug

Training-serving skew is the most common production ML failure. Your model was trained on features computed one way (say, 'average order value over last 30 days' computed from a Spark batch job), but in production the same feature is computed differently (a real-time approximation from Redis). The model's predictions silently degrade because it is seeing inputs it was never trained on. Feature stores (Feast, Tecton, Hopsworks) solve this by computing and storing features once, then serving the same feature values to both training pipelines and production inference. This single abstraction eliminates an entire class of subtle, hard-to-debug production failures.

Model Deployment Patterns

Models are not deployed like regular code. A bad model update can silently degrade quality without throwing errors - worse than a crash because nobody notices. Shadow deployment runs the new model alongside the old one: both process real traffic, but only the old model's predictions are served. You compare accuracy metrics between the two. Once the new model proves equal or better, you shift traffic via canary deployment: 1% to 10% to 50% to 100%, monitoring quality metrics at each stage. If any metric drops, automated rollback reverts to the previous model version. This is why model registries (MLflow, W&B) track every model version with its training data, hyperparameters, and evaluation metrics.

Interview Tip

ML infrastructure interviews test two things: (1) Do you understand the training-serving divide? Training is offline batch GPU work. Serving is online real-time with strict latency SLAs. (2) Do you know how to make inference fast? Batching (GPUs love parallelism), quantization (smaller weights = faster math), and caching (repeated queries skip inference entirely). Always mention feature stores - they prevent the #1 ML production bug: training-serving skew where the model was trained on different features than it sees in production.

Key Takeaway

ML infrastructure splits into training (offline, batch, GPU clusters) and serving (online, real-time, latency-critical). Feature stores bridge the two by ensuring identical feature computation. Optimize inference with batching, quantization, and KV caching. Deploy models with shadow testing and canary rollouts to catch silent quality degradation before it reaches all users.