The story of two startups, and the one everyone forgets
Startup A launched on a single powerful server. When traffic grew 10x, they bought a bigger box — three hours of downtime to migrate. When traffic grew 100x, there was no bigger box left to buy, and they spent months mid-growth rewriting for horizontal scaling under pressure. Startup B launched on three small servers behind a load balancer. Traffic grew 10x, they added seven more servers with zero downtime. Traffic grew 100x, they auto-scaled to 90 more. No rewrite required.
That story is the standard system-design-interview parable, and it's a genuinely useful one — but it's usually told as if horizontal scaling is simply the correct answer and vertical scaling is a beginner's mistake. That's not quite true, and the clearest counter-example is a company most engineers have used directly: Stack Overflow ran its entire Q&A platform — 209 million HTTP requests a day, 5.8 billion Redis operations a day — on roughly 9 to 11 web servers for well over a decade, deliberately choosing vertical scaling and a monolith while much of the industry moved toward microservices and horizontal-first designs. It worked because of how they scaled vertically, not despite it: aggressive caching, SQL Server boxes with 384GB of RAM and fast NVMe storage, and a monolith that avoided the network-hop latency a horizontally-distributed, service-per-team architecture would have introduced. The lesson isn't "horizontal always wins" — it's that the right choice depends on your read/write pattern, your team size, and how much of that scale is genuinely necessary versus assumed in advance.
Vertical scaling: make the machine bigger
Vertical Scaling (Scale Up):
Before: After:
+------------------+ +------------------------+
| 4 CPUs | | 64 CPUs |
| 16 GB RAM | --> | 512 GB RAM |
| 500 GB SSD | | 4 TB NVMe |
| 1 Gbps network | | 25 Gbps network |
+------------------+ +------------------------+
Cost: $200/month Cost: $3,000/month (15x!)
Handles: 500 req/sec Handles: 3,000 req/sec (6x)
Key insight: 15x the cost for only 6x the performance.
Non-linear cost scaling is the fundamental problem.
The performance-per-dollar drop-off isn't an accident of pricing — it reflects real physical limits. Bigger machines hit diminishing returns from memory bus contention, NUMA (non-uniform memory access) effects across CPU sockets, and cooling and power costs that scale worse than linearly as chassis get denser. Cloud providers also price the largest instance tiers at a premium specifically because so few workloads genuinely need them, and that pricing curve directly mirrors the physical one.
Advantages: zero code changes needed — the application runs identically, just on faster hardware. No distributed systems complexity: no network partitions, no split-brain scenarios, no eventual-consistency trade-offs to reason about. Strong consistency is trivial because everything lives on one machine.
Disadvantages: there's a real ceiling. AWS's largest memory-optimized instances — the U7i family — currently top out around 32 TiB of RAM and up to 896 vCPUs on the largest configuration; that ceiling has moved up over the years and will keep moving, but it is still a ceiling, and a workload that genuinely outgrows it has no next vertical step. A single machine is also a single point of failure — if it dies, the service is fully down, not degraded. And vertical migrations to a bigger box typically require downtime: shut down, migrate, restart.
Horizontal scaling: add more machines
Horizontal Scaling (Scale Out):
Before: After:
+---------+ +-----+ +-----+ +-----+
| Server | | S1 | | S2 | | S3 |
| 8 CPUs | --> | 8 | | 8 | | 8 |
| 32 GB | | 32 | | 32 | | 32 |
+---------+ +-----+ +-----+ +-----+
Cost: $400/month Cost: $1,200/month (3x)
Handles: 1K req/sec Handles: 3K req/sec (3x)
Redundancy: lose 1 server,
still serve 2K req/sec
Advantages: effectively unlimited scaling — add machines as demand grows, with no theoretical ceiling like a single box has. Built-in redundancy: lose one node and the others absorb the load with zero downtime, rather than a total outage. Cost-effective at scale, since commodity hardware priced near-linearly beats the premium pricing curve of the largest single-box tiers.
Disadvantages: requires a load balancer to distribute requests, which is itself a component that needs to be highly available. State management gets genuinely harder — sessions and caches that lived comfortably in one process's memory now have to be externalized somewhere every node can reach. And every request that now crosses a network boundary between machines picks up real latency and a new set of failure modes (timeouts, partial failures, retries) that simply don't exist inside one process.
What actually scales which way
Component Scale Vertically Scale Horizontally
---------------------- -------------------- ----------------------
Stateless API servers Start here Switch early (easy)
Background workers Start here Switch early (easy)
PostgreSQL/MySQL Start here (long!) Shard later (hard!)
Redis cache Start here Cluster later (medium)
Elasticsearch N/A Horizontal from day 1
Kafka N/A Horizontal from day 1
File storage N/A Object storage (S3)
The dividing line is state. Stateless services — anything where any instance can handle any request because it holds nothing in memory between requests — are easy to scale horizontally because adding a node is just adding capacity, with no data to reconcile. Stateful services, above all databases, are hard to scale horizontally because the data itself has to be split, replicated, or coordinated across nodes, and that's a fundamentally harder problem than routing more HTTP requests to more identical processes. The practical rule: make everything stateless except the database, and treat the database as the one component where vertical scaling buys you the most time before you're forced into real distributed-systems complexity.
The database scaling ladder: vertical, then replicas, then sharding
This is the sequence most system design answers skip past, and it's the part that actually matters once you're past the "add more app servers" stage. When a single database instance starts to strain, the typical order of escalation is:
Vertical scaling first. A bigger database box, more RAM for caching the working set, faster storage. This buys real headroom with zero application changes and is almost always the right first move.
Read replicas next. Replication duplicates the dataset onto additional nodes that serve read traffic while all writes still go to a single primary. This is usually the easiest horizontal step for a database, and a strong fit when reads — dashboards, search pages, reporting queries — are what's overloading the primary. The trade-off: replica lag. A replica is asynchronously catching up to the primary, so a read immediately following a write can return stale data unless you specifically route read-your-own-writes traffic back to the primary.
Sharding last, and only if writes demand it. Sharding partitions the dataset itself across multiple independent database nodes, each owning a subset of the data by some shard key. This is what actually scales write throughput and total data size, because unlike replication, it's not duplicating the same data everywhere — but it comes at a real complexity cost: queries that don't include the shard key become scatter-gather operations across every shard, cross-shard joins and transactions get genuinely hard, and resharding later (because your shard key assumption turned out wrong) is one of the more painful migrations in distributed systems.
The rule of thumb worth stating plainly: replication solves read load and availability; sharding solves write load and dataset size. They solve different problems, and reaching for sharding to fix a read-heavy bottleneck is a common, costly mistake — read replicas would have been the far simpler fix. In large systems the two are frequently combined: each shard gets its own set of read replicas, giving both write scalability and read distribution at once.
Designing for horizontal from day one
Even if you scale the database vertically for a long time — as Stack Overflow did — the application tier is cheap to make horizontally-ready from the start, and expensive to retrofit under pressure later:
Make services stateless. No in-memory sessions, no local file state. Store sessions in Redis, files in object storage like S3, so any instance can serve any request.
Externalize state. The database, the cache, and the message queue should be the only stateful components in the system — everything else should be freely replaceable and restartable.
Use environment/config-based configuration, not anything baked into a specific instance, so any server can run any request without special setup.
Containerize services so deployments are identical regardless of which machine or how many machines they run on.
Put a load balancer in front from the start, even with a single server behind it. Adding a second server later becomes a config change instead of an architecture change.
Common mistakes
Treating horizontal as automatically superior. Stack Overflow's decade-plus run on a handful of well-specced servers is proof that vertical-first, read-heavy, cache-aggressively architectures can outperform premature distribution — both in raw performance and in engineering velocity, since a monolith avoids the network-hop tax a service-per-team architecture pays on every internal call.
Sharding to fix a read problem. If read traffic is the bottleneck, read replicas are almost always the right first step; sharding solves write throughput and dataset size, and reaching for it prematurely adds cross-shard query complexity you didn't need to take on yet.
Ignoring replica lag. An application that reads its own just-written data from a replica without accounting for lag will intermittently see stale results — a subtle bug that's easy to miss in testing and painful in production.
Leaving state in the app tier "for now." In-memory sessions or local file caches are the single most common thing that blocks horizontal scaling later, and they're also the cheapest thing to avoid from the start — externalizing them after the fact, once dozens of code paths assume local state, is a much larger project than doing it upfront.
Assuming the vertical ceiling is fixed. Cloud providers keep raising the top end of vertical scaling (AWS's largest memory-optimized instances now reach well beyond what was available even a couple of years ago), so "we'll hit a wall eventually" is true, but the wall is further away than most rough mental models assume.
Interview framing
A strong answer sounds like: "I'd start with vertical scaling for the database, because distributed transactions and cross-shard queries add real complexity I don't want to take on before I need to — and there's a lot of headroom in modern hardware before that becomes necessary. But I'd design the application tier to be stateless from day one: no local sessions, no local file state, everything externalized to Redis or object storage, sitting behind a load balancer even with just one instance running. That way, scaling the app tier horizontally later is just adding servers, not a rewrite. When the database itself becomes the bottleneck, I'd add read replicas first, since that's the easiest horizontal step and solves read-heavy load without touching my data model — and I'd only move to sharding if write volume specifically demands it, since that's the point where the complexity trade-off is actually worth paying." That sequencing — stateless app tier immediately, vertical database scaling first, replicas before sharding — is what separates a memorized "always go horizontal" answer from one that shows real judgment about when complexity is worth it.
Key takeaway
Vertical scaling is simple — no code changes, no distributed systems complexity — but it has a real ceiling and creates a single point of failure; it's also, as Stack Overflow's decade on a handful of servers shows, a legitimate long-term strategy for read-heavy, well-cached workloads, not just a stopgap. Horizontal scaling is close to unlimited and adds built-in redundancy, at the cost of load balancing, externalized state, and network-related failure modes. The practical path: make the application tier stateless from day one so horizontal scaling is always available as an option, scale the database vertically until it strains, add read replicas before reaching for sharding, and treat sharding as the tool for write-volume and dataset-size problems specifically — not a default step in every scaling story.
FAQ
Is horizontal scaling always better than vertical? No. It's better for unlimited growth and fault tolerance, but it adds real complexity (state externalization, network latency, distributed failure modes) that isn't worth paying for every workload. Stack Overflow's read-heavy, aggressively-cached monolith on a handful of powerful servers is a well-documented example of vertical scaling outperforming a more distributed alternative for over a decade.
What's the actual ceiling on vertical scaling today? It keeps moving upward. AWS's current largest memory-optimized instances (the U7i family) reach up to roughly 32 TiB of RAM and hundreds of vCPUs on the biggest configurations — far beyond what most workloads will ever need, though a true ceiling does still exist.
Should I add read replicas or shard my database first? Read replicas first, almost always. They're simpler to introduce, solve read-heavy bottlenecks directly, and don't require redesigning how your data is partitioned. Reach for sharding only once write throughput or total dataset size — not read load — is the actual constraint.
What does "stateless" really mean for an application server? It means no request depends on data stored only in that specific process's memory or local disk — sessions, uploaded files, and cached computed results all live in a shared external store (Redis, a database, object storage) that every instance can reach, so any instance can handle any request interchangeably.
Can you combine vertical and horizontal scaling? Yes, and most real systems do. A common pattern is scaling each individual node vertically to a reasonable, cost-effective size, then scaling horizontally by adding more of those right-sized nodes — rather than either scaling one machine to its absolute limit or running a large fleet of underpowered ones.