System Design From Scratch — Floor 5: The Patterns and the One Tension

System Design From Scratch — Floor 4: When One Machine Isn’t Enough

Fifth post in the series. We can store data durably on one machine. But one machine is a single point of failure, and it has a capacity ceiling. This floor goes wide — and it’s the floor the whole distributed world stands on.

ProfractNeeraj Dana

Pattern 1: the load balancer

You've got 10 identical machines now. A user opens your app and fires a request — which machine does it hit? If everyone always hit machine #1, you'd have 9 idle machines and one on fire.

The answer is a load balancer: a component in front of the pool that spreads requests across the machines. Common strategies:

• Round robin — rotate 1, 2, 3, … Simple, even, but ignores how busy each machine is.
• Least connections — send to whichever machine has the fewest requests in flight.
• Hash by user / session (sticky sessions) — the same user always lands on the same machine.

Pattern 2: stateless services

That last option — sticky sessions — hides a trap. If machine #4 keeps Alice's login and cart in its own memory, and #4 dies, Alice is logged out with an empty cart. Routing her to #7 doesn't help — #7 never saw her.

The fix is one of the biggest patterns in the field: stateless services. The app machines hold no per-user state; it all lives in a shared external store every machine can reach.

Now any machine can serve any user, the load balancer can be dumb, a dead machine loses nothing, and you can add/remove machines at will. The deep insight: you didn't eliminate the hard state problem — you moved it off the disposable compute machines into a dedicated data tier, where you have all of Floors 3–4's tools (durability, replication, partitioning, consistency) to handle it. Stateless compute in front, stateful data behind — that's the skeleton of nearly every scalable system.

Pattern 3: asynchronous work (message queues)

So far everything is synchronous: request in → do the work → respond, user waiting the whole time. But some work is slow. A user uploads a video that takes 45 seconds to transcode — you can't make them stare at a spinner (and tie up a machine) for 45 seconds.

Instead: drop the job on a message queue, ack the user instantly ("got it, processing"), and a background worker does the heavy work later. This buys three things:

1. Responsiveness — the user isn't blocked.
2. Load smoothing — a burst of 10,000 uploads queues up instead of melting you; workers drain at their own pace.
3. Resilience — a worker crash just means the message gets retried (which needs idempotency, from Floor 2).

The cost: the result is eventual (notify or poll), and you inherit queue concerns (at-least-once delivery, ordering). Not "fast vs safe" — this one is synchronous (answer now, fragile under load) vs asynchronous (instant ack, robust, but eventual).

The one tension

Step back across all six floors, and here's the payoff. You didn't learn a pile of unrelated facts. You learned one master tension — fast vs. safe (speed vs. correctness) — wearing a different costume on every floor:

Every design decision is one of these knobs. And the meta-skill — the thing that makes someone sound like a principal engineer — is this: you pick a side based on the cost of being wrong for that specific data. A stale like-count is free. A stale bank balance is a lawsuit. Same knob, opposite choice.

That's it. That's the whole game. Not memorizing answers — recognizing which knob you're turning and choosing deliberately.

The building, complete

• Floor 0 — why anything is fast or slow (latency, the memory hierarchy, caching)
• Floor 1 — what one machine can and can't do (CPU/IO-bound, concurrency, parallelism)
• Floor 2 — how machines talk (unreliable networks, TCP/UDP, HTTP/gRPC)
• Floor 3 — storing data so it survives (durability, fsync, WAL)
• Floor 4 — when one machine isn't enough (replication, partitioning, CAP)
• Floor 5 — the patterns (load balancers, stateless services, queues) and the one tension

Six floors, and every term hangs off a physical reason, every trade-off is one tension in disguise. That's a foundation you can build on — which is exactly what comes next.

Thanks for climbing the whole building with me.

System Design From Scratch — Floor 0: Why Anything Is Fast or Slow

The first post in a series where I build system-design intuition from the ground up. No jargon dumped on you up front — we start with an everyday situation, reason it out, and only then attach the technical name.

ProfractNeeraj Dana

System Design From Scratch — Floor 1: What One Machine Can (and Can’t) Do

Second post in the series. Floor 0 taught us that some data is far away and slow to reach. This floor asks the question that follows: while you’re stuck waiting for something slow, what should you actually be doing?

ProfractNeeraj Dana

System Design From Scratch — Floor 2: How Machines Talk to Each Other

Third post in the series. Floor 0 taught us the network is the slow shelf. Floor 1 squeezed everything out of one machine. This floor is where single-machine thinking ends and distributed thinking begins — and it’s where a lot of engineers quietly get things wrong.

ProfractNeeraj Dana

System Design From Scratch — Floor 3: How Data Survives a Crash

Fourth post in the series. We can move data reliably between machines (Floor 2). Now: once data arrives, how do you store it so it survives a crash, a power cut, a reboot? This is where the word durability finally gets a precise, physical meaning.

ProfractNeeraj Dana

System Design From Scratch — Floor 4: When One Machine Isn’t Enough

Fifth post in the series. We can store data durably on one machine. But one machine is a single point of failure, and it has a capacity ceiling. This floor goes wide — and it’s the floor the whole distributed world stands on.

ProfractNeeraj Dana