Building Blocks

System Design From Scratch — Floor 1: What One Machine Can (and Can't) Do

Second post in the series. Floor 0 taught us that some data is far away and slow to reach. This floor asks the question that follows: while you're stuck waiting for something slow, what should you actually be doing?

The premise

Floor 0 ended on a villain: waiting. Some data lives on a slow shelf (the disk, the network), so fetching it takes ages in computer terms. Fine — but that raises the real question, the one this whole floor is about:

While one part of your program is stuck waiting, is the rest of it sitting idle too?

Get this right and a single cheap machine serves thousands of users. Get it wrong and a monster server chokes on a handful. And it all shows up in the kitchen.

Two kinds of work

You're making dinner. Four tasks tonight:

Chopping onions
Boiling a pot of water
Baking garlic bread (15 min)
Whisking a sauce by hand

Sort them into two buckets: tasks that need your hands the whole time, versus tasks you start and then just wait on an appliance.

Chopping and whisking need you non-stop. Walk away and progress stops.
Boiling and baking run themselves. You start them, then you're free — you're just waiting on the stove.

That split has names, and each comes with a dead-simple test: who is the bottleneck — you, or the appliance?

CPU-bound work — you are the bottleneck. Limited by how fast your hands/brain (the "processor") can go. Chopping, whisking, doing math, resizing an image.
I/O-bound work — an external device is the bottleneck and you're just waiting on it. "I/O" = input/output = the processor talking to something outside itself. Waiting on a database, downloading a file, reading a disk.

One distinction — is this limited by my effort, or by something I'm waiting on? — decides how you make it faster. Here's how.

Blocking, and the fix for it

Picture a clueless cook: starts the water, then stands and stares at the pot until it boils, then starts the bread, stares again, and only then picks up a knife. One task at a time, idle through every wait.

That idle staring has a name: blocking. The cook is "blocked" — frozen, zero progress, while waiting on an appliance. (A blocked program does exactly this: stuck waiting on the disk or network, doing nothing useful.)

The fix is what any real cook does on instinct: start the slow stuff, then do other things while it cooks. Start the water and the bread, and chop and whisk during the 15-minute bake. That overlap is concurrency — juggling many tasks in overlapping time by never sitting idle during a wait. (And the moment you put one task down to pick up another? That's a context switch — a real term you probably already felt.)

Watch what it does to the clock. Say chopping = 5 min, whisking = 5 min, water = 10, bread = 15:

Same cook, same kitchen, less than half the time. That's why concurrency is the beating heart of every server: the waits are where all the wasted time hides, and concurrency fills them.

The ceiling: where concurrency can't help

Now take away the waits. Two pure hands-on jobs: chop 5 minutes and whisk 5 minutes, both needing your full attention. No appliance, no waiting to hide inside.

Can one cook finish in 5 minutes by "switching" cleverly? No. Ten minutes is the floor — there's no wait to overlap, and one pair of hands can only do one thing at a time. The only way to 5 minutes is a second cook.

That second cook is parallelism — actually doing multiple things at the same instant. The distinction is one interviewers love:

Concurrency is dealing with many things at once; parallelism is doing many things at once.

The second cook is a CPU core; the whole kitchen is a machine (one machine has several cores).

The takeaway: match the fix to the work

This is the whole floor in two lines:

I/O-bound (lots of waiting)? → concurrency. One cook serves a flood of orders by filling the waits. No extra cores needed.
CPU-bound (pure effort, no waits)? → parallelism. Concurrency does nothing; you need more cores.

A quick gut check to lock it in. A web server spends 90% of its time waiting on the database. To serve far more users, do you add cores or add concurrency?

It's I/O-bound — the cook is standing around waiting on an appliance. Add cores and you just get idle chefs standing around, because effort was never the bottleneck. The answer is concurrency: one cook taking a hundred orders and filling every wait. Cores only help when hands are the limit.

When one kitchen isn't enough

Eventually a real workload maxes out even a smart, fully-parallel kitchen — every core busy, every wait already filled. Then you have two moves:

A bigger kitchen — a beefier machine with more cores. This is vertical scaling.
More kitchens — spread the work across many machines. This is horizontal scaling.

That second door — many machines cooperating — is the entire upstairs of this building, and it's where the really interesting problems live.

What you now own

CPU-bound vs I/O-bound — is the task limited by my effort, or by something I'm waiting on?
blocking — sitting idle during a wait — and context switching
concurrency — one worker filling the waits → the fix for I/O-bound work
parallelism — multiple workers at the same instant → the fix for CPU-bound work (and concurrency ≠ parallelism)
the single-machine ceiling → vertical vs horizontal scaling

Next: Floor 2

We've squeezed everything out of one machine. Floor 2 opens the door to the thing that makes distributed systems both possible and painful: machines talking to each other over a network — why the network is not just slow but unreliable, and where TCP, timeouts and retries, and the choice between HTTP and gRPC come from.

See you on Floor 2.