System Design From Scratch — Floor 4: When One Machine Isn't Enough

Two problems with one machine

Put your whole app's data on one database machine. As you grow, it fails you in two distinct ways:

1. Fault tolerance — it's a single point of failure. Disk dies, power dies, machine dies → your data's gone and your service is down.
2. Scalability — one machine has a hard capacity ceiling (RAM, disk, CPU). Grow past it and you're out of room or crawling.

Different problems — and they have two different fixes. Mixing them up is a classic mistake.

• Replication = keep full copies of the data on multiple machines. Fixes fault tolerance (and read scaling). But every machine still holds everything, so it adds no capacity.
• Partitioning (sharding) = split the data into slices, each machine holding a different one. Fixes capacity (and write scaling). But each slice still lives on one machine, so it's not fault-tolerant.

Each fixes only its own problem. So real systems do both: partition to spread the load, then replicate each partition so no slice has a single point of failure.

Replication's price: consistency

The moment you keep 3 copies and someone changes the data, a new problem appears: the copies can disagree. User updates their email; the leader and one replica have the new value, but a lagging replica still serves the old one. A read that lands there gets stale data. That's consistency — do the copies agree? — and it's Floor 0's cache-staleness problem wearing a replication costume.

• Strong consistency — every read sees the latest write (read the leader, or wait for replicas to sync). Correct, but slower.
• Eventual consistency — reads may be briefly stale, but any replica can answer, fast and scalable. Copies converge "eventually."

The decision rule: the cost of being briefly wrong picks the side. A stale like-count is free; a stale bank balance is a lawsuit.

Partitioning's price: routing and hotspots

Split your data across machines and you now must answer: which machine holds Alice? You route by a partition key — hash(user_id) → a shard — using consistent hashing so that adding a machine moves only ~1/N of the keys, not everything.

The nastier trap is a hot partition (data skew, the "celebrity problem"): if one key gets a huge share of the traffic, its shard melts while the others idle — defeating the whole point. The fix is in the key choice (pick a high-cardinality key that spreads evenly), and for a single hot key, salting (append a suffix to fan it across sub-shards).

The capstone: CAP

Now combine replication (copies on machines A and B) with Floor 2's lesson (the network is unreliable — sometimes machines can't talk). The link between A and B breaks. A write arrives at B, which can't reach A. It has exactly two options, both bad:

• Serve it → Available (AP): the customer gets an answer, but B's data now diverges from A. You gave up Consistency.
• Refuse it → Consistent (CP): the data stays correct, but the customer gets an error. You gave up Availability.

There's no third option — B physically cannot coordinate with A. That's the CAP theorem: when a network Partition happens, you must choose C or A.

• AP fits a shopping cart or feed (Amazon's cart is always writable, and merges conflicts later). Downtime costs more than a little divergence.
• CP fits a bank transfer — better an error than a double-spend.

The nuance most people botch: CAP only forces the choice during a partition. When the network is healthy, you get both. It's "when the network splits, which do you sacrifice."

What you now own

• two problems — fault tolerance + scalability → two moves — replication + partitioning (each fixes only its own → do both)
• replication's price: consistency (strong vs eventual, chosen by the cost of being wrong)
• partitioning's prices: routing (consistent hashing), rebalancing, hot partitions (fix via key choice + salting)
• the capstone: CAP — during a partition, choose C or A (AP = stay up, reconcile; CP = refuse rather than diverge)

Next: Floor 5

We've built the whole stack, bottom to top. Floor 5 is the meta-floor: the everyday patterns engineers assemble from these blocks — load balancers, stateless services, message queues — and the one tension that runs through every single floor.

See you on Floor 5.