System Design From Scratch — Floor 2: How Machines Talk to Each Other

System Design From Scratch — Floor 1: What One Machine Can (and Can’t) Do

Second post in the series. Floor 0 taught us that some data is far away and slow to reach. This floor asks the question that follows: while you’re stuck waiting for something slow, what should you actually be doing?

ProfractNeeraj Dana

The premise

On Floor 0 you learned the network is slow — the "another country" shelf, a 3-year trip in human terms. That's only half the bad news. The other half is nastier, and it's the whole point of this floor:

The network isn't just slow — it's unreliable. And worst of all, when something goes wrong, you often can't tell what.

Let's find that out the way you'd find out anything — with a text message.

The silent phone

It's Saturday. You text a friend: "Dinner at 7?" And then… nothing. No reply.

Where could it have gone wrong? A message makes a round trip — you → network → friend → network → reply back — so it can break in more places than you'd think: your send never leaves, the request is lost mid-network, your friend's phone is dead, or he replied and the reply got lost, or nothing broke at all and he's just in the shower.

Now the killer. Focus on two of those:

From where you sit — a blank screen — you cannot tell Reality A from Reality B. "He never saw it" and "he saw it, acted, and the confirmation got lost" look exactly the same: silence. This is the single most important idea in distributed systems:

Silence proves nothing. No response ≠ failure.

A timeout doesn't mean the other side didn't do the work. It might have done it perfectly and the reply vanished on the way back. Assume "no response = it failed" and you'll make some very expensive mistakes (we'll see one in a second).

Living with an unreliable network

So how do you get anything reliable across a channel that loses things silently? You already know, if you think about the phone:

• You never know a message arrived unless you get a reply back. That reply is an acknowledgment — an ACK.
• You don't wait on silence forever. You give it a reasonable window, then give up — a timeout.
• When the timeout hits, you send again — a retry.

ACK, timeout, retry. But retrying quietly opens a hole. Suppose your message was an action — "book us a table" — and Reality B was true (he got the first one, his reply just got lost). Your retry means he now books two tables. A duplicate.

The fix is one you keep meeting: an idempotency key. Stamp the request with a unique id; the receiver recognizes the repeat and ignores it, so the duplicate is harmless.

That whole recipe — ACK + timeout + retry + idempotency — is about 80% of practical networking. It's also the correct answer to the trap from earlier: "charge the card, got a timeout, so retry." No! The charge might have succeeded — retry blindly and you double-charge. Retry with an idempotency key (and/or check whether it already happened) so the repeat is safe.

Under the hood: packets, IP, and TCP

You don't hand-code ACKs and retries for every message — something does it for you. To see what, you need to know how the network actually moves data: it chops everything into tiny packets and sends each one independently. And the raw packet layer — called IP — is a careless courier: it can lose a packet, deliver them out of order, or even duplicate one. It makes zero promises.

On top of that careless courier sits a smart assistant: TCP. You hand TCP your whole message, and it babysits IP for you:

• it numbers each packet (sequence numbers) so the far end can reorder the jumble,
• the far end ACKs what it got, and anything unacknowledged gets retransmitted (there's your resend),
• a gap in the numbers (…4, 6… — missing 5) is how it detects loss and waits for the fill,
• and a "last" marker (the FIN flag) tells the receiver the message is complete.

The result: a reliable, in-order, complete stream built on top of a network that guarantees none of that. That's the stack clicking together:

But reliability has a cost — meet UDP

All that numbering, ACKing, and waiting-for-#5 takes time. Sometimes that's exactly wrong. On a live video call, a packet carrying video from 2 seconds ago goes missing — do you want the call to freeze and wait while TCP re-fetches a stale frame? Of course not. You want it skipped.

That's UDP: fire-and-forget. No numbering, no ACKs, no resends — whatever arrives gets used. You lose reliability; you gain freshness and low latency.

Notice the shape: TCP buys reliability by paying latency; UDP keeps latency low by giving up reliability. That's the same fast-vs-safe tension that runs through this whole series.

What you actually say: HTTP vs gRPC

TCP gives you a perfect pipe — but a pipe isn't a conversation. Hand someone a flawless phone line and they reply in a language you don't speak: the clear line got you nothing. Both machines still need a shared interface — an agreed way to make a request and shape a response. That agreement is an application protocol, and the one that runs the web is HTTP:

• a request has a verb (GET to fetch, POST to send) and a path (/menu),
• a response has a status code (200 OK, 404 not found, 500 server error) plus data.

Client asks, server answers. It's text-based, human-readable, and universal — every browser and language speaks it.

gRPC is the leaner alternative for internal, machine-to-machine traffic. Instead of shipping bulky text with the field names spelled out every time ({"price": 12}), both sides agree on a contract up front and send compact binary (just 12, because both know field 3 is the price). The trade-off: you give up human-readability and universality (browsers can't speak it natively; outsiders must adopt your contract) to gain speed and small size.

The rule of thumb: public / browser-facing → HTTP+JSON; internal high-volume service-to-service → gRPC.

What you now own

• the network is not just slow but unreliable — and silence is ambiguous (you can't tell a lost request from a lost reply)
• the reliability recipe: ACK + timeout + retry, and retries need idempotency
• packets and the careless IP layer (lose / jumble / duplicate)
• TCP — reliable, ordered, complete (sequence numbers, ACKs, retransmission, a "last" marker) — vs UDP — fire-and-forget, always fresh
• application protocols: HTTP (request/response, verbs, status codes) vs gRPC (binary, shared contract)
• the stack, bottom to top: IP → TCP/UDP → HTTP/gRPC

Next: Floor 3

We can now move data reliably between machines. Floor 3 asks the next question: once data arrives, how do you store it so it survives — a crash, a power cut, a reboot? That's where durability finally gets its real, precise meaning: memory vs disk, and what "it's really saved" actually costs.

See you on Floor 3.