Designing a Content Delivery Network

A CDN is one of those systems that sounds trivial until you have to build it. "Put a cache near the user." Done. Then the interviewer asks how it behaves when ten million people want the same video in the same minute, and the trivial answer collapses. This post builds one the way an interview expects: start naive, find the failure, and add only the mechanism that failure demands. Each layer exists because the previous one broke.

Step 0 — Clarify the problem before you design

Before a single box on the whiteboard, nail down what we're actually building. A CDN for static assets (images, JS, video) is a different problem from one for dynamic API responses. Assume the classic case the interviewer means: read-heavy static content, globally distributed users, low latency as the primary goal.

Three numbers to agree on, because every later decision references them:

1. Read:write ratio. Overwhelmingly read-heavy — say 99:1. We optimize for reads; writes (publishing) are rare and can be slower.
2. Object profile. A mix of small (scripts, thumbnails) and large (video), with a heavy tail — a few objects are insanely hot, most are cold.
3. Scale. Say 100M users, tens of thousands of edge locations' worth of geography, content updated occasionally but not continuously.

State these out loud. "I'm assuming read-heavy static content with a power-law popularity distribution — if any of those are wrong, the design changes." Half of system design is refusing to solve the wrong problem.

Step 1 — The naive origin, and why it immediately dies

Start with nothing. One origin server. Users hit it directly.

This fails in three independent ways, and each one motivates a different layer:

• Latency. A user in Singapore fetching from an origin in Virginia pays the round-trip distance on every request. Physics. → motivates geographic caching.
• Origin load. Every single read hits the origin. One origin dies at a few thousand concurrent requests. → motivates caching at all.
• Bandwidth cost. Serving 9 MB across an ocean, per user, per view, is economically unsustainable. → motivates caching close to the user.

All three point the same way: we need a cache. Where do we put it? Close to the user.

Step 2 — The edge cache: first hit at the latency problem

Place cache servers ("edge nodes") in many geographic locations. A user is directed to the nearest one. On a request, the edge checks its local store.

Two things are already true and worth stating, because they drive everything after:

• Hit ratio is the whole game. Every hit is served locally and cheaply; every miss is a slow, expensive trip to the origin. Our success metric is what fraction of requests are hits, and we will spend the rest of the design raising it.
• The hot set is tiny. Because popularity follows a power law, a small number of objects generate most requests. Caching those few well gets you most of the win.

But the moment we deploy this, we discover a new, acute failure.

Step 3 — The cache stampede: why the edge cache kills the origin

A popular object expires (or is evicted). In the same instant, a million users request it. Every edge node has a miss, and every one of them independently fetches from the origin. A single expiry just turned into a million simultaneous origin fetches. The origin is now down. We built a system that DoSes our own origin whenever something goes viral.

This is the cache stampede (or "thundering herd"), and it is the single most important failure mode in CDN design. The naive fix — each edge fetches on miss — is exactly what causes it. We need a layer whose entire job is to collapse concurrent misses into a single origin fetch.

Step 4 — The shield layer: protect the origin

Insert a second cache tier — the shield (or mid-tier / origin shield) — between the edge and the origin. There's typically one shield per region.

The shield's job is not speed; it is origin protection. When N edge nodes miss on the same object at once, the shield lets exactly one request through to the origin and makes the rest wait for that result (request coalescing / "request collapsing"). The origin now sees one fetch, not N.

Remember the three layers and their distinct purposes: edge = latency, shield = origin protection, origin = source of truth. If you can't say which layer solves which problem, you will over-engineer one and starve another.

At this point the architecture holds up structurally. The remaining work is all about making the cache hit more, and hit correctly — which is where the subtle, interview-worthy decisions live.

Step 5 — The cache key: you cache decisions, not bytes

A cache stores objects, but its correctness comes from the key. Get the key wrong and you either miss constantly (low hit ratio) or serve the wrong thing (correctness bug). The key encodes the answer to "is this the same object?"

• Normalize before you key. /img?a=1&b=2 and /img?b=2&a=1 are the same object; key on the sorted query string, or you fragment one object across two keys and halve your hit ratio.
• Vary carefully. Every response can vary by Accept-Encoding, Accept- Language, etc. Each Vary value multiplies stored variants. Vary: Accept-Encoding (gzip/br/identity) is sane; Vary on a free-form header is a memory leak disguised as correctness.
• Know what you do NOT cache. Personalized or auth-gated responses must be excluded from shared edge caches. Pushing them there is how user A's data leaks to user B. The key must make non-cacheable requests unkeyable — bypass, don't store.

The hit-ratio problems that survive Step 4 are almost all key problems. When the interviewer asks "why is hit ratio low," start at the key, not the hardware.

Step 6 — Eviction: LRU is a default, not a strategy

Edge memory is the scarcest resource in the system. You will evict — the only question is how. Least-Recently-Used is the right starting point, but it is blind to two things that matter:

1. Object size. Plain LRU treats a 1 KB script and a 1 MB image as equal "one entry." Evicting a hot image to keep a cold script wastes the budget. Size-aware / cost-weighted eviction treats memory as the finite budget it is.
2. One-shot scans. A crawler walking the catalog once will, under plain LRU, evict your entire hot set to make room for objects that will never be touched again. A tiny admission policy — "don't promote to cache on the first miss, only on the second" — protects the hot set from being scanned to death.

These two refinements are the difference between a cache that survives a background re-crawl and one that falls over during it.

Step 7 — Invalidation: TTL is a guess, purge is a promise

Now the hardest part. Two mechanisms govern freshness, and they fail in opposite ways.

• TTL (time-to-live) is optimistic and lazy: "probably fresh for this long." Cheap, infinitely scalable, and serves stale content during the window. It is how you achieve high hit ratio without a control channel.
• Purge / invalidation is explicit and eager: "this object changed, drop it now." Correct, expensive at global scale, and the source of most incidents.

The first time a purge is "lost" and you serve the wrong price for an hour, you learn that purge semantics deserve as much design effort as the cache itself.

The hard question is what to purge. Object-level purges are precise but expensive (you must invalidate across thousands of edge nodes). Prefix/wildcard purges are cheap but coarse — purge /images/* may evict far more than intended. Most outages live in the gap between "what the publisher thought changed" and "what the key actually matches." A robust design makes purge scoped, idempotent, and verifiable — and instruments the lag between "purge issued" and "last edge node confirmed."

Step 8 — Measure the tail, not the average

The system is built. The last responsibility is to instrument it honestly, because a CDN hides failure behind high hit ratios.

• Hit ratio per layer, separately. Edge hit ratio and shield hit ratio tell different stories. Edge ratio high + origin requests spiking means the shield isn't coalescing.
• Origin request rate. The alarm bell. If it climbs while hit ratio holds, something is leaking through every layer.
• p95 / p99 latency, never mean. One origin fetch at the tail drags a fast median into a slow experience without moving the average. Angry users come from the tail; report the tail.

When not to build one

A CDN is global infrastructure. If your users are in one region and your catalog is small, the edge layer buys latency you can't measure, at the cost of a system you now operate. There, a well-tuned origin cache plus a CDN provider beats a home-grown edge. Build the layers yourself only where scale, cost, or control actually earns the complexity — and even then, build them in the order above, because the order is the argument.