Content Delivery Networks

The name layer chose which zone the request would go to, but what it chose was an address — the request still has not reached the application once it arrives there. In that gap sits a stop that can answer the request without letting it reach the application. Caching was built in the Caching, Queues and Asynchronous Processing course, where the cache sat next to the application, as the Introduction to System Design course’s arithmetic assumed too: a 90% hit rate brought reads behind cache/s down to 41.67.

This lesson relocates the same cache; strategy, invalidation, and key design are not repeated, since the only thing that changes is where the cache sits. A content delivery network keeps the same content at many points close to the user, and each such point is an edge cache. The question: at the same hit rate, next to the user instead of next to the application, which number changes and which does not.

What the Location Decision Changes

A cache next to the application versus next to the user separates three things at the same hit rate. First, a hit never reaches the application — it drops out of the load application processes see. Second, a hit’s response does not use the link between origin and edge, so bytes crossing the boundary fall. Third, a miss now passes through three stops instead of two.

None of this grows the hit rate on its own. What this lesson measures is that location does not change the rate — it changes what the rate is worth.

Bytes on the Wire

The first setup is two local processes: origin and edge, measuring how many bytes the same body takes up on a real connection. The introductory course’s calculation left this out, saying header and protocol overhead would be a separate assumption; here a measurement replaces it.

// edge/server.mjs — two local processes: the origin listens on 8471, the edge cache on 8472.
// Cache capacities are three keys to keep the example small; both expose their counters at /counter.
import http from "node:http";

const BODY = 480;                          // K01 V5: tracking response body
const body = (num) => {
  const head = JSON.stringify({ tracking: num, state: "in-transit", zone: "b34" });
  return head + " ".repeat(BODY - head.length);
};
const newCache = (capacity) => ({
  capacity, m: new Map(),
  get(num) { const b = this.m.get(num); if (b !== undefined) this.m.delete(num); return b; },
  put(num, b) {
    if (this.m.size >= this.capacity) this.m.delete(this.m.keys().next().value);
    this.m.set(num, b);
  },
});
const counterRoute = (s) => (request, response) => {
  response.writeHead(200, { "content-type": "application/json" });
  response.end(JSON.stringify(s));
};

const originStats = { requests: 0, hits: 0, stored: 0 };
const originCache = newCache(3);
http.createServer((request, response) => {
  if (request.url === "/counter") return counterRoute(originStats)(request, response);
  const num = request.url.slice("/tracking/".length);
  originStats.requests += 1;
  let b = originCache.get(num);
  if (b === undefined) { originStats.stored += 1; b = body(num); } else originStats.hits += 1;
  originCache.put(num, b);
  response.writeHead(200, { "content-type": "application/json", "content-length": BODY });
  response.end(b);
}).listen(8471);

const edgeStats = { requests: 0, hits: 0, misses: 0, originRequests: 0, originBytes: 0 };
const edgeCache = newCache(3);
http.createServer(async (request, response) => {
  if (request.url === "/counter") return counterRoute(edgeStats)(request, response);
  const num = request.url.slice("/tracking/".length);
  edgeStats.requests += 1;
  let b = edgeCache.get(num), status = "hit";
  if (b === undefined) {
    status = "miss";
    edgeStats.misses += 1; edgeStats.originRequests += 1;
    b = await (await fetch(`http://127.0.0.1:8471/tracking/${num}`)).text();
    edgeStats.originBytes += Buffer.byteLength(b);
  } else edgeStats.hits += 1;
  edgeCache.put(num, b);
  response.writeHead(200, { "content-type": "application/json",
    "content-length": Buffer.byteLength(b), "x-edge": status });
  response.end(b);
}).listen(8472);

# edge-measure.sh — first one direct request to the origin, then the same two tracking numbers
# queried three times each through the edge. Header bytes depend on the header set the server
# produces; the ratio between body and wire is not tied to that.
node edge/server.mjs & SERVER=$!
until curl -sf http://127.0.0.1:8472/counter >/dev/null; do sleep 0.2; done
measure() { curl -s -o /dev/null -D - "$1" -w '%{size_download} %{size_header}\n' 2>/dev/null; }

printf '%-18s %-7s %6s %7s %5s\n' request edge body header wire
measure http://127.0.0.1:8471/tracking/TR900010 |
  awk 'END{printf "%-18s %-7s %6d %7d %5d\n", "origin direct", "-", $1, $2, $1+$2}'
for round in 1 2 3; do
  for num in TR900011 TR900012; do
    measure "http://127.0.0.1:8472/tracking/$num" |
      awk -v n="$num" -v t="$round" '/^x-edge/{k=$2; sub(/\r/, "", k)}
        END{printf "%-18s %-7s %6d %7d %5d\n", "edge " n " r" t, k, $1, $2, $1+$2}'
  done
done
printf '\nedge counter = %s\n' "$(curl -s http://127.0.0.1:8472/counter)"
printf 'origin counter = %s\n' "$(curl -s http://127.0.0.1:8471/counter)"
kill $SERVER

request            edge      body  header  wire
origin direct      -          480     156   636
edge TR900011 r1   miss       480     170   650
edge TR900012 r1   miss       480     170   650
edge TR900011 r2   hit        480     169   649
edge TR900012 r2   hit        480     169   649
edge TR900011 r3   hit        480     169   649
edge TR900012 r3   hit        480     169   649

edge counter = {"requests":6,"hits":4,"misses":2,"originRequests":2,"originBytes":960}
origin counter = {"requests":3,"hits":0,"stored":3}

These numbers belong to the measurement class. Header bytes depend on the server’s header set and differ in another implementation; the ratio between body and wire, 649 / 480 = 1.352, carries forward as a lower bound, since the header set can only grow.

The counters give the location decision’s first result: the edge saw six requests, two misses, four hits. Only two requests reached the origin — the second and third query for the same two tracking numbers never got there. The origin’s own counter shows three, because the direct measurement counted too, and its hit counter is zero: requests reaching it are so sparse that its own cache never helped. A small preview of the result the second half of the lesson measures.

Where the Hit Rate Comes From

The second setup is an in-process model, labeled as one: the same request trace runs first with a single cache, then with an edge cache placed in front of it. The trace is this course’s assumption; the rates and counts are measurements.

The model’s input derives from the introductory course: 12,000,000 daily tracking queries, 400,000 new shipments a day. One assumption is added — T6: a shipment stays in transit for 3 days — giving 1,200,000 active shipments; daily queries per shipment is 12,000,000 / 1,200,000 = 10, a computed value.

Ten queries are spread across a shipment’s three-day life. A cache entry cannot live three days, so what sets the rate is how many of the ten land in the same short window: a user opens the tracking page and refreshes it a few times. This is the second assumption — T7: burst length, how many of the ten queries share a burst. The third is T8: 100 bursts open at once; the fourth is T9: each cache holds 1,000 keys.

// edge/trace.mjs — in-process model: the same request trace runs first with a single cache,
// then with an edge cache placed in front of it. The trace is an assumption (T7); ratios and counts are measurements.
const REQUESTS = 200_000;    // K01 peak read 416.67 requests/s x 480 s
const QUERIES = 10;          // computed value: V1*V2 / (V3*T6) = 12,000,000 / 1,200,000
const OPEN = 100;            // T8: tracking pages open at the same time (burst)
const CAPACITY = 1_000;      // T9: each cache's key capacity

function* trace(burst) {     // burst: how many of a shipment's 10 queries land in the same burst
  const open = [];
  let fresh = 1, single = 1, index = 0;
  for (let i = 0; i < REQUESTS; i++) {
    if (i % QUERIES >= burst) { yield `D${single++}`; continue; }  // key seen once in the window
    while (open.length < OPEN) open.push({ id: `G${fresh++}`, remaining: burst });
    index = (index + 1) % open.length;
    const o = open[index];
    yield o.id;
    if (--o.remaining === 0) open.splice(index, 1);
  }
}

class Cache {                // most-recently-used sits last; the Map's insertion order is enough
  constructor(k) { this.k = k; this.m = new Map(); this.hits = 0; this.misses = 0; }
  probe(a) {
    if (this.m.has(a)) { this.hits += 1; this.m.delete(a); this.m.set(a, 1); return true; }
    this.misses += 1;
    if (this.m.size >= this.k) this.m.delete(this.m.keys().next().value);
    this.m.set(a, 1);
    return false;
  }
  get rate() { return this.hits / (this.hits + this.misses); }
}

export function run(burst) {
  const single = new Cache(CAPACITY);
  const edge = new Cache(CAPACITY), app = new Cache(CAPACITY);
  for (const a of trace(burst)) {
    single.probe(a);
    if (!edge.probe(a)) app.probe(a);
  }
  return { requests: REQUESTS, singleRate: single.rate, singleStored: single.misses,
    edgeRate: edge.rate, appRequests: edge.misses,
    appRate: app.rate, edgeStored: app.misses };
}

if (process.argv[1].endsWith("trace.mjs")) {    // does not print when imported, only exposes run
  console.log(`${REQUESTS} requests, ${QUERIES} queries per shipment, ${OPEN} open bursts, ` +
    `cache capacity ${CAPACITY} keys`);
  console.log();
  console.log("burst | single cache | edge hit | reaching app | app hit | reaching store");
  console.log("------|--------------|----------|---------------|---------|----------------");
  for (const burst of [10, 5, 2]) {
    const r = run(burst);
    console.log(`${String(burst).padStart(5)} | ${r.singleRate.toFixed(4).padStart(12)} | ` +
      `${r.edgeRate.toFixed(4).padStart(8)} | ${String(r.appRequests).padStart(13)} | ` +
      `${r.appRate.toFixed(4).padStart(7)} | ${String(r.edgeStored).padStart(14)}`);
  }
  const r = run(10);
  console.log();
  console.log(`burst 10: requests reaching store without edge ${r.singleStored}, with edge ${r.edgeStored} ` +
    `(x${(r.edgeStored / r.singleStored).toFixed(3)})`);
  console.log(`requests reaching app ${r.requests} -> ${r.appRequests} ` +
    `(x${(r.appRequests / r.requests).toFixed(3)})`);
  console.log(`measured edge hit rate ${r.edgeRate.toFixed(4)}, V9 assumption 0.90 -> difference ` +
    `${(Math.abs(r.edgeRate - 0.9) * 100).toFixed(2)} points`);
}

200000 requests, 10 queries per shipment, 100 open bursts, cache capacity 1000 keys

burst | single cache | edge hit | reaching app | app hit | reaching store
------|--------------|----------|---------------|---------|----------------
   10 |       0.8998 |   0.8998 |         20046 |  0.0000 |          20046
    5 |       0.3998 |   0.3998 |        120044 |  0.0000 |         120044
    2 |       0.0759 |   0.0759 |        184815 |  0.0021 |         184418

burst 10: requests reaching store without edge 20046, with edge 20046 (x1.000)
requests reaching app 200000 -> 20046 (x0.100)
measured edge hit rate 0.8998, V9 assumption 0.90 -> difference 0.02 points

The first line exposes an assumption from the introductory course: the cache hit ratio there, V9 = 0.90, was justified only as “the same tracking number gets asked again within a short interval.” At burst length 10, the measured rate is 0.8998 — a 0.02-point difference — so V9 = 0.90 was quietly assuming all ten of a shipment’s queries land in the same burst. Sensitivity shows in the next two lines: at burst 5 the rate is 0.3998, at burst 2 it is 0.0759. Half the queries falling outside the burst drops the rate from 0.90 to 0.40. A hit rate is not a cache property — it is a trace property.

The second result is the location decision’s real finding: adding the edge cache dropped requests reaching the application from 200,000 to 20,046 — a tenth. Requests reaching the store did not change at all — 20,046 with or without the edge, a ratio of 1.000 — because the application cache’s hit rate is 0.0000. The edge cache absorbs every repeat, so only first-seen keys reach the cache behind it, and a first-seen key cannot be found in any cache.

This is a general property of chaining two caches in series: if the last cache holds every distinct key in the window, requests reaching the store are independent of the caches in front of it. The third row shows the limit: at burst 2 the edge cache is already failing (0.0759), and the cache behind it adds a small contribution (0.0021), bringing requests reaching the store from 184,815 to 184,418. The second cache only earns its keep once the first fails. Capacity and hit rate’s relationship was measured in the Cache Metrics lesson; added here is that caching the same key twice is not free the second time.

Back to the Numbers

// edge/compute.mjs — folding the measured hit rate and wire bytes back into the K01 computations
import { run } from "./trace.mjs";

const PEAK_READ = 416.67, PEAK_WRITE = 97.22;   // K01 calculation
const V5 = 480, V9 = 0.90;                      // K01 assumption table
const K01_BEHIND_CACHE = 41.67, K01_STORE = 138.89, K01_EGRESS = 1.60;
const WIRE_EDGE = 649, WIRE_ORIGIN = 636;       // measurement: body + header (local processes)

const r = run(10);
const app = PEAK_READ * (1 - r.edgeRate);
const storeReads = PEAK_READ * (r.edgeStored / r.requests);
const mbit = (requests, bytes) => (requests * bytes * 8) / 1e6;

console.log(`measured edge hit rate = ${r.edgeRate.toFixed(4)} (V9 assumption ${V9})`);
console.log();
console.log(`${"calculation".padEnd(28)}${"K01".padStart(9)}${"with edge".padStart(11)}  note`);
const row = (name, before, after, note) =>
  console.log(`${name.padEnd(28)}${before.toFixed(2).padStart(9)}${after.toFixed(2).padStart(11)}  ${note}`);
row("reads reaching app/s", PEAK_READ, app, "edge cache removed these");
row("reads behind cache/s", K01_BEHIND_CACHE, storeReads, "unchanged");
row("requests reaching store/s", K01_STORE, storeReads + PEAK_WRITE, "unchanged");
row("read egress Mbit/s", K01_EGRESS, mbit(PEAK_READ, WIRE_EDGE), "K01 counted body only");
row("origin egress Mbit/s", K01_EGRESS, mbit(app, WIRE_ORIGIN), "edge sits in front of the origin");
console.log();
console.log(`wire/body ratio = ${(WIRE_EDGE / V5).toFixed(3)}; K01's ${K01_EGRESS.toFixed(2)} Mbit/s is a lower bound`);
console.log(`origin egress / user egress = ` +
  `${(mbit(app, WIRE_ORIGIN) / mbit(PEAK_READ, WIRE_EDGE)).toFixed(4)}`);
console.log(`app cache's contribution = ${(r.appRate * 100).toFixed(2)}% hit`);

measured edge hit rate = 0.8998 (V9 assumption 0.9)

calculation                       K01  with edge  note
reads reaching app/s           416.67      41.76  edge cache removed these
reads behind cache/s            41.67      41.76  unchanged
requests reaching store/s      138.89     138.98  unchanged
read egress Mbit/s               1.60       2.16  K01 counted body only
origin egress Mbit/s             1.60       0.21  edge sits in front of the origin

wire/body ratio = 1.352; K01's 1.60 Mbit/s is a lower bound
origin egress / user egress = 0.0982
app cache's contribution = 0.00% hit

The table gives four results at once. Reads reaching the application drop from 416.67 to 41.76 — a number the introductory course never computed, since there the cache sat inside the application and reaching it was unavoidable. reads behind cache/s and requests reaching store/s do not change (41.67 against 41.76, 138.89 against 138.98; the gap is the trace measurement’s 0.02-point deviation). The edge cache takes the same 0.90 and moves it from one place to another.

Bandwidth gets corrected in both directions: egress to the user, with measured wire bytes, rises from 1.60 to 2.16 Mbit/s — the introductory course’s number was a lower bound, since it counted only the body. Origin egress is 0.21 Mbit/s, 0.0982 of egress to the user. One link in the same design demands 2.16 Mbit/s, the other 0.21, and the introductory course’s single number cannot stand for both.

Summary

• The edge cache is a location decision: a hit never reaches the application and does not use the link between origin and edge, but the hit rate does not grow because of this decision.
• The same body’s footprint on a real connection was measured: a 480-byte body takes up 649 bytes on the wire, a ratio of 1.352; the introductory course’s 1.60 Mbit/s is therefore a lower bound.
• At a burst length of 10, the measured hit rate came out to 0.8998: the V9 = 0.90 assumption assumed all ten of a shipment’s queries land in the same burst. At burst 5 the rate falls to 0.3998.
• The edge cache brought requests reaching the application from 200,000 down to 20,046, but requests reaching the store stayed at 20,046 (ratio 1.000), and the application cache’s hit rate was 0.0000.
• If the last cache can hold every distinct key in the window, requests reaching the store are independent of the caches in front of it; the second cache only contributes once the first one fails.
• Back to the numbers: reads reaching the application fell from 416.67 to 41.76, requests reaching store/s stayed at 138.89, and origin egress was 0.21 Mbit/s, 0.0982 of egress to the user.

Next Step

The edge cache in this lesson waited for the request: the first time a key was asked for, it missed, went to the origin, and stored the response. The measurement also counted those first requests — 20,046 of the 200,000 missed at the edge, each a trip to the origin. This waiting is not mandatory: content can be sent to the edge before it is ever asked for. The choice between the two models changes the miss count, the content volume held at the edge, and how long an update takes to become visible at every edge. The next lesson measures both on the same setup and shows which fits which kind of content.