Cache Server

The last three lessons built and encrypted the request’s path: who arrived was relayed, encryption terminated somewhere, the certificate refreshes itself. The whole path runs start to finish for every request.

The measurement network’s dashboard asks for the same regional map, the same meter list, and the same nightly summary file every time it opens. Most of these requests produce the same response. The response can be held somewhere — but “somewhere” is two separate places. The edge cache sits close to the client and never sends the request across the network at all. The origin cache sits in front of the app and keeps the app from generating the same response twice.

What a cache is, how the hit rate is calculated, and what freshness means were measured before; this lesson does not repeat that mechanics. What is measured here is three things: how the hit rate splits between the two layers, how many more seconds the old response stays at the edge after a piece of content changes, and the surface a wrong cache key opens.

• WS25. The request stream is produced by our own generator; the seed 20260 is visible and gives the same sequence run to run. Two thousand requests are spread across six hundred seconds.
• WS26. There are five pieces of content and their weights are fixed; one (/invoice/summary) is personal, meaning its response varies by the subscriber making the request.
• WS27. A subscriber always lands on the same edge node: the node is the remainder of the subscriber’s ID divided by the node count.
• WS28. The default cache key is the path alone; identity does not enter the key. Two more options are measured: adding identity to the key, and never storing personal content at all.
• WS29. The time for the invalidation broadcast to reach the three edge nodes is fictional: two, five, and eleven seconds. The window is computed against the slowest node.
• WS30. The model does not measure time; the second values are derived from the freshness settings and the fictional delays, and the count of requests receiving a stale response is computed from the changed content’s share of traffic.

// cache/layer.mjs — model: a two-layer cache (edge + origin). The fictional measurement network's
// request stream comes from our own generator; the seed is visible. The cache mechanism is not
// redefined; what is measured is the hit split between the two layers and the invalidation window.
const PERIOD = 600, REQUESTS = 2000, SUBSCRIBERS = 6, SEED = 20260;
// [path, weight, type]
const CONTENT = [
  ["/region/map", 40, "shared"], ["/meter/list", 25, "shared"],
  ["/nightly/summary", 15, "shared"], ["/asset/logo", 10, "shared"],
  ["/invoice/summary", 10, "personal"],
];
const TOTAL = CONTENT.reduce((a, x) => a + x[1], 0);
const rng = (seed) => { let s = seed >>> 0; return () => (s = (Math.imul(s, 1664525) + 1013904223) >>> 0) / 4294967296; };
const pick = (u) => { let e = u * TOTAL; return CONTENT.find((x) => (e -= x[1]) < 0) ?? CONTENT[0]; };

function simulate({ node, hasOrigin, edgeFreshness, originFreshness, key }) {
  const r = rng(SEED);
  const edge = Array.from({ length: node }, () => new Map());
  const origin = new Map();
  const s = { edgeHits: 0, originHits: 0, app: 0, leakedRequests: 0 };
  const leakedSubscribers = new Set();
  for (let i = 0; i < REQUESTS; i += 1) {
    const t = Math.floor((i * PERIOD) / REQUESTS);
    const [path, , type] = pick(r());
    const subscriber = Math.floor(r() * SUBSCRIBERS);
    const personal = type === "personal";
    if (personal && key === "personal-bypass") { s.app += 1; continue; }   // personal content is never cached
    const cacheKey = key === "path+id" && personal ? `${path}#${subscriber}` : path;
    const edgeNode = edge[subscriber % node];
    const record = (entry) => {                         // whose response answered the hit
      if (personal && entry.subscriber !== subscriber) { s.leakedRequests += 1; leakedSubscribers.add(entry.subscriber); }
    };
    const hit = edgeNode.get(cacheKey);
    if (hit && t - hit.t < edgeFreshness) { s.edgeHits += 1; record(hit); continue; }
    const originHit = hasOrigin ? origin.get(cacheKey) : undefined;
    if (originHit && t - originHit.t < originFreshness) {
      s.originHits += 1; record(originHit); edgeNode.set(cacheKey, { t, subscriber: originHit.subscriber }); continue;
    }
    s.app += 1;                                          // neither layer hits: the app generates it
    if (hasOrigin) origin.set(cacheKey, { t, subscriber });
    edgeNode.set(cacheKey, { t, subscriber });
  }
  return { ...s, leakedSubscribers: leakedSubscribers.size };
}

const E = [8, 9, 18, 12, 14, 14, 13, 13];
const write = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](E[i])).join("");
console.log(write(["node", "origin", "key", "edge hits", "origin hits", "reaches app",
  "leaked req", "leaked sub"]));
const SETTING = [
  [1, false, "path"], [3, false, "path"], [3, true, "path"],
  [3, true, "path+id"], [3, true, "personal-bypass"],
];
for (const [node, hasOrigin, key] of SETTING) {
  const o = simulate({ node, hasOrigin, edgeFreshness: 60, originFreshness: 300, key });
  console.log(write([node, hasOrigin ? "yes" : "no", key, o.edgeHits, o.originHits,
    o.app, o.leakedRequests, o.leakedSubscribers]));
}

// --- Invalidation's propagation window -------------------------------------
// Content at the origin changed at t=0. The window is how many seconds the edge keeps the old
// content. Broadcast delay is fictional: it reaches three edge nodes at these seconds, in order.
const DELAY = [2, 5, 11];
const CHANGED = CONTENT[0];                             // changed content: /region/map
const RATE = (REQUESTS / PERIOD) * (CHANGED[1] / TOTAL);   // req/second arriving for that content
function window(method, edgeFreshness, originFreshness) {
  if (method === "none") return edgeFreshness + originFreshness;      // the two freshnesses stack
  if (method === "at-origin") return edgeFreshness;                    // the edge waits out its own duration
  if (method === "every-layer") return Math.max(...DELAY);
  return edgeFreshness;                                                // one node missed the broadcast
}
const P = [18, 16, 18, 12, 24];
const write2 = (h) => h.map((d, i) => String(d)[i ? "padStart" : "padEnd"](P[i])).join("");
console.log("\n" + write2(["invalidation", "edge freshness", "origin freshness", "window(s)",
  "stale responses served"]));
for (const [y, ef, of] of [["none", 60, 300], ["none", 10, 60], ["at-origin", 60, 300],
  ["at-origin", 10, 300], ["every-layer", 60, 300], ["one-node-missed", 60, 300]]) {
  const p = window(y, ef, of);
  console.log(write2([y, ef, of, p, Math.round(p * RATE)]));
}

node       origin               key   edge hits   origin hits   reaches app   leaked req   leaked sub
1              no              path        1950             0            50          169            5
3              no              path        1857             0           143           95            6
3             yes              path        1857           133            10          179            2
3             yes           path+id        1837           143            20            0            0
3             yes   personal-bypass        1664           107           229            0            0

invalidation        edge freshness  origin freshness   window(s)  stale responses served
none                            60               300         360                     480
none                            10                60          70                      93
at-origin                       60               300          60                      80
at-origin                       10               300          10                      13
every-layer                     60               300          11                      15
one-node-missed                 60               300          60                      80

The Cost of Splitting the Edge

The first two rows pass the same two thousand requests through the same freshness settings. The only thing that changes is the edge node count.

With a single node, the edge hit count is nineteen hundred fifty, and requests reaching the app are fifty. When the edge splits into three nodes, hits drop to eighteen hundred fifty-seven, and requests reaching the app climb from fifty to one hundred forty-three. Roughly triple.

The reason is direct: every node has its own cache and requests the same content three times, separately. When a piece of content refreshes, it is no longer one request but three that reach the app. The edge node count is a scaling decision — more nodes means a response closer to the client — and it multiplies the load reaching the app by the node count.

This is the two-layer cache’s reason for existing. The third row adds the origin cache: the edge hit count does not change (eighteen hundred fifty-seven), one hundred thirty-three requests are met at the origin cache, and requests reaching the app drop to ten. The origin cache absorbs nearly all of the request the three nodes multiplied.

Read together, the numbers make the origin cache’s job visible: it does not reduce the request coming from the edge itself, it keeps that request from reaching the app. The third row’s app load is even lower than the single-node first row’s — the origin cache absorbs not just the node multiplication but a single node’s own freshness refills too.

The Propagation Window

The second table measures another question. The regional map changed; when does the new map become visible?

The first row is the default state: no invalidation, both layers wait out their own freshness duration. The window is sixty plus three hundred, that is, three hundred sixty seconds. The two freshnesses add up because they stack: when the edge pulls a copy from the origin, that copy may have already waited up to three hundred seconds at the origin, and the edge holds it for sixty seconds more. Over the window, four hundred eighty requests for the changed content get the old map.

The second row shortens the freshness settings: ten and sixty instead of sixty and three hundred. The window drops to seventy seconds, stale responses served drop to ninety-three. The cost does not show up in this table, but it can be read from the first table: as freshness shortens, the hit rate drops and requests reaching the app rise. Shortening freshness is the expensive way to close the window.

The third row sends invalidation to the origin. The origin cache clears immediately, the edge waits out its own sixty seconds: the window is sixty seconds, stale responses served are eighty. A gain from three hundred sixty down to sixty is achieved with a single message sent to the origin.

The fourth row sends invalidation to the origin while also dropping the edge freshness to ten: the window is ten seconds. The fifth row broadcasts invalidation to every layer: the window drops to the slowest edge node’s delay, eleven seconds, and edge freshness no longer determines the window at all.

The order matters here. Shortening the freshness setting closes the window and drops the hit rate; broadcasting invalidation closes the window and does not touch the hit rate. The fifth row runs with a sixty-second edge freshness — that is, the first row’s hit rate — and its window is eleven seconds.

Where the Broadcast Misses

The sixth row shows what the fifth row cannot close. Invalidation broadcasts to every layer, but one edge node does not get the message: a network outage, the node restarting, a queue overflowing. On that node, the window falls back to the edge freshness: sixty seconds.

This is the new surface invalidation introduces. The freshness duration provides an upper bound, and it holds even if nothing else is done. The invalidation broadcast does not provide an upper bound; it provides a hope, and when the hope does not hold, the system silently falls back to the upper bound. The system’s state is eleven seconds on two nodes, sixty seconds on one node, and no one watching from outside knows which node they landed on.

The conclusion is that freshness must be kept alongside the invalidation broadcast. The broadcast provides the fast path, freshness provides the fallback bound. Stretching freshness to infinity and pinning everything on the broadcast turns a single missed message into indefinitely stale content.

The Shared-Key Surface

The last columns are the lesson’s surface measurement. The default cache key is the path alone, and /invoice/summary is personal: every subscriber gets their own invoice summary.

In the first row, one hundred sixty-nine requests received another subscriber’s invoice summary; five subscribers’ data went to someone else. In the three-node second row, leaked requests drop to ninety-five, but the leaked-subscriber count climbs to six — splitting does not reduce the leak, it spreads it.

The third row is the sharpest. When the origin cache is added, leaked requests climb to one hundred seventy-nine, and the leaked-subscriber count drops to two. A single shared origin entry distributes two subscribers’ invoices to all three edge nodes at once. The origin cache concentrates the leak: fewer people’s data, to more people.

The name of this surface is the shared-key surface. Opening it does not require a bug; it is enough for the default key to include the path and not the identity. The knowledge that a response is personal lives in the app, not in the cache, and neither layer asks for that knowledge.

The fourth row adds identity to the key: leaked requests zero, leaked subscribers zero. Requests reaching the app climb from ten to twenty — since six separate entries are held for six separate subscribers, freshness refills multiply. For a cost of ten requests, one hundred seventy-nine leaks closed.

The fifth row never stores personal content at all. The leak is zero again, but requests reaching the app climb to two hundred twenty-nine. There is a twentyfold difference between the two safe options, and both close the same surface. Setting up the key correctly is twenty times cheaper than turning caching off.

The Cache’s Own Surface

The two layers dropped the app’s load from fifty to ten, and in doing so introduced four new items.

The first is the stored response itself. Already-generated responses now sit at the edge and the origin; they live somewhere on disk or in memory, and anyone able to read that place reads the app’s responses without authentication. The second is the invalidation channel: an endpoint that can send a message and delete an entry adds two items to the management-endpoint list the previous lessons counted — one at the edge, one at the origin. If this endpoint has no identity, an invalidation stream sent from outside can flush the entire cache, and requests reaching the app climb from ten to two thousand.

The third is the key rule itself. As the fifth row showed, the rule is a configuration line, and getting it wrong throws no error; it just returns another subscriber’s response. The fourth is the freshness setting being split in two: the window’s real value comes out of two separate numbers on two separate layers, and no single layer knows that value by itself.

None of the four items show up in the sentence “the cache is on.” The course’s rule holds here too: a decision cannot be defended unless the load it closes is written down alongside the surface it opens. If a step that reduces the requests reaching the app by forty items adds four items of open surface, both numbers belong on the same line of the report.

Summary

• Splitting the edge into three nodes dropped edge hits from nineteen hundred fifty to eighteen hundred fifty-seven and raised requests reaching the app from fifty to one hundred forty-three; the node count multiplies the app’s load.
• The origin cache met one hundred thirty-three requests and dropped requests reaching the app to ten — lower even than the single-node arrangement; the origin layer’s job is absorbing the request the edge multiplied.
• With no invalidation, the window is the sum of the two freshnesses: three hundred sixty seconds and four hundred eighty stale responses. Invalidation to the origin dropped the window to sixty; broadcasting to every layer dropped it to eleven seconds.
• Shortening freshness closes the window and drops the hit rate; broadcasting invalidation closes the window and does not touch the hit rate — but on a node that misses the broadcast, the window silently falls back to the freshness duration.
• Because the default key contained only the path, one hundred seventy-nine requests received another subscriber’s invoice summary; the origin cache concentrated the leak — two subscribers’ data went out in one hundred seventy-nine requests.
• Adding identity to the key zeroed the leak and raised the app load to twenty, not ten; never storing personal content also zeroed the leak but raised the load to two hundred twenty-nine.

Next Step

This lesson set two rules on traffic at the edge and the origin: what gets stored and for how long it stays fresh. Both rules sat in the request’s path and applied to traffic coming from outside.

Services also send requests to each other. The verifier connects to billing, billing to work orders, work orders to the notification gateway, and none of that traffic passes through the edge proxy. Timeout, retry, which service can call which — today these rules sit in each service’s own code. The same rules can be moved to a proxy placed next to each service. The next lesson compares the two arrangements on the same rule change: how many services get touched when a rule changes, how many hops and how many milliseconds get added to every request, what the sidecar proxy itself consumes, and what new surface it introduces. In a three-service system, the point where the cost outweighs the gain is shown with numbers.