Cache Metrics

Throughout this topic, every lesson produced a number: queries reaching the origin, stale reads, hits and misses, downloaded bytes. Each of these numbers was meaningful on its own, but none of them answers the question “is the cache working” by itself. In the Cache Key Design lesson, the highest hit ratio belonged to the wrong implementation; this lesson shows what a single ratio hides even in correct implementations.

The metric’s definition is simple. The hit ratio is the share of requests served from the cache:

$$h = \frac{\text{hits}}{\text{hits} + \text{misses}}$$

Average response time is made up of two components: every request checks the cache, and the ones that miss also go to the origin. If cache access is $t_c$ and origin access is $t_o$:

$$T = t_c + (1 - h)\,t_o$$

This form explains why an improvement in the hit ratio pays off disproportionately. When the ratio rises from 90% to 95%, the improvement is five points, but because the $(1-h)$ factor drops from 0.10 to 0.05, the load placed on the origin is cut in half.

Measurement Setup

# setup.sh — book and loan data for the measurement
rm -f library.db
sqlite3 library.db <<'SQL'
CREATE TABLE book (book_id INTEGER PRIMARY KEY, title TEXT NOT NULL, author TEXT NOT NULL,
                    publication_year INTEGER, branch_id INTEGER NOT NULL);
CREATE TABLE loan (loan_id INTEGER PRIMARY KEY, book_id INTEGER NOT NULL,
                    member_id INTEGER NOT NULL, return_date TEXT);
INSERT INTO book SELECT n, 'Book ' || n, 'Author ' || (n % 40 + 1), 1950 + n % 70, n % 4 + 1
  FROM (WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n < 300) SELECT n FROM s);
INSERT INTO loan SELECT n, n % 300 + 1, n % 200 + 1, CASE WHEN n % 5 = 0 THEN NULL ELSE '2025-12-31' END
  FROM (WITH RECURSIVE s(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM s WHERE n < 4000) SELECT n FROM s);
CREATE INDEX loan_member ON loan(member_id);
SQL
sqlite3 library.db "SELECT 'book=' || (SELECT count(*) FROM book) || ' loan=' || (SELECT count(*) FROM loan);"

book=300 loan=4000

The measurement needs a cache that gathers, in one place, the behaviors built piece by piece throughout this topic: a lifetime, least-recently-used eviction once capacity fills, and counters.

// cache.mjs — a cache with expiry, least-recently-used eviction, and counters
export function openCache({ capacity = 128, lifetime = Infinity, clock = () => Date.now() } = {}) {
  const box = new Map();
  const counter = { hit: 0, miss: 0, eviction: 0, expiry: 0, hitBytes: 0, missBytes: 0 };
  const sizeOf = (d) => Buffer.byteLength(JSON.stringify(d));
  return {
    counter,
    get(key) {
      const entry = box.get(key);
      if (entry === undefined) { counter.miss += 1; return undefined; }
      if (entry.validUntil <= clock()) {
        box.delete(key); counter.expiry += 1; counter.miss += 1; return undefined;
      }
      box.delete(key); box.set(key, entry);   // the most recently used entry moves to the end
      counter.hit += 1; counter.hitBytes += entry.size;
      return entry.value;
    },
    set(key, value) {
      const size = sizeOf(value);
      counter.missBytes += size;
      box.delete(key);
      box.set(key, { value, size, validUntil: clock() + lifetime });
      while (box.size > capacity) { box.delete(box.keys().next().value); counter.eviction += 1; }
    },
    get entryCount() { return box.size; },
    hitRatio: () => counter.hit / (counter.hit + counter.miss),
    byteHitRatio: () => counter.hitBytes / (counter.hitBytes + counter.missBytes),
  };
}

The eviction order comes from the Map’s insertion order: a read entry is deleted and re-inserted, moving it to the end; once capacity is exceeded, the entry at the front — the one unused for the longest — drops.

Capacity and Hit Ratio

The first measurement turns a single dial: cache capacity. The workload is 2000 requests and covers three operation types; different key counts produce different repeat rates.

// capacity.mjs — how the hit ratio and origin queries change as capacity grows
import { DatabaseSync } from "node:sqlite";
import { openCache } from "./cache.mjs";

const db = new DatabaseSync("library.db");
const query = { count: 0 };
const origin = {
  book: (id) => { query.count += 1; return db.prepare(
    "SELECT book_id, title, author, publication_year FROM book WHERE book_id = ?").get(id); },
  stock: (id) => { query.count += 1; return db.prepare(
    `SELECT ? AS branch_id, count(*) AS on_shelf FROM book k WHERE k.branch_id = ? AND NOT EXISTS
     (SELECT 1 FROM loan o WHERE o.book_id = k.book_id AND o.return_date IS NULL)`).get(id, id); },
  list: (id) => { query.count += 1; return db.prepare(
    "SELECT book_id, title FROM book WHERE branch_id = ? ORDER BY book_id").all(id); },
};

function workload(count) {                          // book requests concentrate on 60 titles
  let seed = 20250729;
  const random = () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648);
  return Array.from({ length: count }, () => {
    const r = random();
    if (r < 0.75) return ["book", Math.floor(random() * 60) + 1];
    if (r < 0.9) return ["stock", Math.floor(random() * 4) + 1];
    return ["list", Math.floor(random() * 4) + 1];
  });
}

const requests = workload(2000);
console.log(["capacity", "hits", "misses", "hit ratio", "evictions", "origin queries", "duration~"]
  .map((h, i) => (i === 0 ? h.padEnd(10) : h.padStart([0, 8, 8, 14, 11, 16, 11][i]))).join(""));

for (const capacity of [4, 8, 16, 32, 64, 128]) {
  query.count = 0;
  const cache = openCache({ capacity });
  const t = performance.now();
  for (const [kind, id] of requests) {
    const key = `library:s1:g1:t${kind === "book" ? "shared" : `branch-${id}`}:${kind}:${id}`;
    let value = cache.get(key);
    if (value === undefined) { value = origin[kind](id); cache.set(key, value); }
  }
  const duration = Math.round((performance.now() - t) / 10) * 10;
  const c = cache.counter;
  console.log(String(capacity).padEnd(10) + [String(c.hit).padStart(8), String(c.miss).padStart(8),
    (`%` + (cache.hitRatio() * 100).toFixed(1)).padStart(14), String(c.eviction).padStart(11),
    String(query.count).padStart(16), (duration + " ms").padStart(11)].join(""));
}
db.close();

capacity      hits  misses     hit ratio  evictions  origin queries  duration~
4              133    1867          %6.7       1863            1867     910 ms
8              289    1711         %14.4       1703            1711     760 ms
16             588    1412         %29.4       1396            1412     460 ms
32            1053     947         %52.6        915             947     200 ms
64            1851     149         %92.5         85             149      10 ms
128           1932      68         %96.6          0              68      10 ms

The duration column is rounded to ten milliseconds and is machine-dependent; the first five columns are not.

The shape of the curve tells the story. When capacity rises from 32 to 64, the hit ratio jumps from 52.6% to 92.5%; going from 64 to 128, it gains only four points. The location of the jump gives the workload’s working set: 60 book keys, 4 stock keys, and 4 list keys, 68 distinct keys in total. While capacity sits below this number, every new key evicts another one, and the eviction count comes out nearly equal to the miss count — at capacity 4, 1867 misses against 1863 evictions. Once capacity passes the working set, evictions drop to zero, and the remaining 68 misses are just the first loads.

This gives the right question for a capacity decision: not “how much memory should we allocate,” but “how many keys are in the working set.” The eviction counter answers this question directly. If evictions are close to the miss count, the cache is feeding its own entries to each other; if they are close to zero, capacity is sufficient.

The duration column shows what the formula predicts. At capacity 4, the hit ratio was 6.7% and the workload took 910 milliseconds; at 92.5%, it fell to 10 milliseconds. The gap comes from the miss penalty being large in this workload — the stock query is more expensive than the others.

What the Total Ratio Hides

The table above had a single ratio because every part of the workload was cache-friendly. Real applications are not like that. The workload below adds two hundred different members’ loan histories: a read with many keys and a low repeat rate.

// breakdown.mjs — what a single hit ratio hides: broken down by namespace and by byte
import { DatabaseSync } from "node:sqlite";
import { openCache } from "./cache.mjs";

const db = new DatabaseSync("library.db");
const origin = {
  book: (id) => db.prepare("SELECT book_id, title, author FROM book WHERE book_id = ?").get(id),
  stock: (id) => db.prepare(
    `SELECT ? AS branch_id, count(*) AS on_shelf FROM book k WHERE k.branch_id = ? AND NOT EXISTS
     (SELECT 1 FROM loan o WHERE o.book_id = k.book_id AND o.return_date IS NULL)`).get(id, id),
  list: (id) => db.prepare("SELECT book_id, title FROM book WHERE branch_id = ? ORDER BY book_id").all(id),
  history: (id) => db.prepare(
    "SELECT loan_id, book_id FROM loan WHERE member_id = ? ORDER BY loan_id").all(id),
};

let seed = 20250729;
const random = () => ((seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648);
const requests = Array.from({ length: 2000 }, () => {
  const r = random();
  if (r < 0.70) return ["book", Math.floor(random() * 60) + 1];
  if (r < 0.80) return ["stock", Math.floor(random() * 4) + 1];
  if (r < 0.85) return ["list", Math.floor(random() * 4) + 1];
  return ["history", Math.floor(random() * 200) + 1];    // 200 members, low repeat rate
});

function run(cacheable) {
  const cache = openCache({ capacity: 64 });
  const perKind = { book: [0, 0], stock: [0, 0], list: [0, 0], history: [0, 0] };  // [hits, misses]
  let originQueries = 0;
  for (const [kind, id] of requests) {
    if (!cacheable.has(kind)) { origin[kind](id); originQueries += 1; perKind[kind][1] += 1; continue; }
    const key = `library:s1:g1:t${kind === "book" ? "shared" : `branch-${(id % 4) + 1}`}:${kind}:${id}`;
    const previousHits = cache.counter.hit;
    let value = cache.get(key);
    if (value === undefined) { value = origin[kind](id); originQueries += 1; cache.set(key, value); }
    perKind[kind][cache.counter.hit > previousHits ? 0 : 1] += 1;
  }
  return { cache, perKind, originQueries };
}

const { cache, perKind, originQueries } = run(new Set(["book", "stock", "list", "history"]));

console.log(["namespace", "requests", "hits", "misses", "hit ratio"]
  .map((h, i) => (i === 0 ? h.padEnd(10) : h.padStart([0, 8, 8, 8, 14][i]))).join(""));
for (const [name, [i, k]] of Object.entries(perKind)) {
  console.log(name.padEnd(10) + [String(i + k).padStart(8), String(i).padStart(8), String(k).padStart(8),
    ("%" + ((i / (i + k)) * 100).toFixed(1)).padStart(14)].join(""));
}
const c = cache.counter;
console.log(`\ntotal request hit ratio : %${(cache.hitRatio() * 100).toFixed(1)}`);
console.log(`byte hit ratio          : %${(cache.byteHitRatio() * 100).toFixed(1)}`);
console.log(`bytes returned from hits: ${c.hitBytes}   bytes from the origin: ${c.missBytes}`);
console.log(`origin queries          : ${originQueries}`);

const second = run(new Set(["book", "stock", "list"]));    // history is never cached at all
console.log(`\nwith history not cached:`);
console.log(`  book hit ratio        : %${((second.perKind.book[0] / (second.perKind.book[0] + second.perKind.book[1])) * 100).toFixed(1)}`);
console.log(`  evictions             : ${second.cache.counter.eviction} (previously: ${c.eviction})`);
console.log(`  origin queries        : ${second.originQueries}`);
db.close();

namespace requests    hits  misses     hit ratio
book          1380     965     415         %69.9
stock          202     189      13         %93.6
list           110      84      26         %76.4
history        308      19     289          %6.2

total request hit ratio : %62.8
byte hit ratio          : %51.7
bytes returned from hits: 283673   bytes from the origin: 264714
origin queries          : 743

with history not cached:
  book hit ratio        : %90.7
  evictions             : 77 (previously: 679)
  origin queries        : 449

The total ratio is 62.8%, and reported on its own, it gives the impression of a “moderately effective cache.” The breakdown says something else: the loan-history namespace’s hit ratio is 6.2%. Two hundred distinct keys and a low repeat rate make this data a poor fit for caching. Worse, this namespace is actively harmful — every entry it places takes up capacity and gets evicted without ever being read again. This is what drags the book namespace’s hit ratio down to 69.9%.

The last section proves this. When loan history was not cached at all, the book namespace’s hit ratio rose from 69.9% to 90.7%, the eviction count fell from 679 to 77, and the total queries reaching the origin dropped from 743 to 449. Removing a namespace from the cache lowered the load on the origin. This result could only be found with a broken-down measurement; the total ratio would have looked similar under both conditions.

The operating rule that follows is this: the hit ratio is measured per namespace. This is the second job of the namespace segment placed in the key in the Cache Key Design lesson — making the metric decomposable.

Byte Hit Ratio

The second pair of lines in the output shows a separate distinction. The request hit ratio is 62.8%, the byte hit ratio is 51.7%. The difference comes from the values not being equal in size: list and history responses are much larger than a book record.

The two ratios answer different questions. The request hit ratio says how many calls did not go to the origin; this number sets the round-trip and query cost. The byte hit ratio says how much data did not cross the network; this number sets bandwidth and transfer time. These two metrics, defined for the edge cache in the Static Hosting lesson of the Frontend Quality course, carry the same distinction in the server cache.

Which one to track depends on the bottleneck. If the origin’s query count is under strain, the request hit ratio is decisive; if the network is under strain, the byte hit ratio is.

Summary

• The hit ratio is the share of requests served from the cache; average time behaves as $T = t_c + (1-h)t_o$, which is why rising from 90% to 95% cuts the origin load in half.
• While capacity sits below the working set, the eviction count approaches the miss count; in a 68-key working set, raising capacity from 32 to 64 made the hit ratio jump from 52.6% to 92.5%.
• The eviction counter is the direct measure for a capacity decision: close to the miss count means the cache is eating its own entries; close to zero means capacity is sufficient.
• A single total ratio hides differences between namespaces: inside a 62.8% total there was a namespace at 6.2%, and once that namespace was removed from the cache, origin queries fell from 743 to 449.
• The request hit ratio measures the number of calls reaching the origin, and the byte hit ratio measures the amount of data that does not cross the network; in the same run, they came out to 62.8% and 51.7%.

Next Step

The work done throughout this topic can be summed up in a single sentence: reproducing the same response was prevented. The 1867 origin queries in the capacity measurement fell to 68; the read path genuinely got shorter.

One path remains that did not get shorter. A cache only helps reads; a write always goes to the origin and adds to the request’s duration. Work attached alongside a write is the same way: a notification sent when a loan is issued, an overdue report produced at the end of the day, processing an uploaded cover image. None of this is necessary for the user’s response, but all of it delays the response from arriving. The second way to shorten the request path is not to speed the work up, but to take the work out of the request. The next lesson opens this path: it takes up when running a job asynchronously is the right call, what drops out of the request’s duration, and what new responsibilities arise in exchange.