Managed Cache Service

The previous lesson widened the access pattern: the secondary index and the change stream reopened a share of the queries the partition key had closed off. But every reopened query still lands on the data layer.

In-memory stores and eviction policies were measured earlier; this lesson does not repeat that mechanics. Here the in-memory layer is a rented service, and the three things measured are what the scaling unit is, what is lost when a node changes, and what the cache’s location writes onto the bill.

• MD25. The fictional regional measurement network’s read set is three families: latest meter reading (2.4 million keys, 180 bytes), invoice summary (600 thousand, 2,400 bytes), work-order record (4 million, 600 bytes); 4.272 GB total.
• MD26. Each family is modeled with three popularity tiers. The model fills the cache in order of per-key probability, does not count first-touch misses, and uses no randomness.
• MD27. Reads average four hundred, peak three thousand requests/s; there is also a steady four hundred requests/s of writes, and writes never touch the cache. A month is counted as 2,592,000 seconds.
• MD28. Prices are fictional, the currency’s name is a fictional unit: a cache node runs sixty units per GB a month; read capacity is bought in hundred-requests/s steps, and a step is thirty units a month. The provider opens up 75% of the node’s memory for use.
• MD29. Latencies are fictional: the data layer is 6 ms, a managed cache in the same zone 0.8 ms, in another zone 1.9 ms, in-process 0.02 ms. Cross-zone transfer is one unit per GB.
• MD30. The meter collector accumulates its minute-by-minute totals in the cache and flushes every five seconds; the cache copy’s replication lag is 0.25 seconds. Both are fictional.

// cache/cache.mjs — model: what changes when the in-memory layer is rented as a service.
// Eviction policies were measured in M17/K06 and are not repeated. The working set is modeled
// with popularity tiers, no randomness; prices are fictional and 1 GB is counted as 1e9 bytes.
const TIER = [                        // [family, tier, keys, bytes, access share]
  ["meter", "hot", 150000, 180, 0.15], ["meter", "warm", 850000, 180, 0.20],
  ["meter", "cold", 1400000, 180, 0.16],
  ["invoice", "hot", 18000, 2400, 0.10], ["invoice", "warm", 140000, 2400, 0.09],
  ["invoice", "cold", 442000, 2400, 0.04],
  ["workorder", "hot", 80000, 600, 0.06], ["workorder", "warm", 620000, 600, 0.10],
  ["workorder", "cold", 3300000, 600, 0.10],
];
const ORDER = [...TIER].sort((a, b) => b[4] / b[2] - a[4] / a[2]);  // probability per key
const fill = (capacity) => {          // the cache fills from the most-requested key down
  let remaining = capacity, keys = 0, hitRate = 0;
  const retained = [];
  for (const [, , n, bytes, share] of ORDER) {
    const taken = Math.min(n, Math.floor(remaining / bytes));
    if (taken === 0) break;
    remaining -= taken * bytes; keys += taken; hitRate += share * taken / n;
    retained.push([share * taken / n, share / n]);   // [share retained, probability per key]
  }
  return { keys, hitRate, retained };
};
const PEAK = 3000, WRITES = 400, USABLE_SHARE = 0.75, GB_PRICE = 60;
const capacityFee = (toBackend) => Math.ceil((toBackend + WRITES) / 100) * 30;
const columns = (widths, left = 1) => (row) =>
  row.map((d, i) => String(d)[i < left ? "padEnd" : "padStart"](widths[i])).join("");
const fmt1 = columns([14, 5, 16, 14, 11, 20, 10, 8, 15]);
console.log(fmt1(["node x GB", "GB", "usable", "keys(k)", "hit(%)",
  "to backend(req/s)", "capacity", "node", "total(unit)"]));
for (const [label, nodes, gbPerNode] of [["no cache", 0, 0], ["1 x 1", 1, 1], ["1 x 2", 1, 2],
  ["2 x 1", 2, 1], ["1 x 4", 1, 4], ["1 x 8", 1, 8]]) {
  const o = nodes ? fill(nodes * gbPerNode * 1e9 * USABLE_SHARE) : { keys: 0, hitRate: 0 };
  const toBackend = Math.round(PEAK * (1 - o.hitRate));
  const cap = capacityFee(toBackend), nodeFee = nodes * gbPerNode * GB_PRICE;
  console.log(fmt1([label, nodes * gbPerNode, (nodes * gbPerNode * USABLE_SHARE).toFixed(2), (o.keys / 1e3).toFixed(0),
    (o.hitRate * 100).toFixed(1), toBackend, cap, nodeFee, cap + nodeFee]));
}

// --- When a node changes -----------------------------------------------------------
// 2 GB layout. Since the hash spreads key names, every shard gets an equal share of every tier.
const BASE = fill(2 * 1e9 * USABLE_SHARE), TO_BACKEND = Math.round(PEAK * (1 - BASE.hitRate)) + WRITES;
const PURCHASED = capacityFee(TO_BACKEND - WRITES) / 30 * 100, PENDING = WRITES * 5, LAG = 0.25;
const fmt2 = columns([16, 18, 16, 20, 12, 17]);
console.log("\n" + fmt2(["shards", "lost keys", "lost access", "to backend(req/s)",
  "overage(%)", "pending records"]));
for (const [shards, replicated] of [[1, 0], [2, 0], [4, 0], [8, 0], [16, 0], [4, 1]]) {
  const lostKeys = Math.round((replicated ? TO_BACKEND * LAG : BASE.keys) / shards);
  const lostAccess = replicated ? 0 : BASE.hitRate / shards;
  const toBackend = Math.round(PEAK * (1 - BASE.hitRate + lostAccess)) + WRITES;
  console.log(fmt2([replicated ? `${shards} (replicated)` : shards, lostKeys, (lostAccess * 100).toFixed(1), toBackend,
    ((toBackend / PURCHASED - 1) * 100).toFixed(0),
    Math.round((replicated ? WRITES * LAG : PENDING) / shards)]));
}
const recovered = (t) => BASE.retained.reduce((a, [share, r]) => a + share * (1 - Math.exp(-r * PEAK * t)), 0);
const warmup = (target) => {
  let lo = 1, hi = 1e6;
  for (let k = 0; k < 60; k++) {
    const t = Math.sqrt(lo * hi);
    if (recovered(t) < target * BASE.hitRate) lo = t; else hi = t;
  }
  return hi.toFixed(0);
};
console.log(`warm-up, independent of shard count: 50% ${warmup(0.5)} s, 90% ${warmup(0.9)} s`);

// --- Where the cache sits -----------------------------------------------------------
const DATA_MS = 6, ZONE_PRICE = 1, MONTH = 2592000;   // ms; cross-zone unit/GB; seconds/month
const AVG_BYTES = TIER.reduce((a, [, , , bytes, share]) => a + bytes * share, 0);
const fmt3 = columns([12, 26, 11, 14, 11, 11, 10, 15], 2);
const row = (set, label, h, ms, reads, bytes, peak, writes, cacheFee, remote) => {
  const toBackend = Math.round(peak * (1 - h));
  const capacity = Math.max(1, Math.ceil((toBackend + writes) / 100)) * 30;
  const transfer = (remote ? h * reads * bytes / 1e9 : 0) * ZONE_PRICE;
  console.log(fmt3([set, label, (h * 100).toFixed(1), (ms + (1 - h) * DATA_MS).toFixed(2),
    cacheFee, capacity, transfer.toFixed(0), (cacheFee + capacity + transfer).toFixed(0)]));
};
console.log("\n" + fmt3(["set", "layout", "hit(%)", "latency(ms)", "cache", "capacity",
  "transfer", "total(unit)"]));
const FULLSET = (label, h, ms, cacheFee, remote) =>
  row("full set", label, h, ms, 400 * MONTH, AVG_BYTES, PEAK, WRITES, cacheFee, remote);
FULLSET("no cache", 0, 0, 0, false);
FULLSET("in-process 12 x 256 MB", fill(256e6).hitRate, 0.02, 0, false);
FULLSET("managed 2 GB same zone", BASE.hitRate, 0.8, 2 * GB_PRICE, false);
FULLSET("managed 2 GB other zone", BASE.hitRate, 1.9, 2 * GB_PRICE, true);
// Rate table: 20 thousand keys x 2000 bytes = 40 MB, 320 million reads a month, peak 900/s.
const RATETABLE = (label, h, ms, cacheFee, remote) =>
  row("rate table", label, h, ms, 320e6, 2000, 900, 0, cacheFee, remote);
RATETABLE("in-process 12 x 40 MB", 1, 0.02, 0, false);
RATETABLE("managed 1 GB same zone", 1, 0.8, GB_PRICE, false);
RATETABLE("managed 1 GB other zone", 1, 1.9, GB_PRICE, true);

node x GB        GB          usable       keys(k)     hit(%)   to backend(req/s)  capacity    node    total(unit)
no cache          0            0.00             0        0.0                3000      1020       0           1020
1 x 1             1            0.75          1476       63.8                1085       450      60            510
1 x 2             2            1.50          3370       87.0                 390       240     120            360
2 x 1             2            1.50          3370       87.0                 390       240     120            360
1 x 4             4            3.00          4880       93.6                 193       180     240            420
1 x 8             8            6.00          7000      100.0                   0       120     480            600

shards                   lost keys     lost access   to backend(req/s)  overage(%)  pending records
1                          3370000            87.0                3400         325             2000
2                          1685000            43.5                2095         162             1000
4                           842500            21.8                1442          80              500
8                           421250            10.9                1116          40              250
16                          210625             5.4                 953          19              125
4 (replicated)                  49             0.0                 790          -1               25
warm-up, independent of shard count: 50% 539 s, 90% 3514 s

set         layout                         hit(%)   latency(ms)      cache   capacity  transfer    total(unit)
full set    no cache                          0.0          6.00          0       1020         0           1020
full set    in-process 12 x 256 MB           34.7          3.94          0        720         0            720
full set    managed 2 GB same zone           87.0          1.58        120        240         0            360
full set    managed 2 GB other zone          87.0          2.68        120        240       722           1082
rate table  in-process 12 x 40 MB           100.0          0.02          0         30         0             30
rate table  managed 1 GB same zone          100.0          0.80         60         30         0             90
rate table  managed 1 GB other zone         100.0          1.90         60         30       640            730

Two Axes, One Curve

The managed cache service offers two scaling levers: growing the node and adding nodes. The first table puts them side by side, and the first reading is this: no difference. Two GB is reached with one node just as well as with two; both ways the cache holds 3,370,000 keys, the hit rate is 87%, the bill is 360 units. The scaling-unit decision cannot be made on hit rate and price alone.

The bill itself is the second reading. The no-cache layout runs 1,020 units; one GB lifts the hit rate to 63.8% and drops the bill to 510, two GB to 360. Then the curve turns: 420 at four GB, 600 at eight GB. The 100%-hit-rate row is 67% pricier than the two-GB row; the remaining 120 units goes to the four hundred requests/s of writes that never touch the cache at all, and there is no cache-side way to bring that down.

The usable column is the third reading: buy eight GB and six GB gets used, the remaining quarter goes into the provider’s ledger. Since the working set is 4.272 GB, the node needed to hold all of it in memory is not four GB, it is eight.

When a Node Changes

A node change is a routine event — a version upgrade, a patch window, or a failed node being replaced — and the cache is not durable: a changed node’s memory comes back empty.

In a single-shard layout, all 3,370,000 keys are lost. Requests hitting the backend rise from 790 to 3,400, landing 325% over the purchased 800 requests/s capacity. This is not a crash, it is throttling: the data layer turns away the excess, and the dashboard comes back empty.

Shard count brings down the wave’s height: overage is 80% at four shards, 19% at sixteen. The number falls, but it never zeroes out at any shard count. The last row shows the one fix: add a replica to every shard and a node change turns into a failover, and lost keys drop to forty-nine — the keys written within the quarter-second replication lag. The cost is read off the first table: the node fee rises from 120 to 240, the total from 360 to 480.

The last column is this lesson’s irreversible item. The collector accumulates its minute-by-minute total in the cache and writes it out every five seconds; at any moment two thousand records exist only in memory, and when a node is lost in a four-shard layout, five hundred of them go with it. A cached copy can be brought back; a cached original cannot.

The warm-up row gives the wave’s length: half the access share comes back in five hundred thirty-nine seconds, 90% in 3,514. This time is independent of shard count: because the hash spreads key names, every shard draws the same share from every popularity tier, and one shard’s warm-up curve is the same as the whole’s. Splitting brings down the wave’s height, not its length. In a four-shard layout, the data layer carries above-purchased load for close to an hour; when that window opens is not your decision.

Where the Cache Sits

The third table’s first four rows meet the same read load under four layouts. The managed cache is both cheapest and fastest when it sits in the same zone: 360 units, 1.58 ms.

The in-process cache row shows what being a service brings. Each of twelve application instances allocates two hundred fifty-six MB — three GB in total — and the hit rate sits at 34.7%; the managed cache reaches 87% using one and a half GB. The difference is not the amount of memory, it is the singleness of the pool: twelve separate caches hold twelve copies of the same keys, and each one only warms up on the requests it personally sees.

The fourth row is this lesson’s transfer item. Putting the cache in another zone does not change the hit rate, and writes a new 722-unit line onto the bill: 87% of a monthly 1,036,800,000 reads, an average 800-byte response, one unit per GB. The cache node itself costs 120 units; putting it in the wrong zone costs six times the node and brings the total to 1,082 units — pricier than running no cache at all.

Where It Gains Nothing

The last three rows measure the zonal rate table: forty MB, three hundred twenty million reads a month, a set that fits entirely into every layout. The in-process cache costs 30 units and 0.02 ms; a managed cache in the same zone costs 90 units and 0.80 ms — three times the price, forty times slower. In another zone it is 730 units — twenty-four times.

The managed cache gains nothing here: since the set already fits in every process’s memory, a shared pool is unnecessary, and access over the network is pure cost. The rule is this: a managed cache pays off when the working set does not fit in a single process’s memory. The in-process layout’s cost is not in the table: twelve copies, twelve separate freshness windows.

The Delegated Decision

In a rented cluster, decisions split up as follows:

Group	Yours	Provider’s	Fixed at creation
version	engine version, patch window	when the patch becomes mandatory	—
maintenance	whether backups are on	change timing, node refresh, backup location	—
scaling	node size, shard count, replica count	usable memory ratio	shard count
data	eviction policy, key naming, time to live	data lost on a change	—
network	the cluster’s zone	which node a client lands on	the cluster’s zone

Ten of seventeen items are yours, seven are the provider’s, two are fixed at creation. Three of the maintenance group’s four items are delegated; the second table’s hour-long warm-up window ties back to this — you are not the one deciding when that window opens.

You choose the engine version, but the choice is one-way: an upgraded cluster does not go back to the old version. The item fixed in scaling is the shard count; change it and every key gets redistributed, with the hit rate dropping through the redistribution. That is why the first table’s “same hit rate, same price” result is misleading: the two levers land in the same place, but one is reversible and the other is not. The cluster’s zone is also fixed, and the 722-unit item ties directly back to it.

Summary

• Whichever path reaches two GB, the hit rate is 87% and the bill is 360 units: the scaling-unit decision cannot be read off hit rate and price.
• The bill’s curve turns: 1,020 units with no cache, 510 at one GB, 360 at two, 420 at four, 600 at eight. 100% hit rate is the most expensive row.
• A single-shard node change pushes backend-bound requests 325% over purchased capacity; at sixteen shards this falls to 19% but never zeroes out. Warm-up is 3,514 seconds under every layout. The irreversible item is the records pending in write-behind mode.
• Putting the cache in another zone writes a 722-unit transfer item and pushes the total above the no-cache layout. On the 40 MB rate table, the in-process cache costs 30 units and the managed cache 90: when the working set fits in a single process, the managed cache gains nothing.

Next Step

In this lesson, transfer was a single column with a single price: cross-zone, one unit per GB. That one column became the biggest item on the bill once the placement decision went wrong.

Across the rest of the measurement network, that same column gets written over and over: the collector writes to the data layer, the data layer replicates to the read replica, backups go out, logs stream, dashboard responses go out to the open network. Each flow prices differently depending on where its source and destination sit: the same zone, a neighboring zone, a distant region, or the internet.

The next lesson is this course’s lesson on money: the same architecture is run across three placements and the egress fee is worked out item by item, and then the effect of replica count, log and backup shipping, edge use, and compression decisions on that fee is counted.