--- title: 'Managed Cache Service' source: 'https://academia.sh/en/courses/cloud-storage/managed-cache-service' course: 'Storage and Data in the Cloud' language: en updated: '2026-09-13T05:07:36+00:00' license: 'CC BY-SA 4.0' objectives: - 'Shows that node size and shard count give the same hit rate and the same fee, and which table the difference actually opens up in' - 'Computes the keys lost on a node change, the wave that lands on the data layer, and the warm-up time, as a function of shard count' - 'Measures how the zone the cache sits in affects the transfer fee, and separates out the working set where the managed cache gains nothing' - 'Counts, item by item, how many of the version, maintenance, scaling, data, and network decisions are delegated' --- # Managed Cache Service Measuring the renting of the in-memory layer as a service: two scaling axes giving the same hit rate and the same bill, the 100% hit-rate row turning out the most expensive, the 3,370,000 keys lost when a node changes putting the purchased capacity 325% over, warm-up not shortening with shard count, and a cache in another zone writing a 722-unit transfer item and coming out pricier than no cache at all. The previous lesson widened the access pattern: the secondary index and the change stream reopened a share of the queries the partition key had closed off. But every reopened query still lands on the data layer. In-memory stores and eviction policies were measured earlier; this lesson does not repeat that mechanics. Here the in-memory layer is a **rented service**, and the three things measured are what the scaling unit is, what is lost when a node changes, and what the cache's location writes onto the bill. - **MD25.** The fictional regional measurement network's read set is three families: latest meter reading (2.4 million keys, 180 bytes), invoice summary (600 thousand, 2,400 bytes), work-order record (4 million, 600 bytes); 4.272 GB total. - **MD26.** Each family is modeled with three popularity tiers. The model fills the cache in order of per-key probability, does not count first-touch misses, and uses no randomness. - **MD27.** Reads average four hundred, peak three thousand requests/s; there is also a steady four hundred requests/s of writes, and writes never touch the cache. A month is counted as 2,592,000 seconds. - **MD28.** Prices are fictional, the currency's name is a fictional unit: a cache node runs sixty units per GB a month; read capacity is bought in hundred-requests/s steps, and a step is thirty units a month. The provider opens up **75%** of the node's memory for use. - **MD29.** Latencies are fictional: the data layer is 6 ms, a managed cache in the same zone 0.8 ms, in another zone 1.9 ms, in-process 0.02 ms. Cross-zone transfer is one unit per GB. - **MD30.** The meter collector accumulates its minute-by-minute totals in the cache and flushes every five seconds; the cache copy's replication lag is 0.25 seconds. Both are fictional. ```js // cache/cache.mjs — model: what changes when the in-memory layer is rented as a service. // Eviction policies were measured in M17/K06 and are not repeated. The working set is modeled // with popularity tiers, no randomness; prices are fictional and 1 GB is counted as 1e9 bytes. const TIER = [ // [family, tier, keys, bytes, access share] ["meter", "hot", 150000, 180, 0.15], ["meter", "warm", 850000, 180, 0.20], ["meter", "cold", 1400000, 180, 0.16], ["invoice", "hot", 18000, 2400, 0.10], ["invoice", "warm", 140000, 2400, 0.09], ["invoice", "cold", 442000, 2400, 0.04], ["workorder", "hot", 80000, 600, 0.06], ["workorder", "warm", 620000, 600, 0.10], ["workorder", "cold", 3300000, 600, 0.10], ]; const ORDER = [...TIER].sort((a, b) => b[4] / b[2] - a[4] / a[2]); // probability per key const fill = (capacity) => { // the cache fills from the most-requested key down let remaining = capacity, keys = 0, hitRate = 0; const retained = []; for (const [, , n, bytes, share] of ORDER) { const taken = Math.min(n, Math.floor(remaining / bytes)); if (taken === 0) break; remaining -= taken * bytes; keys += taken; hitRate += share * taken / n; retained.push([share * taken / n, share / n]); // [share retained, probability per key] } return { keys, hitRate, retained }; }; const PEAK = 3000, WRITES = 400, USABLE_SHARE = 0.75, GB_PRICE = 60; const capacityFee = (toBackend) => Math.ceil((toBackend + WRITES) / 100) * 30; const columns = (widths, left = 1) => (row) => row.map((d, i) => String(d)[i < left ? "padEnd" : "padStart"](widths[i])).join(""); const fmt1 = columns([14, 5, 16, 14, 11, 20, 10, 8, 15]); console.log(fmt1(["node x GB", "GB", "usable", "keys(k)", "hit(%)", "to backend(req/s)", "capacity", "node", "total(unit)"])); for (const [label, nodes, gbPerNode] of [["no cache", 0, 0], ["1 x 1", 1, 1], ["1 x 2", 1, 2], ["2 x 1", 2, 1], ["1 x 4", 1, 4], ["1 x 8", 1, 8]]) { const o = nodes ? fill(nodes * gbPerNode * 1e9 * USABLE_SHARE) : { keys: 0, hitRate: 0 }; const toBackend = Math.round(PEAK * (1 - o.hitRate)); const cap = capacityFee(toBackend), nodeFee = nodes * gbPerNode * GB_PRICE; console.log(fmt1([label, nodes * gbPerNode, (nodes * gbPerNode * USABLE_SHARE).toFixed(2), (o.keys / 1e3).toFixed(0), (o.hitRate * 100).toFixed(1), toBackend, cap, nodeFee, cap + nodeFee])); } // --- When a node changes ----------------------------------------------------------- // 2 GB layout. Since the hash spreads key names, every shard gets an equal share of every tier. const BASE = fill(2 * 1e9 * USABLE_SHARE), TO_BACKEND = Math.round(PEAK * (1 - BASE.hitRate)) + WRITES; const PURCHASED = capacityFee(TO_BACKEND - WRITES) / 30 * 100, PENDING = WRITES * 5, LAG = 0.25; const fmt2 = columns([16, 18, 16, 20, 12, 17]); console.log("\n" + fmt2(["shards", "lost keys", "lost access", "to backend(req/s)", "overage(%)", "pending records"])); for (const [shards, replicated] of [[1, 0], [2, 0], [4, 0], [8, 0], [16, 0], [4, 1]]) { const lostKeys = Math.round((replicated ? TO_BACKEND * LAG : BASE.keys) / shards); const lostAccess = replicated ? 0 : BASE.hitRate / shards; const toBackend = Math.round(PEAK * (1 - BASE.hitRate + lostAccess)) + WRITES; console.log(fmt2([replicated ? `${shards} (replicated)` : shards, lostKeys, (lostAccess * 100).toFixed(1), toBackend, ((toBackend / PURCHASED - 1) * 100).toFixed(0), Math.round((replicated ? WRITES * LAG : PENDING) / shards)])); } const recovered = (t) => BASE.retained.reduce((a, [share, r]) => a + share * (1 - Math.exp(-r * PEAK * t)), 0); const warmup = (target) => { let lo = 1, hi = 1e6; for (let k = 0; k < 60; k++) { const t = Math.sqrt(lo * hi); if (recovered(t) < target * BASE.hitRate) lo = t; else hi = t; } return hi.toFixed(0); }; console.log(`warm-up, independent of shard count: 50% ${warmup(0.5)} s, 90% ${warmup(0.9)} s`); // --- Where the cache sits ----------------------------------------------------------- const DATA_MS = 6, ZONE_PRICE = 1, MONTH = 2592000; // ms; cross-zone unit/GB; seconds/month const AVG_BYTES = TIER.reduce((a, [, , , bytes, share]) => a + bytes * share, 0); const fmt3 = columns([12, 26, 11, 14, 11, 11, 10, 15], 2); const row = (set, label, h, ms, reads, bytes, peak, writes, cacheFee, remote) => { const toBackend = Math.round(peak * (1 - h)); const capacity = Math.max(1, Math.ceil((toBackend + writes) / 100)) * 30; const transfer = (remote ? h * reads * bytes / 1e9 : 0) * ZONE_PRICE; console.log(fmt3([set, label, (h * 100).toFixed(1), (ms + (1 - h) * DATA_MS).toFixed(2), cacheFee, capacity, transfer.toFixed(0), (cacheFee + capacity + transfer).toFixed(0)])); }; console.log("\n" + fmt3(["set", "layout", "hit(%)", "latency(ms)", "cache", "capacity", "transfer", "total(unit)"])); const FULLSET = (label, h, ms, cacheFee, remote) => row("full set", label, h, ms, 400 * MONTH, AVG_BYTES, PEAK, WRITES, cacheFee, remote); FULLSET("no cache", 0, 0, 0, false); FULLSET("in-process 12 x 256 MB", fill(256e6).hitRate, 0.02, 0, false); FULLSET("managed 2 GB same zone", BASE.hitRate, 0.8, 2 * GB_PRICE, false); FULLSET("managed 2 GB other zone", BASE.hitRate, 1.9, 2 * GB_PRICE, true); // Rate table: 20 thousand keys x 2000 bytes = 40 MB, 320 million reads a month, peak 900/s. const RATETABLE = (label, h, ms, cacheFee, remote) => row("rate table", label, h, ms, 320e6, 2000, 900, 0, cacheFee, remote); RATETABLE("in-process 12 x 40 MB", 1, 0.02, 0, false); RATETABLE("managed 1 GB same zone", 1, 0.8, GB_PRICE, false); RATETABLE("managed 1 GB other zone", 1, 1.9, GB_PRICE, true); ``` ``` node x GB GB usable keys(k) hit(%) to backend(req/s) capacity node total(unit) no cache 0 0.00 0 0.0 3000 1020 0 1020 1 x 1 1 0.75 1476 63.8 1085 450 60 510 1 x 2 2 1.50 3370 87.0 390 240 120 360 2 x 1 2 1.50 3370 87.0 390 240 120 360 1 x 4 4 3.00 4880 93.6 193 180 240 420 1 x 8 8 6.00 7000 100.0 0 120 480 600 shards lost keys lost access to backend(req/s) overage(%) pending records 1 3370000 87.0 3400 325 2000 2 1685000 43.5 2095 162 1000 4 842500 21.8 1442 80 500 8 421250 10.9 1116 40 250 16 210625 5.4 953 19 125 4 (replicated) 49 0.0 790 -1 25 warm-up, independent of shard count: 50% 539 s, 90% 3514 s set layout hit(%) latency(ms) cache capacity transfer total(unit) full set no cache 0.0 6.00 0 1020 0 1020 full set in-process 12 x 256 MB 34.7 3.94 0 720 0 720 full set managed 2 GB same zone 87.0 1.58 120 240 0 360 full set managed 2 GB other zone 87.0 2.68 120 240 722 1082 rate table in-process 12 x 40 MB 100.0 0.02 0 30 0 30 rate table managed 1 GB same zone 100.0 0.80 60 30 0 90 rate table managed 1 GB other zone 100.0 1.90 60 30 640 730 ``` ## Two Axes, One Curve The managed cache service offers two scaling levers: growing the node and adding nodes. The first table puts them side by side, and the first reading is this: **no difference.** Two GB is reached with one node just as well as with two; both ways the cache holds 3,370,000 keys, the hit rate is 87%, the bill is 360 units. The scaling-unit decision **cannot be made on hit rate and price alone.** The bill itself is the second reading. The no-cache layout runs 1,020 units; one GB lifts the hit rate to 63.8% and drops the bill to 510, two GB to 360. Then the curve **turns**: 420 at four GB, 600 at eight GB. The 100%-hit-rate row is **67% pricier** than the two-GB row; the remaining 120 units goes to the four hundred requests/s of writes that never touch the cache at all, and there is no cache-side way to bring that down. The usable column is the third reading: buy eight GB and six GB gets used, the remaining quarter goes into the provider's ledger. Since the working set is 4.272 GB, the node needed to hold all of it in memory is not four GB, it is eight. ## When a Node Changes A node change is a routine event — a version upgrade, a patch window, or a failed node being replaced — and the cache is not durable: a changed node's memory comes back empty. In a single-shard layout, **all** 3,370,000 keys are lost. Requests hitting the backend rise from 790 to 3,400, landing **325% over** the purchased 800 requests/s capacity. This is not a crash, it is throttling: the data layer turns away the excess, and the dashboard comes back empty. Shard count brings down the wave's **height**: overage is 80% at four shards, 19% at sixteen. The number falls, but **it never zeroes out at any shard count.** The last row shows the one fix: add a replica to every shard and a node change turns into a failover, and lost keys drop to forty-nine — the keys written within the quarter-second replication lag. The cost is read off the first table: the node fee rises from 120 to 240, the total from 360 to 480. The last column is this lesson's **irreversible item**. The collector accumulates its minute-by-minute total in the cache and writes it out every five seconds; at any moment two thousand records exist only in memory, and when a node is lost in a four-shard layout, five hundred of them go with it. A cached copy can be brought back; a cached **original** cannot. The warm-up row gives the wave's length: half the access share comes back in five hundred thirty-nine seconds, 90% in 3,514. This time is **independent of shard count**: because the hash spreads key names, every shard draws the same share from every popularity tier, and one shard's warm-up curve is the same as the whole's. **Splitting brings down the wave's height, not its length.** In a four-shard layout, the data layer carries above-purchased load for close to an hour; when that window opens is not your decision. ## Where the Cache Sits The third table's first four rows meet the same read load under four layouts. The managed cache is both cheapest and fastest when it sits in the same zone: 360 units, 1.58 ms. The in-process cache row shows what being a service brings. Each of twelve application instances allocates two hundred fifty-six MB — three GB in total — and the hit rate sits at 34.7%; the managed cache reaches 87% using one and a half GB. The difference is not the amount of memory, it is the **singleness of the pool**: twelve separate caches hold twelve copies of the same keys, and each one only warms up on the requests it personally sees. The fourth row is this lesson's transfer item. Putting the cache in another zone does not change the hit rate, and writes a new **722-unit** line onto the bill: 87% of a monthly 1,036,800,000 reads, an average 800-byte response, one unit per GB. The cache node itself costs 120 units; putting it in the wrong zone costs **six times the node** and brings the total to 1,082 units — **pricier than running no cache at all**. ## Where It Gains Nothing The last three rows measure the zonal rate table: forty MB, three hundred twenty million reads a month, a set that fits entirely into every layout. The in-process cache costs 30 units and 0.02 ms; a managed cache in the same zone costs 90 units and 0.80 ms — three times the price, forty times slower. In another zone it is 730 units — **twenty-four times**. The managed cache gains nothing here: since the set already fits in every process's memory, a shared pool is unnecessary, and access over the network is pure cost. The rule is this: **a managed cache pays off when the working set does not fit in a single process's memory.** The in-process layout's cost is not in the table: twelve copies, twelve separate freshness windows. ## The Delegated Decision In a rented cluster, decisions split up as follows: | Group | Yours | Provider's | Fixed at creation | |---|---|---|---| | version | engine version, patch window | when the patch becomes mandatory | — | | maintenance | whether backups are on | change timing, node refresh, backup location | — | | scaling | node size, shard count, replica count | usable memory ratio | shard count | | data | eviction policy, key naming, time to live | data lost on a change | — | | network | the cluster's zone | which node a client lands on | the cluster's zone | **Ten of seventeen items are yours**, seven are the provider's, two are fixed at creation. Three of the maintenance group's four items are delegated; the second table's hour-long warm-up window ties back to this — you are not the one deciding **when that window opens**. You choose the engine version, but the choice is one-way: an upgraded cluster does not go back to the old version. The item fixed in scaling is the **shard count**; change it and every key gets redistributed, with the hit rate dropping through the redistribution. That is why the first table's "same hit rate, same price" result is misleading: the two levers land in the same place, but one is reversible and the other is not. The cluster's zone is also fixed, and the 722-unit item ties directly back to it. ## Summary - Whichever path reaches two GB, the hit rate is 87% and the bill is 360 units: the scaling-unit decision cannot be read off hit rate and price. - The bill's curve turns: 1,020 units with no cache, 510 at one GB, 360 at two, 420 at four, 600 at eight. 100% hit rate is the most expensive row. - A single-shard node change pushes backend-bound requests 325% over purchased capacity; at sixteen shards this falls to 19% but never zeroes out. Warm-up is 3,514 seconds under every layout. The irreversible item is the records pending in write-behind mode. - Putting the cache in another zone writes a 722-unit transfer item and pushes the total above the no-cache layout. On the 40 MB rate table, the in-process cache costs 30 units and the managed cache 90: when the working set fits in a single process, the managed cache gains nothing. ## Next Step In this lesson, transfer was a single column with a single price: cross-zone, one unit per GB. That one column became the biggest item on the bill once the placement decision went wrong. Across the rest of the measurement network, that same column gets written over and over: the collector writes to the data layer, the data layer replicates to the read replica, backups go out, logs stream, dashboard responses go out to the open network. Each flow prices differently depending on where its source and destination **sit**: the same zone, a neighboring zone, a distant region, or the internet. The next lesson is this course's lesson on money: the same architecture is run across three placements and the egress fee is worked out item by item, and then the effect of replica count, log and backup shipping, edge use, and compression decisions on that fee is counted.