--- title: 'Edge Storage' source: 'https://academia.sh/en/courses/serverless/edge-storage' course: 'Serverless and Edge Computing' language: en updated: '2026-09-13T05:07:39+00:00' license: 'CC BY-SA 4.0' objectives: - "Measures the edge key-value store's read and write paths separately: extra round trips and latency" - 'Scans the propagation window to compute the count and share of stale reads for each value' - "Compares three access paths to the object store on round trips and egress cost, and counts the store's value size limit" --- # Edge Storage The edge key-value store is fast for reads, slow for writes, and its consistency is delayed. The propagation window is scanned and stale reads are counted for each value; the round-trip count and egress cost of reaching the object store are compared in a fictional unit, and the store's size limit is worked out as a constraint. All four jobs in the previous lesson were stateless: routing looked at a table, authentication at a signature, the A/B split at a split rule. What the three jobs that could not run at the edge had in common was that they wanted state. In the fictional **measurement network**, what branch sites need often is well known: a branch's tariff table, its configuration, a session's validity summary. These are not large, and they are mostly read. A store exists at the edge, and its name is the **edge key-value store**. This lesson does not weigh it with a single metric, because a single number would mislead: the store in this model is faster for reads than anything in the region, and slower for writes than anything in the region — real edge stores do not all make that trade the same way, and some offer strongly consistent writes. The difference between the two is not a latency item — it is a **consistency window**. **EC13.** Writes are applied at the center; the site that wrote sees the new value immediately, other sites return the old value for the length of the propagation window. **EC14.** The window is fixed, and eventually every site sees the new value; partial propagation is not modeled. **EC15.** The event stream is produced by our own generator, and the seed is visible. ## Reads Are Cheap, Writes Are Expensive The length of the read path depends on whether the value sits at the edge. If the value sits at the edge, the read never leaves the network at all. The length of the write path never changes: a write always goes to the center, because that is the single point that determines ordering. ```js // measurement-network/propagation.mjs — read/write asymmetry in the edge key-value store // and the count of stale reads across the propagation window. // MODEL: the event stream is generated by our own generator; the seed is visible. const SEED = 20_260_801; let s = SEED % 2_147_483_647; const rnd = () => (s = (s * 48_271) % 2_147_483_647) / 2_147_483_647; const SITES = 6, KEYS = 40, EVENT_COUNT = 20_000, WRITE_PERCENT = 8; const PATH = { "read (copy at edge)": { trip: 0, ms: 2 }, "read (no copy)": { trip: 1, ms: 62 }, "write (to center)": { trip: 1, ms: 64 }, }; console.log("read and write are not the same thing in this store (model input)"); for (const [ad, v] of Object.entries(PATH)) console.log(" " + ad.padEnd(24) + `${v.trip} extra trip ${String(v.ms).padStart(3)} ms`); console.log(" write/read latency ratio: " + (PATH["write (to center)"].ms / PATH["read (copy at edge)"].ms).toFixed(0) + "x"); const events = []; let t = 0; for (let i = 0; i < EVENT_COUNT; i++) { t += 1 + Math.floor(rnd() * 60); const site = Math.floor(rnd() * SITES), key = Math.floor(rnd() * KEYS); events.push({ t, site, key, write: rnd() * 100 < WRITE_PERCENT }); } const writes = events.filter((o) => o.write).length; console.log(`\n${EVENT_COUNT} events, ${SITES} sites, ${KEYS} keys, seed ${SEED}`); console.log(` writes ${writes} reads ${EVENT_COUNT - writes} stream duration ${(t / 1000).toFixed(1)} s`); console.log("\nstale reads across the scanned propagation window"); console.log("window".padEnd(12) + "reads in window".padEnd(20) + "stale reads".padEnd(14) + "read share".padEnd(14) + "per key"); for (const window of [1_000, 5_000, 10_000, 30_000, 60_000]) { const last = new Array(KEYS).fill(null); let inWindow = 0, stale = 0, reads = 0; for (const o of events) { if (o.write) { last[o.key] = o; continue; } reads++; const y = last[o.key]; if (!y || o.t - y.t >= window) continue; inWindow++; if (y.site !== o.site) stale++; } console.log(`${window} ms`.padEnd(12) + String(inWindow).padEnd(20) + String(stale).padEnd(14) + `${((stale / reads) * 100).toFixed(2)}%`.padEnd(14) + (stale / KEYS).toFixed(1)); } console.log(" a 0 ms window would give 0 stale reads; the cost is every read going to the center"); ``` ``` read and write are not the same thing in this store (model input) read (copy at edge) 0 extra trip 2 ms read (no copy) 1 extra trip 62 ms write (to center) 1 extra trip 64 ms write/read latency ratio: 32x 20000 events, 6 sites, 40 keys, seed 20260801 writes 1582 reads 18418 stream duration 609.8 s stale reads across the scanned propagation window window reads in window stale reads read share per key 1000 ms 1154 957 5.20% 23.9 5000 ms 5019 4147 22.52% 103.7 10000 ms 8690 7230 39.26% 180.8 30000 ms 15466 12857 69.81% 321.4 60000 ms 17572 14621 79.38% 365.5 a 0 ms window would give 0 stale reads; the cost is every read going to the center ``` The asymmetry is read in the first three rows. Reading from the copy at the edge costs 0 extra trips and 2 ms, reading with no copy costs 1 trip and 62 ms, writing costs 1 trip and **64 ms** in every case. The ratio of write to read is **32-fold**. This ratio says what the store is built for: if a value is written once and read many times, the store sits in the right place; if it is written once per request, the store sits in the wrong place. ## The Count of Stale Reads Consistency being delayed is not an adjective; it is measured by how many reads see the old value. The scan below holds the window at five values and counts stale reads at each one. Being counted as stale requires two conditions: the read must arrive after the last write, before the window closes, and the reading site must differ from the writing site. ```js // measurement-network/object.mjs — trips and egress cost of reaching the object store from // the edge, then the edge store's size limit. MODEL: rates are a fictional unit, // not a real currency or price. const REQUESTS = 10_000, OBJECT_KB = 250, GB = 1_048_576; // 1 GB in KB const RATE = { "object->region": 0, "region->edge": 0, "object->edge": 2, "edge->client": 1 }; const PATH = { "via region": { trip: 2, ms: 80, legs: ["object->region", "region->edge", "edge->client"] }, "direct from edge": { trip: 1, ms: 45, legs: ["object->edge", "edge->client"] }, "copy at edge": { trip: 0, ms: 2, legs: ["edge->client"] }, }; const volumeGB = (n) => (n * OBJECT_KB) / GB; const cost = (path, n) => PATH[path].legs.reduce((t, b) => t + RATE[b], 0) * volumeGB(n); console.log(`${REQUESTS} object requests, object ${OBJECT_KB} KB, rates are a fictional unit`); console.log("path".padEnd(18) + "trips".padEnd(10) + "latency".padEnd(10) + "legs".padEnd(14) + "units/GB".padEnd(11) + "set cost"); for (const [ad, v] of Object.entries(PATH)) { const unitsGB = v.legs.reduce((t, b) => t + RATE[b], 0); console.log(ad.padEnd(18) + String(v.trip).padEnd(10) + `${v.ms} ms`.padEnd(10) + String(v.legs.length).padEnd(14) + String(unitsGB).padEnd(11) + `${cost(ad, REQUESTS).toFixed(2)} units`); } console.log(`set volume ${volumeGB(REQUESTS).toFixed(2)} GB ` + `leg rate: ` + Object.entries(RATE).map(([b, u]) => `${b} ${u}`).join(", ")); const HIT_RATE = 70; const hits = Math.round((REQUESTS * HIT_RATE) / 100), misses = REQUESTS - hits; const blendedMs = (hits * PATH["copy at edge"].ms + misses * PATH["via region"].ms) / REQUESTS; const blendedCost = cost("copy at edge", hits) + cost("via region", misses); console.log(`\nblend: ${HIT_RATE}% copy at edge, the rest via region`); console.log(" " + `${hits} + ${misses} requests`.padEnd(22) + `${blendedMs.toFixed(1)} ms/req ${blendedCost.toFixed(2)} units ` + `${(cost("direct from edge", REQUESTS) / blendedCost).toFixed(2)}x cheaper than direct`); // The edge key-value store's limit: 512-byte keys, 25 KB values. const VALUE_KB = 25, KEY_BYTES = 512; const RECORD = { "branch configuration": 2, "tariff table": 18, "session summary": 0.4, "daily reading summary": 140, "monthly invoice document": 900, "raw counter log": 12_000, }; console.log(`\nedge store limit: value ${VALUE_KB} KB, key ${KEY_BYTES} bytes`); console.log("record".padEnd(26) + "size".padEnd(12) + "fits".padEnd(9) + "workaround"); let fits = 0; for (const [ad, kb] of Object.entries(RECORD)) { const ok = kb <= VALUE_KB; if (ok) fits++; const chunks = Math.ceil(kb / VALUE_KB); console.log(ad.padEnd(26) + `${kb} KB`.padEnd(12) + (ok ? "yes" : "no").padEnd(9) + (ok ? "-" : `${chunks} chunks, +${chunks - 1} reads per read or the object store (+2 trips)`)); } console.log(` ${fits}/${Object.keys(RECORD).length} records fit in the store`); ``` ``` 10000 object requests, object 250 KB, rates are a fictional unit path trips latency legs units/GB set cost via region 2 80 ms 3 1 2.38 units direct from edge 1 45 ms 2 3 7.15 units copy at edge 0 2 ms 1 1 2.38 units set volume 2.38 GB leg rate: object->region 0, region->edge 0, object->edge 2, edge->client 1 blend: 70% copy at edge, the rest via region 7000 + 3000 requests 25.4 ms/req 2.38 units 3.00x cheaper than direct edge store limit: value 25 KB, key 512 bytes record size fits workaround branch configuration 2 KB yes - tariff table 18 KB yes - session summary 0.4 KB yes - daily reading summary 140 KB no 6 chunks, +5 reads per read or the object store (+2 trips) monthly invoice document 900 KB no 36 chunks, +35 reads per read or the object store (+2 trips) raw counter log 12000 KB no 480 chunks, +479 reads per read or the object store (+2 trips) 3/6 records fit in the store ``` The stream itself is also an input, and it needs to be read: of 20,000 events, 1,582 are writes and 18,418 are reads, spread across **609.8 seconds**. So the mix matches the direction the store was designed for — one write for every eleven reads. The stale-read count coming out this high is therefore not because writes are frequent, but because forty keys are spread across six sites: every write ties the reads at the other five sites to the old value for the length of the window. The numbers grow in one direction and grow fast. At a 1,000 ms window, stale reads are 957, or 5.20% of reads. At 5,000 ms, 4,147 and 22.52%. At 10,000 ms, 7,230 and 39.26%. At 30,000 ms, 12,857 and 69.81%. At 60,000 ms, 14,621 and 79.38%. As the window grows sixtyfold, stale reads grow fifteenfold; the growth is not linear, because as the window widens, most reads already fall in the shadow of some write, and the count approaches saturation. Stale reads per key climb from 23.9 to 365.5. If each of the forty keys is treated as one branch's tariff table in the fiction, then at a 30,000 ms window, 321.4 reads per key see the old tariff. Whether that number is acceptable depends on what the read is for: tolerable if it is read to display a tariff on a screen, not tolerable if it is read to write an amount onto an invoice. The same store, with the same window, gives two different answers for two different jobs. The decision handed over here is replication: the application does not choose which sites get a copy of the value, or when. What comes back in exchange is the propagation window, and it is a number. How to work around the constraint is written in the last line: if the window is brought to zero, stale reads become zero, but then every read goes to the center — reading climbs from 2 ms to 62 ms, a **31-fold** rise. Since the whole reason for using the store is that 2 ms, this workaround cancels the store itself. ## Reaching Out to the Object Store For things that do not fit in the store, the edge reaches out to the object store. Reaching out has two axes, and they work in opposite directions: round-trip count and egress cost. **EC16.** Rates are a **fictional unit**; they are not a real currency or price. **EC17.** Object size is fixed at 250 KB, and the hit rate of the copy kept at the edge is a 70% input. **EC18.** In this model the value limit is 25 KB and the key limit is 512 bytes — in real services these two limits can differ widely; chunking assumes the chunks are read in sequence. The relationship among the three paths is the reverse of what is expected at first glance. Going directly from the edge to the object store is one of the paths with the fewest trips — 1 trip, 45 ms — but the set cost is **7.15 units**, three times going via the region. The reason is the egress leg: on the direct path, the object leaves the store's network and reaches the edge, and this leg charges 2 units per GB. Going via the region keeps the object inside the same network; the only leg charged is the one from edge to client. Cutting the trip count here **triples** the cost. Keeping a copy at the edge wins on both axes: 0 trips, 2 ms, and 2.38 units. But this row comes with an assumption — that the copy is actually there. The blended row gives the real situation: at 70% hit rate, an average of **25.4 ms/request** and 2.38 units; **3.00 times cheaper** than the direct path. As the hit rate falls, latency drifts toward 80 ms, while the cost does not change. So what keeping a copy at the edge buys is not cost, it is latency; the only thing that determines cost is which legs the traffic passes through. ## What Fits in the Store The last table counts the store's limit as a constraint. The value limit is 25 KB, and 3 of the fiction's six record types fit: branch configuration at 2 KB, the tariff table at 18 KB, the session summary at 0.4 KB. Three do not fit: the daily reading summary at 140 KB, the monthly invoice document at 900 KB, the raw counter log at 12,000 KB. The way to force a value that does not fit into the store is chunking, and the cost is collected on the read side. The 140 KB summary comes to 6 chunks; every full read means **+5 reads**. The invoice document is 36 chunks and +35 reads, the raw log is 480 chunks and +479 reads. Chunking is reasonable for a small overshoot and absurd for a large one: for a value four hundred and eighty times the 25 KB limit, the store is not the right tool. The alternative is written in the last line — put it in the object store and pay +2 trips. The decision point is how many chunks a single read requires; the moment the chunk count passes two or three, the object store's two trips become cheaper. The 512-byte key limit is a separate design constraint, and it does not show up in the table. A short key forces a composite key — the branch, counter, and period triple — to be squeezed into a single string; combined with the store's inability to run range queries, this forces data into the design of the key itself. The store is not a database; it is for small, heavily read values reached by key. ## Summary - The same store is two different things in two directions: reading from the copy at the edge is 0 trips / 2 ms, writing is 1 trip / 64 ms. The ratio of write to read is 32-fold; the store is for values written once and read many times. - Consistency is delayed, and the cost is counted: at 1,000 / 5,000 / 10,000 / 30,000 / 60,000 ms windows, stale reads are 957, 4,147, 7,230, 12,857, and 14,621; the read share climbs from 5.20% to 79.38%, and per key from 23.9 to 365.5. - Zeroing out the window zeroes out stale reads but moves reading from 2 ms to 62 ms — canceling the store's whole reason for being used. - There are three paths to the object store: via region 2 trips / 2.38 units, direct from edge 1 trip / 7.15 units, copy at edge 0 trips / 2.38 units. Cutting trips can triple the cost; the 70%-hit blend costs 25.4 ms/request and 2.38 units. - The value limit is 25 KB, and 3 of six record types fit. The ones that do not come to 6, 36, and 480 chunks; once the chunk count passes a few, the object store's +2 trips become cheaper. ## Next Step This lesson's store never solved one thing: what happens when two edge sites want to change the same key at the same time? The propagation window was making reads return the old value; if two writes race, one overwrites the other, and an update goes missing somewhere in between. Incrementing a counter, counting who has joined a room, locking a resource to a single operation — all three need a "read, modify, write" step, and none of the three is safe in this store. The way to keep coordinated state at the edge is for a single writer to exist for a given key. The next lesson runs a single-writer object model, counts lost updates under concurrent requests, and measures the two constraints that come back in exchange — the single instance's bottleneck, and the object's geographic pinning.