--- title: 'Persistent and Interruptible Capacity' source: 'https://academia.sh/en/courses/compute-and-networking-in-the-cloud/persistent-and-interruptible-capacity' course: 'Compute and Networking in the Cloud' language: en updated: '2026-09-13T05:07:37+00:00' license: 'CC BY-SA 4.0' objectives: - 'Compares how four job types respond to interruption notice with a single measure: job-minutes lost per interruption' - 'Computes interruption count, lost job-hours, restart load, and completion rate over a thirty-day period' - 'Names the job type that does not tolerate interruption at all through measurement, and shows the reason via recovery time' - 'Scans the mix of persistent with interruptible capacity, and names the three items that cannot be undone' --- # Persistent and Interruptible Capacity Four job types are run on interruptible capacity, and interruption notice is scanned from zero to three hundred seconds to count the work lost at each type; over a thirty-day period, interruption count, lost job-hours, restart load, and completion rate are worked out; the job type that does not tolerate interruption at all is named, and the mix of persistent with interruptible is scanned. The previous two lessons rented the machine and started it; both rested on the same assumption — the instance stays up for as long as it is wanted. There is also a form of renting that removes this assumption: **interruptible capacity** is capacity the provider can take back with a short notice when it is needed elsewhere, and in exchange it is given noticeably cheaply. M23/K01 measured the price difference between these two forms; that comparison is not repeated here, it is **taken as input**. This lesson's question is not price but **job type**: when the same discount is given to four different jobs, the four do not all gain the same thing, because when interruption arrives the four do not all lose the same thing. The measure is not the interruption itself but **what the interruption takes away**. **CC13.** Four job types are the jobs of the regional measurement network — the fictional software that collects readings from water meters; their durations and instance counts are fictional. **CC14.** The interruption rate is 0.06 interruptions per instance-hour; the generator is self-written and the seed 20260901 is visible. **CC15.** Notice duration is scanned between 0 and 300 seconds; each job type's recovery time is fictional. **CC16.** Persistent capacity is 1.00, interruptible capacity is 0.32, both in **fictional units**; the ratio comes as input from M23/K01, not a real currency. **CC17.** The long run keeps no checkpoint; when interrupted it starts from scratch. **CC18.** The period is 30 days. ## Which Job Loses What Interruption notice is not a news item, it is a window: the time left before capacity is taken back. The only thing that can be done within that window is recovering the job, and each job type's recovery operation takes a different amount of time. If the window is shorter than the recovery time, the notice is of no use at all. ```js // measurement-network/job-types.mjs — what four job types lose on interruption (model; not a product) export const generator = (seed) => () => (seed = (seed * 1103515245 + 12345) % 2147483648) / 2147483648; export const NOTICE = [0, 30, 60, 120, 300]; // interruption notice scanned (s) // recover: seconds needed to save the job within the notice window (loss occurs if not enough). // loss: job lost in one interruption if it cannot be recovered (minutes). restart: restart cost (minutes). export const JOBS = [ { name: "nightly batch job", instances: 12, hours: 6, recover: 45, loss: 15, restart: 4, note: "15 min elapsed since checkpoint" }, { name: "queue worker", instances: 20, hours: 24, recover: 90, loss: 0.75, restart: 3, note: "half task, returns to queue" }, { name: "long single run", instances: 4, hours: 20, recover: 660, loss: 600, restart: 6, note: "half the run, 10 h" }, { name: "user-facing service", instances: 30, hours: 24, recover: 20, loss: 0.08, restart: 2, note: "24 requests in flight" }, ]; export const loss = (job, n) => (n >= job.recover ? 0 : job.loss); if (import.meta.url === `file://${process.argv[1]}`) { console.log("job type".padEnd(32) + "recovery".padEnd(11) + NOTICE.map((n) => `${n} s`.padEnd(9)).join("") + "lost per interruption"); for (const job of JOBS) console.log(job.name.padEnd(32) + `${job.recover} s`.padEnd(11) + NOTICE.map((n) => `${loss(job, n).toFixed(2)}`.padEnd(9)).join("") + job.note); console.log("\n(cells are job-minutes lost per interruption)"); const neverRecovers = JOBS.filter((job) => NOTICE.every((n) => loss(job, n) > 0)); console.log(`job whose loss never zeroes out across the notice scan: ` + neverRecovers.map((x) => x.name).join(", ")); console.log(`reason: recovery needs ${neverRecovers[0].recover} s, the scan's longest ` + `notice is ${Math.max(...NOTICE)} s`); } ``` ``` job type recovery 0 s 30 s 60 s 120 s 300 s lost per interruption nightly batch job 45 s 15.00 15.00 0.00 0.00 0.00 15 min elapsed since checkpoint queue worker 90 s 0.75 0.75 0.75 0.00 0.00 half task, returns to queue long single run 660 s 600.00 600.00 600.00 600.00 600.00 half the run, 10 h user-facing service 20 s 0.08 0.00 0.00 0.00 0.00 24 requests in flight (cells are job-minutes lost per interruption) job whose loss never zeroes out across the notice scan: long single run reason: recovery needs 660 s, the scan's longest notice is 300 s ``` In the table, three rows zero out at a threshold, one row never zeros out. The user-facing service drains in 20 seconds and its loss ends at a 30-second notice; the nightly batch job takes an emergency checkpoint in 45 seconds and recovers at 60 seconds; the queue worker loses nothing if it can finish the 90-second task in hand. In these three, what makes interruption tolerable is not the type of capacity but **whether the job is divisible**: if a job is split into small pieces, no more than one piece is lost. The fourth row shows what an undivided job looks like. The 20-hour run with no checkpoint loses all progress up to that moment when interruption arrives, and recovering that requires a 660-second checkpoint. The scan's longest notice is 300 seconds; **no matter how long the notice gets, the loss does not change**. This is the job that does not tolerate interruption at all, and its name is `long single run`. The fix is not a longer notice — it is splitting the job itself, that is, adding a checkpoint; and that is an application decision, not a capacity decision. ## The Thirty-Day Period Loss per interruption is a unit price; the real question is how many times a month that unit is paid. The run below produces a 30-day period with the same generator and compares two notice levels — zero and 120 seconds — over the same interruption sequence. ```js // measurement-network/interruption-period.mjs — a 30-day interruptible capacity run (model) import { generator, JOBS, loss } from "./job-types.mjs"; export const DAYS = 30, RATE = 0.06; // interruptions/instance-hour (fictional) const r = generator(20260901); // seed visible export const run = (job) => { const hours = job.instances * job.hours * DAYS; let interruptions = 0; for (let i = 0; i < hours; i++) if (r() < RATE) interruptions++; const restart = interruptions * job.restart; const measure = (n) => (60 * hours) / (60 * hours + interruptions * loss(job, n) + restart); return { hours, interruptions, restart, atZero: interruptions * loss(job, 0), at120: interruptions * loss(job, 120), yieldZero: measure(0), yield120: measure(120) }; }; // The 20-hour run with no checkpoint: starts over from scratch on every attempt. export const singleRun = (job) => { const TOTAL = DAYS * 24; let elapsed = 0, attempts = 0, completed = 0, wasted = 0; while (elapsed < TOTAL) { attempts++; let h = 0; while (h < job.hours && elapsed < TOTAL) { h++; elapsed++; if (r() < RATE) break; } if (h === job.hours) completed++; else { wasted += h; elapsed += job.restart / 60; } } return { attempts, completed, wasted, persistentCompletions: Math.floor(TOTAL / job.hours) }; }; if (import.meta.url === `file://${process.argv[1]}`) { console.log(`${DAYS} days, interruption rate ${RATE}/instance-hour, seed 20260901\n`); console.log("job type".padEnd(32) + "instance-hours".padEnd(16) + "interruptions".padEnd(15) + "lost h (0 s)".padEnd(17) + "lost h (120 s)".padEnd(19) + "restart h".padEnd(12) + "yield 0 s".padEnd(12) + "yield 120 s"); const s = {}; for (const job of JOBS) { const k = (s[job.name] = run(job)); console.log(job.name.padEnd(32) + String(k.hours).padEnd(16) + String(k.interruptions).padEnd(15) + (k.atZero / 60).toFixed(1).padEnd(17) + (k.at120 / 60).toFixed(1).padEnd(19) + (k.restart / 60).toFixed(1).padEnd(12) + `${(100 * k.yieldZero).toFixed(1)}%`.padEnd(12) + `${(100 * k.yield120).toFixed(1)}%`); } const longJob = JOBS.find((x) => x.name === "long single run"); const t = singleRun(longJob); console.log(`\n${longJob.hours}-hour run with no checkpoint: ${t.attempts} attempts, ` + `${t.completed} completions; ${t.persistentCompletions} completions on persistent capacity`); console.log(`run hours wasted: ${t.wasted}; completion rate ` + `${(100 * t.completed / t.persistentCompletions).toFixed(0)}%`); const service = s["user-facing service"], queue = s["queue worker"]; console.log(`at 0 s notice: service drops ${service.interruptions * 24} requests, ` + `queue puts ${queue.interruptions} tasks back in the queue`); } ``` ``` 30 days, interruption rate 0.06/instance-hour, seed 20260901 job type instance-hours interruptions lost h (0 s) lost h (120 s) restart h yield 0 s yield 120 s nightly batch job 2160 137 34.3 0.0 9.1 98.0% 99.6% queue worker 14400 964 12.1 0.0 48.2 99.6% 99.7% long single run 2400 164 1640.0 1640.0 16.4 59.2% 59.2% user-facing service 21600 1450 1.9 0.0 48.3 99.8% 99.8% 20-hour run with no checkpoint: 66 attempts, 15 completions; 36 completions on persistent capacity run hours wasted: 415; completion rate 42% at 0 s notice: service drops 34800 requests, queue puts 964 tasks back in the queue ``` The value of notice accumulates in a different place depending on job type. In the nightly batch job, a 120-second notice saves 34.3 job-hours a month — 1.6 percent of a month's total work. In the service row, the saved work is only 1.9 hours, but inside that 1.9 hours are **34,800 requests**, and each one is a response a user was waiting for. The same unit of measure means a different thing in the two job types: in the batch job, what is lost is time; in the service, what is lost is users. The restart column is a cost that notice can never save. The service takes 1,450 interruptions a month, and each one demands a two-minute startup; the total is 48.3 hours — twenty-five times the 1.9 hours saved. On interruptible capacity, startup time is not a one-time cost; the previous lesson's 48.9 seconds is an item paid 1,450 times a month here. The third row must be read twice. The rough calculation counts average loss as half the run and finds a yield of 59.2 percent. Run attempt by attempt, the real figure comes out to **42 percent**: only 15 of 66 attempts complete, 415 run-hours go to waste, and work that finishes 36 times a month on persistent capacity finishes 15 times on interruptible. The average-loss assumption misleads here, because the loss is not half the run but **the entire time elapsed until interruption**, and in a long run that time is larger than expected. ## Cost and Mix Once loss is counted, the price difference becomes meaningful. Interruptible capacity is cheap, but the lost work gets redone, and redone work also takes hours. The comparison is therefore made not on unit price but on **the hours it takes to finish the same job**. ```js // measurement-network/capacity-cost.mjs — persistent versus interruptible capacity cost per job type (model) import { JOBS } from "./job-types.mjs"; import { run, singleRun } from "./interruption-period.mjs"; const PERSISTENT = 1.0, INTERRUPTIBLE = 0.32; // input from M23/K01: fictional unit, not a real price const MIX = [1.0, 0.7, 0.5, 0.3]; // persistent-share scan // Generator order is kept identical to the previous run: four jobs first, then the single-run attempt. const RUNS = JOBS.map(run), SINGLE = singleRun(JOBS[2]); if (import.meta.url === `file://${process.argv[1]}`) { console.log(`persistent ${PERSISTENT.toFixed(2)} unit/instance-hour, interruptible ${INTERRUPTIBLE} unit ` + `(fictional; break-even yield ${(100 * INTERRUPTIBLE / PERSISTENT).toFixed(0)}%)\n`); console.log("job type".padEnd(32) + "yield".padEnd(9) + "persistent unit".padEnd(17) + "interruptible unit".padEnd(20) + "savings".padEnd(11) + "duration multiplier"); JOBS.forEach((job, i) => { const k = RUNS[i], yieldRate = i === 2 ? SINGLE.completed / SINGLE.persistentCompletions : k.yield120; const persistentCost = k.hours * PERSISTENT, interruptibleCost = (k.hours / yieldRate) * INTERRUPTIBLE; console.log(job.name.padEnd(32) + `${(100 * yieldRate).toFixed(1)}%`.padEnd(9) + persistentCost.toFixed(0).padEnd(17) + interruptibleCost.toFixed(0).padEnd(20) + `${(100 * (1 - interruptibleCost / persistentCost)).toFixed(0)}%`.padEnd(11) + (1 / yieldRate).toFixed(2)); }); const service = JOBS[3], s = RUNS[3]; console.log("\npersistent/interruptible mix for the service (30 instances, same capacity):"); console.log("persistent share".padEnd(18) + "persistent instances".padEnd(22) + "interruptible instances".padEnd(25) + "monthly unit".padEnd(14) + "instantly losable"); for (const p of MIX) { const persistentCount = Math.round(service.instances * p), interruptibleCount = service.instances - persistentCount; const monthly = (persistentCount * PERSISTENT + (interruptibleCount / s.yield120) * INTERRUPTIBLE) * service.hours * 30; console.log(`${(100 * p).toFixed(0)}%`.padEnd(18) + String(persistentCount).padEnd(22) + String(interruptibleCount).padEnd(25) + monthly.toFixed(0).padEnd(14) + `${(100 * interruptibleCount / service.instances).toFixed(0)}% capacity`); } const KEPT = ["putting a job type on interruptible capacity", "checkpoint frequency", "the work to do within the notice window", "persistent/interruptible mix ratio", "retry policy"]; const HANDED_OFF = ["the moment of interruption", "notice duration", "how many instances are reclaimed at once", "when capacity comes back"]; const IRREVERSIBLE = ["unsaved work on the interrupted instance's local disk", "an external write left half-done in the notice window", "the inability to add a checkpoint once a run has started"]; console.log(`\ndecision ledger: ${KEPT.length} items kept, ${HANDED_OFF.length} items with the provider`); console.log(` with the provider: ${HANDED_OFF.join(", ")}`); for (const g of IRREVERSIBLE) console.log(` cannot be undone — ${g}`); } ``` ``` persistent 1.00 unit/instance-hour, interruptible 0.32 unit (fictional; break-even yield 32%) job type yield persistent unit interruptible unit savings duration multiplier nightly batch job 99.6% 2160 694 68% 1.00 queue worker 99.7% 14400 4623 68% 1.00 long single run 41.7% 2400 1843 23% 2.40 user-facing service 99.8% 21600 6927 68% 1.00 persistent/interruptible mix for the service (30 instances, same capacity): persistent share persistent instances interruptible instances monthly unit instantly losable 100% 30 0 21600 0% capacity 70% 21 9 17198 30% capacity 50% 15 15 14264 50% capacity 30% 9 21 11329 70% capacity decision ledger: 5 items kept, 4 items with the provider with the provider: the moment of interruption, notice duration, how many instances are reclaimed at once, when capacity comes back cannot be undone — unsaved work on the interrupted instance's local disk cannot be undone — an external write left half-done in the notice window cannot be undone — the inability to add a checkpoint once a run has started ``` The savings column is the same across three job types — 68 percent — and that means every job whose yield stays above 99 percent captures the full discount. In the long run, savings drop to 23 percent. What stands out is that the savings are **still positive**: the break-even yield is 32 percent, and the run's yield is 41.7 percent, above the threshold. Interruptible capacity is cheap on paper for this job too. Cheapness does not decide; the last column decides. The duration multiplier is 2.40: the same run takes two and a half times as long on interruptible capacity and finishes 15 times a month instead of 36. If a nightly report runs two days late, what is being paid is not the unit but **the result arriving late**, and that cost never shows up in this table. Cheapness can be measured; the value of delay varies by job — this is where interruptible capacity's real limit lies. The mix table counts the service's middle path. When 30 percent of capacity is converted to interruptible, the monthly cost drops from 21,600 to 17,198 units, and the capacity that can be lost at once rises to 30 percent; converting 70 percent brings the cost down to 11,329, but 70 percent of capacity becomes reclaimable by a single decision. The persistent share is the smallest capacity that must stay standing at the moment of interruption. In the decision ledger, **five of nine items are kept, four are with the provider** — and the four handed off are the heaviest handoff in this course so far, because among them is the **moment** of interruption itself. The three items that cannot be undone also come from this handoff: unsaved work on the interrupted instance's local disk does not come back, an external write left half-done in the notice window cannot be undone, and a checkpoint cannot be added once a run has started. The third is the most insidious: the decision to move to interruptible capacity is reversible, but whether **the job running at that moment** tolerates interruption cannot be changed mid-run. ## Summary - The notice window only helps when it is longer than the recovery time: the service recovers in 20 seconds, the batch job in 45, the queue worker in 90; the 20-hour run with no checkpoint demands 660 seconds and its loss never zeroes out across the scan. - The job that does not tolerate interruption at all is `long single run`; the fix is not a longer notice but splitting the job — an application decision, not a capacity decision. - The same unit of measure tells a different story: a 120-second notice saves 34.3 job-hours a month in the batch job and 1.9 hours in the service — but that 1.9 hours is 34,800 requests. - Restart is a cost notice cannot save: the service takes 1,450 interruptions a month and 48.3 hours go to startup, twenty-five times the saved time. - The average-loss assumption misleads in the long run: the rough calculation says 59.2 percent yield, run attempt by attempt it comes out to 42 percent — 15 of 66 attempts complete, 415 hours go to waste. - Five of nine items are kept, four are with the provider; interruptible capacity gives 68 percent savings in three job types, gives 23 percent in the long run but multiplies duration by 2.40. ## Next Step These three lessons counted every decision on a single instance: its family, its size, its image, the work it does at startup, whether it is persistent or interruptible. Every item counted is a decision made for a single instance — and the interruptible capacity section showed that these decisions are **given again, often**: 1,450 instances open a month. In each of those 1,450 instances, the same family, the same size, the same image, the same subnet, and the same startup script must be chosen. On a manually launched instance, each of these choices is a separate write, and every write is a chance to drift. The next lesson collects these items into a single definition and asks: at the end of a period, in how many items do manually launched instances and template-launched ones diverge from each other, how many instances fall behind when the template is updated, and how large a surface of inputs does the template itself open?