Cold Start

The latency numbers of the previous two lessons rested on a single assumption: an instance ready to run the function was standing by when the call arrived. Because the capacity decision is handed off, this is not always true. Instances are created as requests arrive and reclaimed when idle; a request that finds no ready instance experiences a cold start — setting up the runtime, unpacking the package, and startup code all land inside the request path.

M23/K02 measured the same trade-off — the item embedded in the image versus the work left for boot — at the scale of seconds; that measurement is not repeated here. Here the scale is milliseconds, and the object is not an image but a package. A single ratio is asked: how many calls start cold, what it depends on, and what mitigation costs.

FC13. The measurement window is 1,800 seconds; the regional measurement network’s three fictional traffic profiles are built with a self-written generator, and the seed is 20260804. FC14. A warm call takes 400 ms; a cold call adds 900 ms; this extra time enters latency, not the bill. FC15. Profiles: field notification is sparse (average 90 s gap), reading validation is regular (average 250 ms gap), and the nightly batch is bursty (once every 600 seconds, a 60-second burst, 25 ms gap). FC16. Pool lifetime is how long an instance sits idle before it is reclaimed; the baseline value is 300 seconds. FC17. The function’s concurrency share is 40; once the pool is full, an incoming request joins the queue rather than being dropped. FC18. Prices are written in fictional units: 0.0005 per call, 0.0000025 per processing millisecond, 0.0002 per pre-warmed instance-second.

Is There an Instance in the Pool

The model rests on a single rule: if the pool has an idle instance, the call is warm; if not and the share has not filled, a new instance is created; if the share has also filled, the request joins the queue.

// measurement-network/warm-pool.mjs — warm if the pool has an instance when the request arrives, cold otherwise (model)

export const WINDOW = 1800000, PROC_MS = 400, COLD_EXTRA_MS = 900;   // ms: measurement window, processing, cold extra
export const LIFETIME = 300000, SHARE_LIMIT = 40;                    // ms idle-kept, the function's concurrency share
export const CALL_UNIT = 0.0005, MS_UNIT = 0.0000025, WARM_UNIT = 0.0002;   // fictional unit

export const rng = (t) => { let s = t >>> 0; return () => {
  s ^= s << 13; s >>>= 0; s ^= s >>> 17; s ^= s << 5; s >>>= 0; return s / 4294967296; }; };

// Three fictional traffic profiles: sparse field notifications, regular validation, bursty nightly batch.
export const PROFILE = {
  sparse: { gap: 90000, windows: [[0, WINDOW]] },
  regular: { gap: 250, windows: [[0, WINDOW]] },
  bursty: { gap: 25, windows: [[0, 60000], [600000, 660000], [1200000, 1260000]] },
};

export const arrivals = (name, seed = 20260804) => {
  const r = rng(seed), prof = PROFILE[name], out = [];
  for (const [start, end] of prof.windows) {
    let t = start;
    while (t < end) { t += -Math.log(1 - r()) * prof.gap; if (t < end) out.push(t); }
  }
  return out;
};

const percentile = (a, p) => [...a].sort((x, y) => x - y)[Math.floor(a.length * p)];
const idle = (inst, end) => Math.max(0, end - inst.born - inst.busy) / 1000;

// Pool: every instance gets a "free" timestamp the moment it frees up; once its lifetime elapses from that stamp, it is reclaimed.
export const run = (arrivalTimes, { lifetime = LIFETIME, shareLimit = SHARE_LIMIT, coldExtra = COLD_EXTRA_MS, proc = PROC_MS,
  warm = 0, extraCallMs = 0 } = {}) => {
  const pool = [];
  for (let i = 0; i < warm; i++) pool.push({ free: 0, born: 0, busy: 0, pinned: true });
  let coldCount = 0, queued = 0, longestWait = 0, idleSec = 0, mostInstances = pool.length, workMs = 0;
  const latency = [];
  for (const t of arrivalTimes) {
    for (let i = pool.length - 1; i >= 0; i--) {
      const o = pool[i];
      if (!o.pinned && o.free + lifetime < t) { idleSec += idle(o, o.free + lifetime); pool.splice(i, 1); }
    }
    let inst = pool.find((o) => o.free <= t), start = t, extra = 0, wait = 0;
    if (!inst && pool.length < shareLimit) {
      inst = { free: 0, born: t, busy: 0, pinned: false };
      pool.push(inst); extra = coldExtra; coldCount++;
    } else if (!inst) {
      inst = pool.reduce((a, b) => (b.free < a.free ? b : a));
      wait = inst.free - t; start = inst.free; queued++;
      if (wait > longestWait) longestWait = wait;
    }
    const duration = extra + proc + extraCallMs;
    inst.free = start + duration; inst.busy += duration; workMs += duration;
    latency.push(wait + duration);
    if (pool.length > mostInstances) mostInstances = pool.length;
  }
  for (const o of pool) idleSec += idle(o, o.pinned ? WINDOW : Math.min(WINDOW, o.free + lifetime));
  return { calls: arrivalTimes.length, cold: coldCount, rate: coldCount / arrivalTimes.length,
    avgLatency: latency.reduce((a, b) => a + b, 0) / latency.length, p99: percentile(latency, 0.99),
    queued, longestWait, mostInstances, idleSec,
    cost: arrivalTimes.length * CALL_UNIT + workMs * MS_UNIT + warm * (WINDOW / 1000) * WARM_UNIT };
};

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log(`window ${WINDOW / 1000} s, processing ${PROC_MS} ms, cold extra ${COLD_EXTRA_MS} ms, ` +
    `lifetime ${LIFETIME / 1000} s, share ${SHARE_LIMIT}`);
  console.log("\n" + "profile".padEnd(10) + "calls".padEnd(9) + "cold".padEnd(8) + "cold rate".padEnd(12) +
    "avg latency".padEnd(14) + "p99".padEnd(12) + "queued".padEnd(15) +
    "longest wait".padEnd(17) + "most instances".padEnd(16) + "idle instance-s");
  for (const name of Object.keys(PROFILE)) {
    const s = run(arrivals(name));
    console.log(name.padEnd(10) + String(s.calls).padEnd(9) + String(s.cold).padEnd(8) +
      `${(s.rate * 100).toFixed(2)}%`.padEnd(12) + `${s.avgLatency.toFixed(0)} ms`.padEnd(14) +
      `${s.p99.toFixed(0)} ms`.padEnd(12) + String(s.queued).padEnd(15) +
      `${s.longestWait.toFixed(0)} ms`.padEnd(17) + String(s.mostInstances).padEnd(16) + s.idleSec.toFixed(0));
  }
  const a = run(arrivals("bursty"));
  console.log(`\nin the bursty profile, cold calls add ${(a.cold * COLD_EXTRA_MS / 1000).toFixed(0)} s to latency; ` +
    `a warm call is ${PROC_MS} ms, a cold call is ${PROC_MS + COLD_EXTRA_MS} ms — ${((PROC_MS + COLD_EXTRA_MS) / PROC_MS).toFixed(2)}x`);
}

window 1800 s, processing 400 ms, cold extra 900 ms, lifetime 300 s, share 40

profile   calls    cold    cold rate   avg latency   p99         queued         longest wait     most instances  idle instance-s
sparse    18       2       11.11%      500 ms        1300 ms     0              0 ms             1               1722
regular   7242     9       0.12%       401 ms        400 ms      0              0 ms             8               10510
bursty    7241     120     1.66%       419 ms        1300 ms     135            539 ms           40              38159

in the bursty profile, cold calls add 108 s to latency; a warm call is 400 ms, a cold call is 1300 ms — 3.25x

The three profiles show three separate behaviors. In regular traffic the cold rate is 0.12%: requests arrive 250 ms apart, the pool settles at eight instances, and only nine calls start cold. In sparse traffic the rate is 11.11%, and that rate comes not from load but from how infrequent the calls are. In bursty traffic the rate is 1.66%, but the real number is in a different column: 135 requests join the queue, the longest wait is 539 ms.

The queue is the second face of cold start. When the burst begins the pool is empty; every new request needs a new instance, and every instance spends 900 ms on setup. Once the share reaches 40, requests start waiting. A cold call does not only raise its own latency; by slowing how fast the pool fills, it builds a queue behind it: the average shown is 419 ms, and the 99th percentile is 1,300 ms.

Pool Lifetime Scan

Pool lifetime is a handed-off decision: the provider decides how long an instance is kept idle. The scan raises lifetime from zero to 1,800 seconds.

// measurement-network/lifetime-scan.mjs — the cold call rate as pool lifetime and traffic profile change (model)
import { PROFILE, arrivals, run } from "./warm-pool.mjs";

export const LIFETIMES = [0, 15, 30, 60, 120, 300, 600, 1800];   // seconds
export const NAMES = Object.keys(PROFILE);

if (import.meta.url === `file://${process.argv[1]}`) {
  const arrivalsByProfile = Object.fromEntries(NAMES.map((a) => [a, arrivals(a)]));
  console.log("pool lifetime".padEnd(15) + NAMES.map((a) => `${a}: cold / idle`.padEnd(24)).join("") +
    "bursty: queued / avg latency");
  const table = {};
  for (const lt of LIFETIMES) {
    const row = Object.fromEntries(NAMES.map((a) => [a, run(arrivalsByProfile[a], { lifetime: lt * 1000 })]));
    table[lt] = row;
    console.log(`${lt} s`.padEnd(15) +
      NAMES.map((a) => `${(row[a].rate * 100).toFixed(2)}% / ${row[a].idleSec.toFixed(0)}`.padEnd(24)).join("") +
      `${row.bursty.queued} / ${row.bursty.avgLatency.toFixed(0)} ms`);
  }
  console.log("");
  for (const name of NAMES) {
    const good = LIFETIMES.find((lt) => table[lt][name].rate < 0.01);
    const atZero = table[0][name], atMax = table[1800][name];
    console.log(`${name.padEnd(9)} at lifetime 0: ${(atZero.rate * 100).toFixed(1)}% cold and ${atZero.idleSec.toFixed(0)} idle instance-s; ` +
      `at lifetime 1800: ${(atMax.rate * 100).toFixed(2)}% and ${atMax.idleSec.toFixed(0)}; ` +
      (good === undefined ? "never drops below 1% at any lifetime" : `drops below 1% at a ${good} s lifetime`));
  }
}

pool lifetime  sparse: cold / idle     regular: cold / idle    bursty: cold / idle     bursty: queued / avg latency
0 s            100.00% / 0             100.00% / 0             49.99% / 0              3621 / 969 ms
15 s           88.89% / 266            1.35% / 7651            1.78% / 3819            135 / 420 ms
30 s           77.78% / 489            0.65% / 8493            1.73% / 5713            135 / 419 ms
60 s           72.22% / 895            0.33% / 9208            1.66% / 9359            135 / 419 ms
120 s          38.89% / 1364           0.18% / 9883            1.66% / 16559           135 / 419 ms
300 s          11.11% / 1722           0.12% / 10510           1.66% / 38159           135 / 419 ms
600 s          5.56% / 1781            0.11% / 10873           0.59% / 57609           47 / 407 ms
1800 s         5.56% / 1781            0.11% / 10873           0.55% / 69047           47 / 406 ms

sparse    at lifetime 0: 100.0% cold and 0 idle instance-s; at lifetime 1800: 5.56% and 1781; never drops below 1% at any lifetime
regular   at lifetime 0: 100.0% cold and 0 idle instance-s; at lifetime 1800: 0.11% and 10873; drops below 1% at a 30 s lifetime
bursty    at lifetime 0: 50.0% cold and 0 idle instance-s; at lifetime 1800: 0.55% and 69047; drops below 1% at a 600 s lifetime

The scan gives three separate curves. In regular traffic, a 15-second lifetime pulls the cold rate from 100% to 1.35%; beyond that it gains nothing but raises idle instance-seconds from 7,651 to 10,873. In bursty traffic the rate does not change at all from 15 seconds to 300 seconds — it sticks at 1.66% — and drops suddenly to 0.59% at 600 seconds. The threshold is not a coincidence: the gap between bursts is 540 seconds, and once lifetime crosses that gap, the pool survives between two bursts. On the same row, queued requests drop from 135 to 47.

The price of that threshold sits on the same row: idle instance-seconds rise from 38,159 to 57,609. Cutting the cold call rate to a third is paid for by raising idle capacity by half.

The sparse profile never drops below 1% at any lifetime; even at 1,800 seconds the rate is 5.56%, because one of the eighteen calls is the window’s first, and there is no pool before it. In sparse traffic, cold start is not a tuning problem — it is a property of the traffic itself.

Three Mitigation Paths

// measurement-network/mitigation.mjs — three ways to reduce cold start, and the cost of each (model)
import { COLD_EXTRA_MS, PROC_MS, WINDOW, arrivals, run } from "./warm-pool.mjs";

// [name, run options, kind of cost]
export const PATH = [
  ["no mitigation", {}, "—"],
  ["pre-warming (16)", { warm: 16 }, "16 instances sit idle for 1800 s"],
  ["package shrink", { coldExtra: 350 }, "3 dependencies are moved out"],
  ["lazy loading", { coldExtra: 500, extraCallMs: 40 }, "40 ms of lazy loading on every call"],
  ["shrink + warming", { coldExtra: 350, warm: 16 }, "the cost of both together"],
];

if (import.meta.url === `file://${process.argv[1]}`) {
  const arr = arrivals("bursty"), baseline = run(arr);
  console.log(`bursty profile, ${arr.length} calls, pool lifetime 300 s; warm call ${PROC_MS} ms`);
  console.log("\n" + "path".padEnd(22) + "cold".padEnd(8) + "cold rate".padEnd(12) +
    "avg latency".padEnd(14) + "p99".padEnd(12) + "queued".padEnd(15) +
    "idle instance-s".padEnd(17) + "cost".padEnd(10) + "latency gain");
  for (const [name, opts] of PATH) {
    const row = run(arr, opts);
    console.log(name.padEnd(22) + String(row.cold).padEnd(8) + `${(row.rate * 100).toFixed(2)}%`.padEnd(12) +
      `${row.avgLatency.toFixed(1)} ms`.padEnd(14) + `${row.p99.toFixed(0)} ms`.padEnd(12) +
      String(row.queued).padEnd(15) + row.idleSec.toFixed(0).padEnd(17) + row.cost.toFixed(2).padEnd(10) +
      `${(baseline.avgLatency - row.avgLatency).toFixed(1)} ms`);
  }
  console.log("\n" + "path".padEnd(22) + "cost".padEnd(38) + "extra cost per ms gained");
  for (const [name, opts, cost] of PATH.slice(1)) {
    const row = run(arr, opts), gain = baseline.avgLatency - row.avgLatency;
    console.log(name.padEnd(22) + cost.padEnd(38) +
      (gain > 0 ? ((row.cost - baseline.cost) / gain).toFixed(3) : "no gain"));
  }
  const threshold = 40 / (COLD_EXTRA_MS - 500);
  console.log(`lazy loading only pays off once the cold rate is above ${(threshold * 100).toFixed(0)}%: ` +
    `40 ms lands on every call, while the 400 ms gained falls only on cold calls — in this profile the rate is 1.66%`);
  const warmRow = run(arr, { warm: 16 });
  console.log(`\npre-warming raises capacity spent idle from ${baseline.idleSec.toFixed(0)} instance-s to ` +
    `${warmRow.idleSec.toFixed(0)} instance-s; over the ${WINDOW / 1000} s window that is ` +
    `${(warmRow.cost - baseline.cost).toFixed(2)} extra units — the cold extra itself is ${COLD_EXTRA_MS} ms`);
}

bursty profile, 7241 calls, pool lifetime 300 s; warm call 400 ms

path                  cold    cold rate   avg latency   p99         queued         idle instance-s  cost      latency gain
no mitigation         120     1.66%       418.9 ms      1300 ms     135            38159            11.13     0.0 ms
pre-warming (16)      49      0.68%       406.1 ms      400 ms      0              42736            16.73     12.8 ms
package shrink        111     1.53%       405.4 ms      750 ms      0              35500            10.96     13.5 ms
lazy loading          120     1.66%       448.5 ms      940 ms      22             38229            11.74     -29.7 ms
shrink + warming      43      0.59%       402.1 ms      400 ms      0              40933            16.66     16.8 ms

path                  cost                                  extra cost per ms gained
pre-warming (16)      16 instances sit idle for 1800 s      0.438
package shrink        3 dependencies are moved out          -0.013
lazy loading          40 ms of lazy loading on every call   no gain
shrink + warming      the cost of both together             0.329
lazy loading only pays off once the cold rate is above 10%: 40 ms lands on every call, while the 400 ms gained falls only on cold calls — in this profile the rate is 1.66%

pre-warming raises capacity spent idle from 38159 instance-s to 42736 instance-s; over the 1800 s window that is 5.60 extra units — the cold extra itself is 900 ms

The three paths each pay a different kind of cost. Pre-warming pulls the cold rate to 0.68%, the queue to zero, and the 99th percentile from 1,300 ms to 400 ms: because sixteen instances are always waiting in the pool, the burst’s first second is not cold. Its cost is the very thing the first lesson avoided — idle instance-seconds rise to 42,736, and the bill climbs from 11.13 units to 16.73 units, a 50% increase.

Package shrinking is the one path that looks free: once the cold extra drops from 900 ms to 350 ms, both latency and cost fall. But the cost is not in the table — three dependencies have been pulled out of the package, and that has to be paid for somewhere. This is the quietest form of a workaround around a constraint.

The third path loses money. Moving startup work pulls the cold extra down to 500 ms but adds 40 ms of lazy loading to every call; average latency rises 29.7 ms, and twenty-two requests join the queue. The last row gives the threshold: 40 ms lands on every call, while the 400 ms gained falls only on cold calls, so the path pays off only once the cold rate is above 10%. In this profile the rate is 1.66%; in the sparse profile it is 11.11%.

Summary

• A cold call takes 1,300 ms, a warm call 400 ms: 3.25x. Even though the bursty profile’s rate is only 1.66%, 135 requests join the queue and the longest wait is 539 ms.
• In the pool lifetime scan, the threshold comes from the shape of the traffic: in the bursty profile the rate sticks at 1.66% between 15 and 300 seconds, then drops to 0.59% at 600 seconds, because the gap between bursts is 540 seconds. The threshold’s cost is written on the same row: idle instance-seconds rise from 38,159 to 57,609.
• Pre-warming drops the queue to zero and raises the bill 50%; package shrinking lowers both, but it demands pulling three dependencies out of the package.
• Moving startup work raises latency 29.7 ms in this profile; it pays off only once the cold rate is above 10% — the mitigation path itself depends on a threshold.

Next Step

In this lesson the concurrency share of 40 was a fixed number, and it explained where the queue came from. But it does not stand alone: the memory choice decides the processor share, the processor share decides processing duration, and processing duration decides how much work the same share can do.

The next lesson applies three thresholds — memory, duration, and concurrency — to a job set and counts how many jobs do not fit. Memory is scanned, and it shows that duration and unit cost move in opposite directions, with the cheapest point landing in the middle. Once the concurrency limit is exceeded, throttling and the queue behind it are counted; sharing the limit at the account level is measured with an example.