Edge Runtime Model

Events were routed, streams were set up, long-running jobs ran step by step. But all of this machinery still sits in a single region. In the fictional measurement network, field crews and branch sites are scattered across the region; the only thing close to the user was the cache, and the cache only stores a response already produced. Whenever a new response has to be produced, the request still goes to the region.

This course moves the computation itself to the edge. The first question is: what runs at the edge? What runs in the region is a container, and the container’s cost has been measured. The same cost cannot be paid at hundreds of edge locations, for dozens of different applications at each one. The thing that runs at the edge is named the isolate, and this lesson adds it to the familiar scale.

EC1. Every number is a model input, not the output of a measured setup; what matters is the order-of-magnitude gap between two tiers. EC2. Interface items take a binary value: offered or not offered; an item offered only in restricted form counts as “not offered.” EC3. The startup and memory values on the container and virtual machine rows are inputs from the earlier measurement; this lesson does not re-measure them, it only adds the third row.

The Row at the Edge of the Scale

Isolation tiers were compared earlier on five metrics and their cost was paid in three items. The isolate is one step further down that scale: it carries no kernel of its own, and no file system of its own either. What it isolates is a single language’s interpreter context — the variable space, the stack, and the object set are separate, but the process underneath is shared. The new metric is therefore the width of the offered interface surface: how many items the runtime offers to the application.

// measurement-network/unit.mjs — adding the isolate to the three-tier scale
// MODEL: every number is a declared model input. What matters is the
// order-of-magnitude gap between two tiers, not a single cell's value.

export const LEVELS = {
  "virtual machine": { startupMs: 25_000, memoryBaselineMB: 512 },
  "container": { startupMs: 300, memoryBaselineMB: 12 },
  "isolate": { startupMs: 3, memoryBaselineMB: 3 },
};

// The interface items the runtime offers to the application, and at which tier.
export const SURFACE = {
  "request/response handling": ["virtual machine", "container", "isolate"],
  "timers":                    ["virtual machine", "container", "isolate"],
  "outbound network request":  ["virtual machine", "container", "isolate"],
  "crypto primitives":         ["virtual machine", "container", "isolate"],
  "stream processing":         ["virtual machine", "container", "isolate"],
  "in-memory data structure":  ["virtual machine", "container", "isolate"],
  "file system":               ["virtual machine", "container"],
  "local disk persistence":    ["virtual machine", "container"],
  "long-running process":      ["virtual machine", "container"],
  "subprocess spawning":       ["virtual machine", "container"],
  "raw sockets":               ["virtual machine", "container"],
  "native binary addon":       ["virtual machine", "container"],
  "kernel tuning":             ["virtual machine"],
};

export const NAMES = Object.keys(LEVELS);
export const ITEMS = Object.keys(SURFACE);
export const offered = (d) => ITEMS.filter((k) => SURFACE[k].includes(d));
const magnitude = (n) => "10^" + Math.round(Math.log10(n));

if (import.meta.main) {
console.log("cost and interface surface (model input)");
console.log("tier".padEnd(17) + "startup".padEnd(11) + "magnitude".padEnd(11) +
  "mem baseline".padEnd(15) + "magnitude".padEnd(11) + "surface");
for (const d of NAMES) {
  const v = LEVELS[d];
  console.log(d.padEnd(17) + `${v.startupMs} ms`.padEnd(11) + magnitude(v.startupMs).padEnd(11) +
    `${v.memoryBaselineMB} MB`.padEnd(15) + magnitude(v.memoryBaselineMB).padEnd(11) +
    `${offered(d).length}/${ITEMS.length}`);
}

console.log("\ninterface item (+ offered, - not offered)");
console.log("item".padEnd(28) + NAMES.map((d) => d.padEnd(17)).join(""));
for (const k of ITEMS)
  console.log(k.padEnd(28) + NAMES.map((d) => (SURFACE[k].includes(d) ? "+" : "-").padEnd(17)).join(""));

const missing = ITEMS.filter((k) => !SURFACE[k].includes("isolate"));
console.log(`\n${missing.length} items missing in the isolate:`);
for (const k of missing) console.log("  - " + k);
}

cost and interface surface (model input)
tier             startup    magnitude  mem baseline   magnitude  surface
virtual machine  25000 ms   10^4       512 MB         10^3       13/13
container        300 ms     10^2       12 MB          10^1       12/13
isolate          3 ms       10^0       3 MB           10^0       6/13

interface item (+ offered, - not offered)
item                        virtual machine  container        isolate          
request/response handling   +                +                +                
timers                      +                +                +                
outbound network request    +                +                +                
crypto primitives           +                +                +                
stream processing           +                +                +                
in-memory data structure    +                +                +                
file system                 +                +                -                
local disk persistence      +                +                -                
long-running process        +                +                -                
subprocess spawning         +                +                -                
raw sockets                 +                +                -                
native binary addon         +                +                -                
kernel tuning               +                -                -                

7 items missing in the isolate:
  - file system
  - local disk persistence
  - long-running process
  - subprocess spawning
  - raw sockets
  - native binary addon
  - kernel tuning

Startup magnitudes drop in two-digit steps: 10^4, 10^2, 10^0. Moving from the virtual machine to the container gains two orders of magnitude; moving from the container to the isolate gains two more. The memory baseline follows the same 10^3, 10^1, 10^0 sequence. Up to this point the table is familiar — only a cheaper isolation tier has been added.

The third column turns the table around. The interface surface is 13/13, 12/13, and 6/13. The virtual machine offers every item; the only item the container loses is kernel tuning, because the kernel is shared. The isolate, by contrast, loses seven of thirteen items at once. This is not a steady decline along the scale — it is a break: the container gives up one item for one order of magnitude of cheaper cost, the isolate gives up seven items for two orders of magnitude.

The Disappearance of Cold Start

Startup latency dropping to the 10^0 order of magnitude changes exactly one thing, and that thing is large. The most familiar flaw of a serverless function is that an instance starting from zero delays the first call. That delay is not noise next to the application’s own runtime — it is an item that drowns it out. The run below runs the same call sequence at three tiers and counts how the delay lands across percentiles.

EC4. The cold-call rate is 6% and its pattern is fixed; there is no randomness. EC5. The application’s own processing time is 12 ms and does not change by tier; network latency is not in this model. EC6. The cost of working around a limit is measured by the number of network round trips added per request; how many milliseconds a round trip costs is the next lesson’s subject.

// measurement-network/gain.mjs — how startup latency shows up in percentiles and
// how the missing surface converts into workload. MODEL: call distribution is deterministic.
import { LEVELS, NAMES, offered } from "./unit.mjs";

const CALLS = 10_000, COLD_PERCENT = 6, PROCESS_MS = 12, THRESHOLD_MS = 100;
const isCold = (i) => (i * COLD_PERCENT) % 100 < COLD_PERCENT;   // pattern: unseeded, fixed
const percentile = (a, p) => a[Math.min(a.length - 1, Math.ceil((p / 100) * a.length) - 1)];

console.log(`${CALLS} calls, ${COLD_PERCENT}% cold, processing ${PROCESS_MS} ms (model)`);
console.log("tier".padEnd(17) + "p50".padEnd(11) + "p95".padEnd(11) + "p99".padEnd(11) +
  "avg extra".padEnd(14) + `over ${THRESHOLD_MS} ms`);
for (const d of NAMES) {
  const g = [];
  for (let i = 0; i < CALLS; i++) g.push(PROCESS_MS + (isCold(i) ? LEVELS[d].startupMs : 0));
  g.sort((x, y) => x - y);
  const extra = g.reduce((t, v) => t + v - PROCESS_MS, 0) / CALLS;
  console.log(d.padEnd(17) + `${percentile(g, 50)} ms`.padEnd(11) + `${percentile(g, 95)} ms`.padEnd(11) +
    `${percentile(g, 99)} ms`.padEnd(11) + `${extra.toFixed(1)} ms`.padEnd(14) +
    String(g.filter((v) => v > THRESHOLD_MS).length));
}

// Zeroing out the cold rate: the memory cost of keeping pre-warmed instances.
const WARM = 40;
console.log(`\nif ${WARM} instances are kept warm to zero out the cold rate`);
for (const d of NAMES)
  console.log("  " + d.padEnd(17) + `${String(LEVELS[d].memoryBaselineMB * WARM).padStart(6)} MB idle baseline`);

const WORK = {
  "request routing":            ["request/response handling"],
  "header editing":              ["request/response handling"],
  "session validation":          ["request/response handling", "crypto primitives"],
  "read-format conversion":      ["request/response handling", "stream processing"],
  "nightly batch processing":    ["long-running process", "file system", "local disk persistence"],
  "invoice document":            ["file system", "native binary addon"],
  "log archiving":               ["local disk persistence", "file system"],
  "persistent device connection":["raw sockets", "long-running process"],
};
const runs = (w, d) => WORK[w].every((k) => offered(d).includes(k));

console.log("\ndoes the workload run at the tier");
console.log("workload".padEnd(30) + NAMES.map((d) => d.padEnd(17)).join(""));
for (const w of Object.keys(WORK))
  console.log(w.padEnd(30) + NAMES.map((d) => (runs(w, d) ? "runs" : "no run").padEnd(17)).join(""));
for (const d of NAMES)
  console.log("  " + d.padEnd(17) + `${Object.keys(WORK).filter((w) => runs(w, d)).length}/${Object.keys(WORK).length} workloads`);

// The cost of working around a missing item: the network round trip added per request.
const WORKAROUND = {
  "file system": [1, "read/write goes out to the object store"],
  "local disk persistence": [1, "state is kept in an external key-value store"],
  "long-running process": [3, "the job is split; each part reads state from outside"],
  "subprocess spawning": [2, "the call is delegated to an instance in the region"],
  "raw sockets": [2, "the connection terminates in the region; the edge only routes it"],
  "native binary addon": [0, "a pure-language equivalent is written instead; processing time grows"],
  "kernel tuning": [0, "there is no setting; the job cannot be held at this tier"],
};
console.log("\nworking around a missing item (extra network round trip per request)");
let total = 0;
for (const [k, [trip, way]] of Object.entries(WORKAROUND)) {
  total += trip;
  console.log("  " + k.padEnd(28) + `+${trip} trip   ` + way);
}
console.log(`  ${"total for the seven items".padEnd(28)}+${total} trips`);

10000 calls, 6% cold, processing 12 ms (model)
tier             p50        p95        p99        avg extra     over 100 ms
virtual machine  12 ms      25012 ms   25012 ms   1500.0 ms     600
container        12 ms      312 ms     312 ms     18.0 ms       600
isolate          12 ms      15 ms      15 ms      0.2 ms        0

if 40 instances are kept warm to zero out the cold rate
  virtual machine   20480 MB idle baseline
  container           480 MB idle baseline
  isolate             120 MB idle baseline

does the workload run at the tier
workload                      virtual machine  container        isolate          
request routing               runs             runs             runs             
header editing                runs             runs             runs             
session validation            runs             runs             runs             
read-format conversion        runs             runs             runs             
nightly batch processing      runs             runs             no run           
invoice document              runs             runs             no run           
log archiving                 runs             runs             no run           
persistent device connection  runs             runs             no run           
  virtual machine  8/8 workloads
  container        8/8 workloads
  isolate          4/8 workloads

working around a missing item (extra network round trip per request)
  file system                 +1 trip   read/write goes out to the object store
  local disk persistence      +1 trip   state is kept in an external key-value store
  long-running process        +3 trip   the job is split; each part reads state from outside
  subprocess spawning         +2 trip   the call is delegated to an instance in the region
  raw sockets                 +2 trip   the connection terminates in the region; the edge only routes it
  native binary addon         +0 trip   a pure-language equivalent is written instead; processing time grows
  kernel tuning               +0 trip   there is no setting; the job cannot be held at this tier
  total for the seven items   +9 trips

All three tiers share the same p50: 12 ms. The median call lands on a warm instance and never sees startup latency at all. The difference shows up at p95, because cold calls fill the 6% slice and the slowest 6% starts right there. p95 is 25,012 ms on the virtual machine, 312 ms on the container, 15 ms on the isolate. The average extra delay is 1,500.0 / 18.0 / 0.2 ms respectively.

The last column makes the call. At a 100 ms threshold, the virtual machine and the container produce the same number: 600 calls. Even though there is an 80-fold gap between the two tiers, both cross the threshold; cold start shrinks on the container but does not disappear. On the isolate, the number of calls that cross the threshold is 0. Cold start is not improved here — it stops being a problem: a 3 ms startup next to a 12 ms operation is not worth measuring.

This gain has to be read together with the cost of buying it on the container. The way to bring the cold rate to zero is pre-warming instances, and with forty instances kept warm, the idle memory baseline is 20,480 MB on the virtual machine, 480 MB on the container. On the isolate this row is meaningless: no warming happens, because there is nothing to warm.

The Missing Interface and the Cost of Working Around It

The decision handed over here is clear: the runtime’s setup, version, capacity, and placement are gone. What comes back in exchange is a number — 7 of 13 items are missing. Item by item: no file system, no local disk persistence, no long-running process, no subprocess spawning, no raw sockets, no native binary addon, no kernel tuning. The last item has the widest consequence: a library that depends on a native binary addon does not run at this tier — even if the application never calls it directly, if it sits anywhere in the dependency chain, the package fails to load.

The workload table turns this into the fictional network. Four of the eight workloads run on the isolate: request routing, header editing, session validation, read-format conversion. Four do not: nightly batch processing, invoice document generation, log archiving, persistent device connection. The ratio is 4/8, which means half of this same network’s jobs cannot sit at the edge. The container and the virtual machine both run all eight; the split is not about the type of workload, it is about which item it needs — the four that run touch only the request-handling, crypto, and stream items.

Working around the constraint is possible, and the cost is totaled in the last table. Using an object store instead of a file system adds +1 network round trip per access, an external key-value store instead of local disk adds +1 trip, splitting a long-running process and reading its state from outside adds +3 trips, and handing off subprocess and raw-socket work to the region adds two trips each. The total for the seven items is +9 trips. Two items add zero trips, but the cost is not zero: the native binary addon’s pure-language equivalent lengthens processing time, and kernel tuning has no equivalent at all — that job cannot be held at this tier.

The meaning of those nine trips is still incomplete here, because how many milliseconds a trip costs depends on where it goes from the edge. A trip from the edge to a nearby store and a trip to the region are not the same item; the first table erased 300 ms as a gain, and each workaround trip can claw back a piece of that gain. The real arithmetic of computing at the edge is the difference between these two numbers.

Summary

• The isolate is one step further down the familiar scale: startup magnitudes are 10^4, 10^2, and 10^0; memory baseline magnitudes are 10^3, 10^1, and 10^0. What it isolates is not a kernel or a file system, it is a single language’s interpreter context.
• The third metric turns the table around: the offered interface surface is 13/13, 12/13, and 6/13. The container gives up one item for one order of magnitude, the isolate gives up seven items for two.
• Cold start disappears: p95 is 25,012, 312, and 15 ms; the number of calls crossing the 100 ms threshold is 600, 600, and 0. Keeping a warm pool of forty instances costs 20,480 and 480 MB, while on the isolate there is nothing to warm.
• The constraint that comes back in exchange is spelled out item by item: no file system, local disk, long-running process, subprocess, raw sockets, native binary addon, or kernel tuning. Only 4 of the fictional network’s eight workloads run at this tier.
• The cost of working around the constraint is a network round trip per request: +9 trips total for the seven items. Two items add no trips, but the cost is written into processing time or into moving the job elsewhere.

Next Step

This lesson wrote down what can run at the edge but never measured where the thing that runs actually sits. If four workloads can run at the edge, the same job can also run in the region or at the origin — what decides the choice among the three places? What is the difference, in network round trips, between deciding a request at the edge and moving it to the region, and how many milliseconds do those round trips cost? When the routing decision is made at the edge, what share of requests never reaches the origin at all, and how does that share change when the same decision is authentication or an A/B split instead? The next lesson runs the same request set separately at three places, works out the round-trip count and latency of each placement, and names the work that cannot be done at the edge.