Server-Requiring Deployment

Every piece of work static hosting leaves out needs code that runs at request time. On the North Slope site these are the archive-write endpoint, session renewal, and authorization-dependent editing views; what they share is using a secret and producing the response based on the request.

This lesson asks where that code runs. There are three options, and the difference between them is not a matter of preference but a measurable difference in behavior.

Three Hosting Forms

Long-lived process. A fixed number of instances stays up continuously and shares incoming requests among them. Each instance can run many requests concurrently; total capacity is the number of instances times the per-instance concurrency limit. This capacity is reserved regardless of whether requests arrive. Because the process stays up, it can hold state in memory and keep a connection pool open to the data store.

Per-request instance. When a request arrives, an instance spins up, handles the request, then stays idle for a while before entering shutdown. Concurrency per instance is usually one; two concurrent requests mean two instances. If no warm instance is available, a cold start is paid — the line item called startup time in the Edge Rendering lesson. Reserved capacity tracks demand, but with a delay equal to the idle timeout.

Edge runtime. The same instance model, spread across many nodes. Cold start is much shorter, because the code that runs has a tight budget. In exchange, the data source is remote: per the rule from the Edge Rendering lesson, every request pays that distance to the extent the data cannot move to the edge.

Running the Same Load in Three Forms

All three forms can be measured on the same load trace. The simulation below uses a virtual clock; since it never reads real time, the output is the same on every run.

// runtime.mjs -- runs the same load in three runtime forms.
// A virtual clock is used; Date.now() is never called, so the result is the same on every run.

// Deterministic generator (splitmix32). The seed is fixed, so the arrival trace never changes.
function generator(seed) {
  let s = seed >>> 0;
  return () => {
    s = (s + 0x9e3779b9) >>> 0;
    let z = s;
    z = Math.imul(z ^ (z >>> 16), 0x21f0aaad) >>> 0;
    z = Math.imul(z ^ (z >>> 15), 0x735a2d97) >>> 0;
    return ((z ^ (z >>> 15)) >>> 0) / 4294967296;
  };
}

const DURATION = 600_000; // ms
const SERVICE = 120;      // ms, the work itself: reading data + templating
const WINDOWS = [
  ["sparse", 0, 200_000, 0.5],
  ["steady", 200_000, 400_000, 60],
  ["burst", 400_000, 410_000, 600],
  ["recovery", 410_000, 600_000, 60],
];

const rateAt = (t) => WINDOWS.find(([, a, b]) => t >= a && t < b)[3];

// Arrival trace: exponential intervals, rate varying by window.
function arrivals() {
  const r = generator(20_260_327);
  const list = [];
  let t = 0;
  while (t < DURATION) {
    t += (-Math.log(1 - r()) / rateAt(t)) * 1000;
    if (t < DURATION) list.push(t);
  }
  return list;
}

// 1) Long-lived process: a fixed number of slots is always reserved, there is no cold
//    start, and once the slots are full a request waits in queue.
function longLived(arrivals, { slots, service }) {
  const free = new Array(slots).fill(0);
  const records = [];
  for (const t of arrivals) {
    let k = 0;
    for (let i = 1; i < slots; i++) if (free[i] < free[k]) k = i;
    const start = Math.max(t, free[k]);
    records.push({ t, latency: start - t + service, cold: false });
    free[k] = start + service;
  }
  return { records, intervals: new Array(slots).fill([0, DURATION]) };
}

// 2) Per-request instance: one instance handles a single request, stays warm for
//    idleLife after finishing, then shuts down. A cold start is paid when no warm
//    instance is available.
function perRequest(arrivals, { service, cold, idleLife }) {
  const records = [], intervals = [];
  let live = [];
  for (const t of arrivals) {
    live = live.filter((o) => {
      if (o.lifeEnd > t) return true;
      intervals.push([o.spawnedAt, o.lifeEnd]);
      return false;
    });
    const warm = live.find((o) => o.busyEnd <= t);
    if (warm) {
      records.push({ t, latency: service, cold: false });
      warm.busyEnd = t + service;
      warm.lifeEnd = warm.busyEnd + idleLife;
    } else {
      records.push({ t, latency: cold + service, cold: true });
      const busyEnd = t + cold + service;
      live.push({ busyEnd, lifeEnd: busyEnd + idleLife, spawnedAt: t });
    }
  }
  for (const o of live) intervals.push([o.spawnedAt, Math.min(o.lifeEnd, DURATION)]);
  return { records, intervals };
}

// 3) Edge runtime: the same instance model, load split across nodes, data is remote.
function edgeRuntime(arrivals, { nodes, ...opts }) {
  const partitions = Array.from({ length: nodes }, () => []);
  arrivals.forEach((t, i) => partitions[i % nodes].push(t));
  const records = [], intervals = [];
  for (const partition of partitions) {
    const s = perRequest(partition, opts);
    records.push(...s.records);
    intervals.push(...s.intervals);
  }
  return { records, intervals };
}

const percentile = (list, p) => {
  const s = [...list].sort((a, b) => a - b);
  return s.length === 0 ? 0 : s[Math.min(s.length - 1, Math.floor((p / 100) * s.length))];
};
const overlap = (interval, a, b) =>
  Math.max(0, Math.min(interval[1], b) - Math.max(interval[0], a));

function summary({ records, intervals }, a, b) {
  const slice = records.filter((k) => k.t >= a && k.t < b);
  const reserved = intervals.reduce((s, iv) => s + overlap(iv, a, b), 0) / 1000;
  const used = (slice.length * SERVICE) / 1000;
  return {
    requests: slice.length,
    p50: percentile(slice.map((k) => k.latency), 50),
    p95: percentile(slice.map((k) => k.latency), 95),
    p99: percentile(slice.map((k) => k.latency), 99),
    cold: (100 * slice.filter((k) => k.cold).length) / slice.length,
    reserved,
    utilization: (100 * used) / reserved,
  };
}

const arrival = arrivals();
console.log("arrival trace: " + arrival.length.toLocaleString("en-US") + " requests, " +
  DURATION / 1000 + " s, the work itself " + SERVICE + " ms");
for (const [name, a, b, rate] of WINDOWS)
  console.log("  " + name.padEnd(12) + (a / 1000) + "-" + (b / 1000) + " s   " + rate + " req/s");

const MODEL = [
  ["long-lived process (75 slots)", longLived(arrival, { slots: 75, service: SERVICE })],
  ["per-request instance", perRequest(arrival, { service: SERVICE, cold: 350, idleLife: 60_000 })],
  ["edge runtime (40 nodes)",
   edgeRuntime(arrival, { nodes: 40, service: SERVICE + 60, cold: 40, idleLife: 5_000 })],
];

console.log("\n" + "window".padEnd(12) + "model".padEnd(33) + "req".padStart(7) +
  "p50".padStart(6) + "p95".padStart(6) + "p99".padStart(6) + "cold".padStart(8) +
  "reserved inst-s".padStart(18) + "util".padStart(10));
for (const [name, a, b] of WINDOWS) {
  for (const [mname, s] of MODEL) {
    const o = summary(s, a, b);
    console.log(name.padEnd(12) + mname.padEnd(33) + String(o.requests).padStart(7) +
      o.p50.toFixed(0).padStart(6) + o.p95.toFixed(0).padStart(6) +
      o.p99.toFixed(0).padStart(6) + o.cold.toFixed(1).padStart(7) + "%" +
      o.reserved.toFixed(0).padStart(18) + o.utilization.toFixed(1).padStart(9) + "%");
  }
}

$ node runtime.mjs
arrival trace: 29,655 requests, 600 s, the work itself 120 ms
  sparse      0-200 s   0.5 req/s
  steady      200-400 s   60 req/s
  burst       400-410 s   600 req/s
  recovery    410-600 s   60 req/s

window      model                                req   p50   p95   p99    cold   reserved inst-s      util
sparse      long-lived process (75 slots)         87   120   120   120    0.0%             15000      0.1%
sparse      per-request instance                  87   120   120   470    3.4%               354      2.9%
sparse      edge runtime (40 nodes)               87   220   220   220  100.0%               446      2.3%
steady      long-lived process (75 slots)      12010   120   120   120    0.0%             15000      9.6%
steady      per-request instance               12010   120   120   120    0.2%              4133     34.9%
steady      edge runtime (40 nodes)            12010   180   180   180    0.3%              7968     18.1%
burst       long-lived process (75 slots)       5947   123   208   235    0.0%               750     95.2%
burst       per-request instance                5947   120   120   470    3.4%              2159     33.0%
burst       edge runtime (40 nodes)             5947   180   180   220    2.0%              1488     48.0%
recovery    long-lived process (75 slots)      11611   120   120   120    0.0%             14250      9.8%
recovery    per-request instance               11611   120   120   120    0.0%             14601      9.5%
recovery    edge runtime (40 nodes)            11611   180   180   180    0.0%              8197     17.0%

The table shows that no form dominates in every window.

A sparse window penalizes reserved capacity. For eighty-seven requests, the long-lived process reserves fifteen thousand instance-seconds and uses a thousandth of it. Per-request instance covers the same load with three hundred fifty-four instance-seconds — forty times less. Edge runtime cold-starts every request in this window, but the line item it pays is forty milliseconds.

A burst window justifies reserved capacity. The long-lived process’s utilization climbs to 95%; capacity is fully used. But the price of running at the limit is queueing: p50 stays at 123 milliseconds while p95 rises to 208 and p99 to 235. That rise comes not from the work taking longer but from a request waiting its turn. Per-request instance never disturbs p50 in the same burst — there is no queue, the instance count tracks demand.

Cold start is invisible at the median. In the burst window, the per-request instance model’s cold-start rate is 3.4%; p50 and p95 show none of it, p99 shows 470 milliseconds. Someone reading the measure from the average never notices the cold start at all. Queueing and cold start are only distinguishable at the upper percentiles.

The recovery window shows the price of the idle timeout. After the burst ends, the per-request instance model’s reserved capacity climbs to fourteen thousand six hundred instance-seconds and its utilization drops to 9.5% — below the long-lived process’s 9.8%. The reason is that the hundreds of instances spun up during the burst stay alive for their idle timeout. Capacity that tracks demand tracks it with a delay.

Edge runtime’s price is paid on every row. p50 is 180 milliseconds in the loaded windows and 220 in the sparse one; the first figure comes from paying the sixty-millisecond distance to the data store on every request, and the forty-millisecond gap in the second comes from every request falling into a cold start under sparse load. The edge’s gain from proximity is not in this table; the table measures only server-side behavior. Taken together, these line items repeat the rule from the Edge Rendering lesson: the edge pays off to the extent its data can move there too.

Where State Lives

State held in memory is meaningful only in the long-lived process, and even there only conditionally. If the instance count is greater than one, two requests can land on two different instances; a session, counter, or cache held in one instance is absent from the other.

This has three consequences. Session state is held in a shared store or lives in a signed value the client carries; the token-storage decision from the Application Architecture course meets the deployment decision at this point. Rate limits and queue counters move to the shared store; a limit counted per instance is a limit multiplied by the instance count. The connection pool is per instance; the data store’s connection limit is set by the instance count times the pool size. In the per-request instance model, this product grows fast during a burst, and it is usually the first line item to hit the store’s limit.

Always routing the same user to the same instance — sticky assignment — looks like a way to keep holding state in memory. It has a price: when the instance shuts down, that user’s state is lost and load distribution breaks. Moving state to the shared store causes neither problem.

Readiness Endpoint and Graceful Shutdown

Server-requiring deployment has a problem with no counterpart in static hosting: a release requires stopping a running process. If the requests it is holding are cut off mid-flight when the process stops, the user sees the release as an error.

The correct shutdown sequence has four steps: the readiness endpoint turns negative, a grace period is waited for the router to notice, the listener closes, and open requests are waited out.

// graceful-shutdown.mjs -- shutdown sequence of the long-lived process.
// 1) the readiness endpoint turns negative, 2) a grace period is waited for the router
// to notice, 3) the listener closes, 4) once open requests finish, the process exits.
import { createServer } from "node:http";

const WORK_DURATION = 1500;   // ms, how long the archive write takes
const READINESS_GRACE = 600;  // ms, gap between the readiness endpoint and the listener closing

let ready = true;
let openRequests = 0;

const server = createServer(async (request, response) => {
  response.sendDate = false;
  if (request.url === "/ready") {
    response.writeHead(ready ? 200 : 503, { "Content-Type": "text/plain" })
      .end(ready ? "ready\n" : "shutting down\n");
    return;
  }
  openRequests += 1;
  await new Promise((c) => setTimeout(c, WORK_DURATION));
  openRequests -= 1;
  response.writeHead(200, { "Content-Type": "text/plain" }).end("archived\n");
});

server.listen(8182, "127.0.0.1", () => console.log("server  : listening"));

process.on("SIGTERM", () => {
  ready = false;
  console.log("server  : SIGTERM received, readiness turned negative, open requests " + openRequests);
  setTimeout(() => {
    server.close(() => {
      console.log("server  : open requests " + openRequests + ", exiting process");
      process.exit(0);
    });
    console.log("server  : listener closed, no new connections accepted");
  }, READINESS_GRACE);
});

The script below runs in the same directory as this file. Port 8182 is arbitrary and must be free; if it is in use, change it in both files.

#!/usr/bin/env bash
# Measures the shutdown sequence: the open request finishes, no new connection is accepted.
node graceful-shutdown.mjs &
server=$!
sleep 0.5

curl -sS -o /dev/null \
  -w 'client  : in-flight request status %{http_code}, duration %{time_total} s\n' \
  http://127.0.0.1:8182/archive &
slow=$!
sleep 0.3

curl -sS -o /dev/null -w 'client  : readiness endpoint %{http_code} (before SIGTERM)\n' \
  http://127.0.0.1:8182/ready
kill -TERM "$server"
sleep 0.3
curl -sS -o /dev/null -w 'client  : readiness endpoint %{http_code} (after SIGTERM)\n' \
  http://127.0.0.1:8182/ready

sleep 0.6
if curl -sS -o /dev/null http://127.0.0.1:8182/archive 2> /dev/null; then
  echo "client  : new request accepted"
else
  echo "client  : new request could not connect, curl exit code $?"
fi

wait "$slow"
wait "$server"

server  : listening
client  : readiness endpoint 200 (before SIGTERM)
server  : SIGTERM received, readiness turned negative, open requests 1
client  : readiness endpoint 503 (after SIGTERM)
server  : listener closed, no new connections accepted
client  : new request could not connect, curl exit code 7
client  : in-flight request status 200, duration 1.504674 s
server  : open requests 0, exiting process

All the durations depend on the environment; what matters is the sequence. When the shutdown signal arrives, there is one open request; it completes with 200 in its full duration. After the signal, the readiness endpoint returns 503 but the port is still open — this grace period is left for the router to notice and drop this instance from its list. Once the grace period ends, the listener closes and no new connection can be made; exiting the process waits for the open request to finish.

Skipping the grace period leads to one of two failures. If the listener closes immediately, the router — not yet aware — keeps sending requests to this instance, and those requests get a connection error. If the readiness endpoint is never used, the same result happens from the first millisecond of shutdown. In the per-request instance and edge forms, this sequence is the platform’s responsibility, but the work’s duration must still be shorter than what the platform allows, so the open request is not cut off mid-flight.

Selection Rule

The decision reduces to three measures: how regular the load is, how long the work takes, and where the data lives.

If the load is regular and has no idle window, the long-lived process is chosen; reserved capacity’s utilization is high, there is no cold start, and a connection pool and an in-memory cache are used. If the load is sparse or bursty, per-request instance is chosen; reserved capacity tracks demand, at the price of cold start visible at the upper percentiles. If the work is short, has little dependency on data, and geographic proximity is decisive, edge runtime is chosen.

This rule does not mean a single choice is made once. On the North Slope site, static hosting can handle the pages, a long-lived process the archive-write endpoint, and edge runtime the routing and header-adding work. The split is made according to the work itself.

Summary

• The three hosting forms split along the axes of reserved capacity, cold start, concurrency, and state; none dominates in every load window.
• Under sparse load, the long-lived process uses a thousandth of the capacity it reserves; per-request instance covers the same load with forty times less capacity.
• In a burst, the long-lived process fully uses its capacity but queueing appears: p50 stays flat while p99 rises. Per-request instance produces no queue, only cold start.
• Cold start and queueing are invisible at the median; the measure must be read from the upper percentiles.
• State in memory is safe only on a single instance; session, counter, and cache move to the shared store, while the connection pool is multiplied by the instance count.
• Graceful shutdown has four steps: the readiness endpoint turns negative, a grace period is waited, the listener closes, open requests finish. Skip the grace period and the router keeps sending requests that then fail.

Next Step

Up to this point, two deployment forms and one shutdown sequence have been set up; all of them looked at a single target, production. But a change needs to be seen working before it reaches production, and the Development Server lesson’s last sentence gave the reason: working in development mode is not proof that it works in production mode. The next lesson sets up a separate environment for every change: a deployment with its own address, built in production mode, that does not touch production data and is not indexed. These environments’ identity is derived from the deterministic naming in the Cache Busting lesson.