Stateless Services

The Traffic Layer course brought the request to the application’s door: its name was resolved, it passed through the edge cache, it landed on a replica, it passed through the gateway and the gate. Behind the door stands an area that was never designed. Whether the services are stateless, how they find each other’s addresses, how long they wait and how many times they retry when a call does not answer, and how their contracts change were all silently assumed in that course. This course opens those assumptions.

The sequence starts at the bottom: what the service holds in its own memory. The previous course’s last measurement already showed why this is the first question — once a session sits in one replica’s memory, the balancer loses the authority to distribute the request, the replica count needed for the 0.50 utilization target rises from 3 to 4, and added replicas stop lowering the busiest replica’s load. This lesson takes up the source of that bond. A stateless service is a service that keeps no data connecting two requests in its own process memory. The stateless process and the twelve-factor principle were established in the Server-Side Foundations course; the definition is not repeated here, it is measured.

Which State Travels

Not every piece of data sitting in a process has to move. The question that separates them is this: when this data is lost, or another replica cannot see it, does the response change.

Data connecting two requests — a session, a multi-step job’s intermediate result, a pagination cursor, an idempotency key record — must travel; the response changes when it is lost. Data kept only for speed — a hot record cache, an open connection pool, compiled configuration — can stay; the response is the same when it is lost, only slower. This distinction also decides what restarting a replica means: in the first set, a restart is a data loss; in the second, it is a warm-up period.

The measurement takes up the first set. A seller integration opens a session on the tracking service and sends tracking queries one after another; the service keeps track of which shipments are being watched in that session.

Setup

The measurement is built with local processes. Two replicas do the same work and run in two modes with a single parameter: in memory mode the session list sits in the replica’s own memory, in external mode it sits in a separate state store process. No real cluster, cloud environment, or session store product is set up.

// state/store.mjs — the state store that keeps the session list outside the process (single process, in-memory)
import { createServer } from "node:http";

const sessions = new Map();
const port = Number(process.argv[2]);

if (Number.isInteger(port) === false) console.log("usage: node state/store.mjs <port>");
else createServer((request, response) => {
  const name = new URL(request.url, "http://local").pathname.split("/").pop();
  if (request.method === "PUT") {
    let m = "";
    request.on("data", (p) => (m += p));
    request.on("end", () => { sessions.set(name, JSON.parse(m)); response.writeHead(204).end(); });
    return;
  }
  response.writeHead(200, { "content-type": "application/json" });
  response.end(JSON.stringify(sessions.get(name) ?? null));
}).listen(port, "127.0.0.1");

// state/replica.mjs — tracking service replica; session list lives in memory or in the state store.
// Usage: node state/replica.mjs <port> <name> <memory|external> <store-port>
import { createServer } from "node:http";

const [port, name, mode, store] = process.argv.slice(2);
const local = new Map();

const read = async (s) => (mode === "memory" ? local.get(s) ?? null
  : (await fetch(`http://127.0.0.1:${store}/session/${s}`)).json());
const write = async (s, list) => {
  if (mode === "memory") { local.set(s, list); return; }
  await fetch(`http://127.0.0.1:${store}/session/${s}`, { method: "PUT", body: JSON.stringify(list) });
};

if (Number.isInteger(Number(port)) === false) console.log("usage: node state/replica.mjs <port> <name> <memory|external> <store-port>");
else createServer(async (request, response) => {
  const u = new URL(request.url, "http://local");
  const session = u.searchParams.get("session") ?? "-";
  const json = (code, body) => response.writeHead(code, { "content-type": "application/json" }).end(JSON.stringify({ name, ...body }));
  if (u.pathname === "/session") { await write(session, []); return json(201, { tracked: 0 }); }
  const list = await read(session);
  if (list === null) return json(409, { error: "no_session" });
  if (u.pathname === "/tracking") { list.push(u.searchParams.get("no")); await write(session, list); }
  json(200, { tracked: list.length });
}).listen(Number(port), "127.0.0.1");

The client opens the session on the first replica, then sends eight tracking requests to the replicas in turn. Sequential dispatch makes the distribution deterministic; the dispatch rule was the Traffic Layer course’s subject, and it is taken here in its plainest form.

// state/client.mjs — opens the session on the first replica, sends eight tracking requests to the replicas in turn, then reads the summary
// Usage: node state/client.mjs <label> <port...>
const [label, ...port] = process.argv.slice(2);
const get = async (n, path) => {
  const y = await fetch(`http://127.0.0.1:${n}${path}`);
  return { code: y.status, ...(await y.json()) };
};

if (port.length === 0) console.log("usage: node state/client.mjs <label> <port...>");
else {
  await get(port[0], "/session?session=S1");
  let found = 0;
  for (let i = 0; i < 8; i += 1) {
    const s = await get(port[i % port.length], `/tracking?session=S1&no=TR-${4820 + i}`);
    if (s.code === 200) found += 1;
  }
  const summary = [];
  for (const n of port) {
    const s = await get(n, "/monitor?session=S1");
    summary.push(`${s.name}=${s.code === 200 ? s.tracked : s.error}`);
  }
  console.log(`${label.padEnd(24)} session found ${found}/8   summary ${summary.join("  ")}`);
}

The driver script runs the same load in two modes and restarts one replica in each mode. Port numbers depend on the environment; change them if they are already in use on your machine.

# measure.sh — same load in two modes: state in the replica's memory and state in the state store
node state/store.mjs 8891 & D=$!
node state/replica.mjs 8881 k1 memory 8891 & A=$!
node state/replica.mjs 8882 k2 memory 8891 & B=$!
sleep 1
node state/client.mjs "in memory" 8881 8882
kill $A; sleep 1
node state/replica.mjs 8881 k1 memory 8891 & A=$!
sleep 1
echo "in memory, k1 restarted -> $(curl -s '127.0.0.1:8881/monitor?session=S1')"
kill $A $B; sleep 1

node state/replica.mjs 8881 k1 external 8891 & A=$!
node state/replica.mjs 8882 k2 external 8891 & B=$!
sleep 1
node state/client.mjs "in state store" 8881 8882
kill $A; sleep 1
node state/replica.mjs 8881 k1 external 8891 & A=$!
sleep 1
echo "in store,   k1 restarted -> $(curl -s '127.0.0.1:8881/monitor?session=S1')"
kill $A $B $D

in memory                session found 4/8   summary k1=4  k2=no_session
in memory, k1 restarted -> {"name":"k1","error":"no_session"}
in state store           session found 8/8   summary k1=8  k2=8
in store,   k1 restarted -> {"name":"k1","tracked":8}

Measured Effect

These numbers are in the measurement class and come from local processes; they are deterministic, because requests are sent in sequence.

The first row shows the bond itself: in memory mode, only four of the eight requests find the session. All the ones that find it are requests that land on the replica that opened the session; the others get no_session. The summary row carries the consequence — k1 tracks four numbers, k2 never sees the session. The list was not lost, it was split. No replica can return a split list correctly.

The second row gives the second cost: when k1 restarts, the four tracked numbers are gone too. A replica restarting here means a release, a scale change, or recovery after a failure; all three are routine events, and all three produce data loss.

The third and fourth rows give the same load in external mode. All eight requests find the session, both replicas count the same eight numbers, and the restarted replica reads the list back whole. The replicas have become each other’s equals: which replica answers no longer changes the answer.

What the measurement shows is not a performance gain — the external mode does more work. What it shows is a condition: the load being freely splittable across replicas depends on the data connecting two requests being outside the process.

Back to the Calculation

The cost of meeting the condition was already turned into a number in the Traffic Layer course’s last lesson. That lesson measured the imbalance in the sticky arrangement, found the replica count needed for the 0.50 utilization target to be 4, and wrote the largest session’s share as a serial fraction. In the stateless arrangement, these numbers are reversed.

// state/capacity.mjs — statelessness's effect on replica count and the cost of moving state outward
const PEAK_EDGE = 513.89;            // K01 Back-of-the-Envelope Estimation: peak edge req/s
const SATURATION = 400, SAFE = 200;  // K02 Role of the Load Balancer: assumption Y1 and Y1 x Y2
const SERIAL_FRACTION = 0.1980;      // K02 Session Stickiness: largest session's share
const STORE_REQUESTS = 138.89;       // K01: requests/s reaching the store
const WRITE_INGRESS = 0.17;          // K01: write ingress Mbit/s
const ARRANGEMENT = [["sticky (K02)", 3, 1.173], ["sticky (K02)", 4, 1.505],
  ["stateless", 3, 1.000], ["stateless", 4, 1.000]];

console.log(`${"arrangement".padEnd(16)}${"replica".padStart(8)}${"imbalance".padStart(12)}` +
  `${"busiest req/s".padStart(18)}${"utilization".padStart(13)}${"replicas needed".padStart(17)}`);
for (const [name, n, d] of ARRANGEMENT) {
  const busiest = (d * PEAK_EDGE) / n;
  console.log(`${name.padEnd(16)}${String(n).padStart(8)}${d.toFixed(3).padStart(12)}` +
    `${busiest.toFixed(2).padStart(18)}${(busiest / SATURATION).toFixed(3).padStart(13)}` +
    `${String(Math.ceil((d * PEAK_EDGE) / SAFE)).padStart(17)}`);
}
console.log(`\nsticky arrangement scaling upper bound = 1/${SERIAL_FRACTION} = ${(1 / SERIAL_FRACTION).toFixed(2)}x;` +
  ` no such upper bound in the stateless arrangement`);

// U1 cost: each tracking request does one read and one write of the session list (behavior in the setup)
const ops = 2 * PEAK_EDGE;
console.log(`\nU1 -> state store ops/s = ${ops.toFixed(2)}, ${(ops / STORE_REQUESTS).toFixed(2)}x ` +
  `K01's ${STORE_REQUESTS} req/s reaching the store`);
console.log(`if U1 were read-only = ${PEAK_EDGE.toFixed(2)} ops/s (${(PEAK_EDGE / STORE_REQUESTS).toFixed(2)}x)`);

// Second path: state travels in the request; the body grows
const list = (k) => JSON.stringify(Array.from({ length: k }, (_, i) => `TR-${4820 + i}`));
console.log(`\n${"numbers in list".padEnd(18)}${"session body bytes".padStart(21)}` +
  `${"added ingress Mbit/s".padStart(22)}${"ratio to K01 write ingress".padStart(27)}`);
for (const k of [8, 40]) {
  const bytes = Buffer.byteLength(list(k));
  const mbit = (PEAK_EDGE * bytes * 8) / 1e6;
  console.log(`${String(k).padEnd(18)}${String(bytes).padStart(21)}${mbit.toFixed(3).padStart(22)}` +
    `${(mbit / WRITE_INGRESS).toFixed(2).padStart(27)}`);
}

arrangement      replica   imbalance     busiest req/s  utilization  replicas needed
sticky (K02)           3       1.173            200.93        0.502                4
sticky (K02)           4       1.505            193.35        0.483                4
stateless              3       1.000            171.30        0.428                3
stateless              4       1.000            128.47        0.321                3

sticky arrangement scaling upper bound = 1/0.198 = 5.05x; no such upper bound in the stateless arrangement

U1 -> state store ops/s = 1027.78, 7.40x K01's 138.89 req/s reaching the store
if U1 were read-only = 513.89 ops/s (3.70x)

numbers in list      session body bytes  added ingress Mbit/s ratio to K01 write ingress
8                                    81                 0.333                       1.96
40                                  401                 1.649                       9.70

The numbers in the top table are in the computed value class. Because the imbalance in the stateless arrangement is 1.000, the 513.89 req/s at the edge splits evenly across three replicas: 171.30 req/s per replica and 0.428 utilization. The required replica count falls from 4 to 3 — the sticky arrangement’s bill was one replica, and it is reversed. The bottom two rows say something more important: in the stateless arrangement, adding a replica works. The fourth replica lowers the busiest replica from 171.30 to 128.47 req/s, while in the sticky arrangement the same addition had only lowered it from 200.93 to 193.35. The scaling upper bound disappears too: with a serial fraction of 0.1980 the upper bound was 5.05x, and in the stateless arrangement there is no such bound.

The cost sits in the sections below and is written with this course’s assumption; it is not added to K01’s table.

U1 — each tracking request does one read and one write of the session state. Rationale: this is the behavior in the setup, the tracking list is read and updated on every query. The sensitivity is computed below.

With U1, the state store takes 1027.78 operations per second. That is 7.40x K01’s 138.89 req/s reaching the shipment store. The size of the number is a warning: moving state out of the process does not eliminate it, it moves it somewhere else, and that somewhere is now the system’s busiest component. The sensitivity runs one way — if the session were only read, it would be 513.89 ops/s, or 3.70x; since the write is behavior-dependent, this is a number that design can shrink.

The second path is to never store the state at all: the list travels in the request itself. The table below gives the bill for that. A list of eight numbers is 81 bytes and means 0.333 Mbit/s of added ingress at 513.89 req/s — 1.96x K01’s write ingress. When the list grows to forty numbers, 401 bytes and 1.649 Mbit/s, or 9.70x. The distinction between the two paths is read from here: if the state is small and bounded it can travel in the request; if it can grow, it cannot.

Summary

• A stateless service is a service that keeps no data connecting two requests in its own process memory; the question that separates the two is whether the response changes when the data is lost.
• In memory mode, four of eight requests found the session and the list split between the two replicas (k1=4, k2 sees nothing); when the replica restarted, the list was lost entirely.
• In external mode, all eight of eight requests found the session, both replicas counted the same eight numbers, and the restarted replica read the list back whole.
• In the stateless arrangement the imbalance is 1.000: the required replicas fall from 4 to 3, the load per replica falls from 200.93 to 171.30 req/s, and the fourth replica now works (128.47 req/s).
• The scaling upper bound disappears: in the sticky arrangement a serial fraction of 0.1980 held the upper bound at 5.05x.
• The cost moves, it does not vanish: with U1 the state store takes 1027.78 ops/s (7.40x K01’s store load), or if the state travels in the request, 0.333 Mbit/s of added ingress for 81 bytes, 1.649 Mbit/s for a list of forty.

Next Step

The replicas are now each other’s equals and the load can be freely split. But one part of the measurement still stood written by hand: the driver script took the replicas’ ports from the command line, and the client carried the same numbers by hand. The Traffic Layer course’s balancer had taken the replica list the same way. When the replica count changes, when a replica drops, or when a new replica comes up, who updates that list is not written anywhere. The next lesson takes up that question: it measures where address information lives, how long it takes for a replica to enter and leave the list, and how many requests go to the wrong address in that window — and it shows why this work is a decision separate from the load balancer’s health check.