Session and Stickiness

The previous lesson moved state out of the application: local files, in-process counters, and anything tied to a node were pulled out, and the node became interchangeable. One piece of state tied to the user remained: the session. A session cannot be deleted, only placed somewhere. This lesson measures three placement options on one setup.

The object being measured is the loan cart of the split loan system: the books a member has reserved but not yet confirmed. The cart lives in the session, and the session lives in one of three places: process memory, a shared store, or a signed token on the client.

• SD10. The loan nodes run the same version; there is no network or hardware difference between them. The only thing that changes between measurements is where the session lives.
• SD11. The client carries a cookie: it stores the cookie from every response and sends it back on the next request.
• SD12. The shared store is a node:sqlite file on the same machine; in a real deployment it is a separate process, and the read count per request does not change.

Three Placements, One Node

The node knows all three placements; which one it uses is chosen by SESSION_PLACE. All three do the same job: read the session from the cookie, write it and return the new cookie, revoke it. What they do is the same — where they do it differs.

// node.mjs — loan cart node; where the session lives is chosen with SESSION_PLACE
import { createServer } from 'node:http';
import { createHmac, randomUUID, timingSafeEqual } from 'node:crypto';
import { DatabaseSync } from 'node:sqlite';

const NAME = process.env.NODE_NAME;
const KEY = process.env.SIGN_KEY;
const LIST = process.env.REVOKE_LIST === 'on';
const counters = { storeReads: 0, signChecks: 0 };

const db = new DatabaseSync(process.env.DB_PATH);
db.exec('PRAGMA journal_mode = WAL');
db.exec('CREATE TABLE IF NOT EXISTS session (id TEXT PRIMARY KEY, cart TEXT, revoked INTEGER)');
db.exec('CREATE TABLE IF NOT EXISTS revoked (id TEXT PRIMARY KEY)');
const read = (query, d) => { counters.storeReads += 1; return db.prepare(query).get(d); };

const sign = (body) => createHmac('sha256', KEY).update(body).digest('base64url');
const empty = () => ({ id: randomUUID(), cart: [], isNew: true });

const PLACEMENT = {
  // 1. Process memory: the session lives only in this process's heap.
  process: {
    memory: new Map(),
    read(cookie) {
      const found = this.memory.get(cookie.session);
      return found ? { id: cookie.session, cart: found, isNew: false } : empty();
    },
    write(o) { this.memory.set(o.id, o.cart); return `session=${o.id}`; },
    revoke(o) { this.memory.delete(o.id); },
  },
  // 2. Shared store: the session lives in one place every node can see.
  store: {
    read(cookie) {
      const s = cookie.session && read('SELECT cart, revoked FROM session WHERE id = ?', cookie.session);
      return !s || s.revoked ? empty() : { id: cookie.session, cart: JSON.parse(s.cart), isNew: false };
    },
    write(o) {
      db.prepare('INSERT INTO session VALUES (?, ?, 0) ON CONFLICT(id) DO UPDATE SET cart = excluded.cart')
        .run(o.id, JSON.stringify(o.cart));
      return `session=${o.id}`;
    },
    revoke(o) { db.prepare('UPDATE session SET revoked = 1 WHERE id = ?').run(o.id); },
  },
  // 3. Signed token: the session lives on the client, the node only verifies the signature.
  token: {
    read(cookie) {
      const [body, signature] = String(cookie.token ?? '').split('.');
      if (!body || !signature) return empty();
      counters.signChecks += 1;
      const expected = Buffer.from(sign(body));
      const given = Buffer.from(signature);
      if (expected.length !== given.length || !timingSafeEqual(expected, given)) return empty();
      const o = JSON.parse(Buffer.from(body, 'base64url').toString());
      if (LIST && read('SELECT 1 FROM revoked WHERE id = ?', o.id)) return empty();
      return { ...o, isNew: false };
    },
    write(o) {
      const body = Buffer.from(JSON.stringify({ id: o.id, cart: o.cart })).toString('base64url');
      return `token=${body}.${sign(body)}`;
    },
    revoke(o) { if (LIST) db.prepare('INSERT OR IGNORE INTO revoked VALUES (?)').run(o.id); },
  },
}[process.env.SESSION_PLACE];

const parseCookie = (b) => Object.fromEntries((b ?? '').split(';').map((p) => p.trim())
  .filter(Boolean).map((p) => [p.slice(0, p.indexOf('=')), p.slice(p.indexOf('=') + 1)]));

createServer((req, res) => {
  const url = new URL(req.url, 'http://local');
  const headers = { 'content-type': 'application/json' };
  if (url.pathname === '/counters') return res.writeHead(200, headers).end(JSON.stringify(counters));
  const session = PLACEMENT.read(parseCookie(req.headers.cookie));
  if (url.pathname === '/logout') {
    PLACEMENT.revoke(session);
    return res.writeHead(200, headers).end(JSON.stringify({ node: NAME, logout: true }));
  }
  if (req.method === 'POST') session.cart = [...session.cart, url.searchParams.get('book')];
  const cookie = PLACEMENT.write(session);
  res.writeHead(200, { ...headers, 'set-cookie': cookie });
  res.end(JSON.stringify({ node: NAME, cart: session.cart.length, isNew: session.isNew, cookieBytes: cookie.length }));
}).listen(Number(process.env.PORT), () => console.log(`ready ${NAME}`));

The run below picks the shared store: two additions, one read.

SESSION_PLACE=store NODE_NAME=a PORT=8719 DB_PATH=session.db \
  SIGN_KEY=library-sign-key node node.mjs > /dev/null &
SERVER=$!
sleep 0.6
curl -s -X POST -c cookie.txt 'http://127.0.0.1:8719/cart?book=K-101'; echo
curl -s -X POST -b cookie.txt -c cookie.txt 'http://127.0.0.1:8719/cart?book=K-102'; echo
curl -s -b cookie.txt 'http://127.0.0.1:8719/cart'; echo
kill $SERVER

{"node":"a","cart":1,"isNew":true,"cookieBytes":44}
{"node":"a","cart":2,"isNew":false,"cookieBytes":44}
{"node":"a","cart":2,"isNew":false,"cookieBytes":44}

The cart rose to two, the cookie stayed at 44 bytes. On a single node, the placement decision has no observable consequence; this explains why the decision gets deferred.

What Happens to the Cart When Dispatch Changes

The measurement starts with two nodes. The dispatcher spreads requests round robin; the client carries the cookie.

// dispatcher.mjs — starts nodes, dispatches a request round-robin or sticky
import { spawn } from 'node:child_process';
import { createHash } from 'node:crypto';
import { mkdtempSync, rmSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';

const root = mkdtempSync(join(tmpdir(), 'session-'));
export const cleanup = () => rmSync(root, { recursive: true, force: true });
export const stop = (d) => d.forEach((x) => x.child.kill('SIGKILL'));

export async function startNode(place, name, port, extra = {}) {
  const child = spawn(process.execPath, ['node.mjs'], { stdio: ['ignore', 'pipe', 'inherit'],
    env: { ...process.env, SESSION_PLACE: place, NODE_NAME: name, PORT: String(port),
           SIGN_KEY: 'library-sign-key', DB_PATH: join(root, 'session.db'), ...extra } });
  await new Promise((c) => child.stdout.once('data', c));
  return { name, port, child };
}

// Client that carries a cookie jar: stores the cookie from each response, sends it back on the next.
export function client(id) {
  const jar = new Map();
  return { id, async call(node, path) {
    const cookie = [...jar].map(([a, d]) => `${a}=${d}`).join('; ');
    const res = await fetch(`http://127.0.0.1:${node.port}${path}`,
      { method: path.includes('book=') ? 'POST' : 'GET', headers: cookie ? { cookie } : {} });
    for (const raw of res.headers.getSetCookie()) {
      jar.set(raw.slice(0, raw.indexOf('=')), raw.slice(raw.indexOf('=') + 1));
    }
    return res.json();
  } };
}

export const roundRobin = (d, n) => d[n % d.length];
export const sticky = (d, id) => d[createHash('sha256').update(id).digest()[0] % d.length];

The first measurement has a single member add eight books through eight round-robin requests. The second turns on sticky dispatch — the same member always reaches the same node — and raises node count to three.

// placement-measurement.mjs — eight additions dispatched round robin, then a third node under sticky dispatch
import { startNode, stop, cleanup, client, roundRobin, sticky } from './dispatcher.mjs';

const BOOKS = ['K-101', 'K-102', 'K-103', 'K-104', 'K-105', 'K-106', 'K-107', 'K-108'];
const MEMBERS = Array.from({ length: 12 }, (_, i) => `U-${String(i + 1).padStart(4, '0')}`);

async function roundRobinMeasure(place) {
  const d = [await startNode(place, 'a', 8710), await startNode(place, 'b', 8711)];
  const member = client('U-0001');
  const reported = [];
  for (const [i, book] of BOOKS.entries()) {
    reported.push((await member.call(roundRobin(d, i), `/cart?book=${book}`)).cart);
  }
  const last = await member.call(d[0], '/cart');
  stop(d);
  return `${place.padEnd(10)} ${reported.join(' ').padEnd(24)} ${last.cart}`;
}

async function stickyMeasure(place) {
  const two = [await startNode(place, 'a', 8712), await startNode(place, 'b', 8713)];
  const clients = MEMBERS.map(client);
  for (const member of clients) {
    for (const book of ['K-201', 'K-202', 'K-203']) await member.call(sticky(two, member.id), `/cart?book=${book}`);
  }
  const three = [...two, await startNode(place, 'c', 8714)];
  let moved = 0, emptied = 0;
  for (const member of clients) {
    if (sticky(three, member.id).name !== sticky(two, member.id).name) moved += 1;
    if ((await member.call(sticky(three, member.id), '/cart')).cart === 0) emptied += 1;
  }
  stop(three);
  return `${place.padEnd(10)} ${String(moved).padEnd(9)} ${String(emptied).padEnd(9)} ${MEMBERS.length - emptied}`;
}

console.log('round-robin dispatch, one member, eight additions (no request returned an error)');
console.log('placement   cart reported while adding  final read');
for (const place of ['process', 'store', 'token']) console.log(await roundRobinMeasure(place));
console.log('');
console.log('sticky dispatch, twelve members, node count rises from two to three');
console.log('placement   moved     emptied   cart intact');
for (const place of ['process', 'store', 'token']) console.log(await stickyMeasure(place));
cleanup();

round-robin dispatch, one member, eight additions (no request returned an error)
placement   cart reported while adding  final read
process    1 1 1 1 1 1 1 1          0
store      1 2 3 4 5 6 7 8          8
token      1 2 3 4 5 6 7 8          8

sticky dispatch, twelve members, node count rises from two to three
placement   moved     emptied   cart intact
process    7         7         5
store      7         0         12
token      7         0         12

The first table’s process row is this lesson’s core. Eight requests were sent, all returned successfully, and each reported one book in the cart: every request arrives with a cookie the previous node does not know, so the node opens a new session and adds the book to it. The cart never accumulates, and the final read shows zero. The application reported no loss, because from its own point of view there is none: every request ended with a valid session.

The second table shows what stickiness buys and what it does not. Under sticky dispatch, process memory looks like it works; the problem shows up when the node set changes. Adding a third node moved seven of twelve members to a different node, and in the process-memory placement, those seven carts emptied. Moved-member count is the same across all three placements — the dispatch rule is the same rule — but emptied-cart count is nonzero only for process memory. Stickiness ties a session to a node; the moment the node set changes, the tie breaks. A release changes the node set too.

The Bill Paid per Request

The shared store and the signed token gave the same result in the second table. Their differences show up in three places: per-request work, bytes carried in the cookie, and what happens after logout.

// cost.mjs — per-request verification work, cookie size, and requests passing after logout
import { startNode, stop, cleanup, client } from './dispatcher.mjs';

const REQUESTS = 200;
const SIZES = [1, 8, 32, 128];

async function measure(place, extra = {}, name = place) {
  const node = await startNode(place, 'a', 8715, extra);
  const member = client('U-0001');
  const size = new Map();
  for (let i = 1; i <= REQUESTS; i += 1) {
    const res = await member.call(node, `/cart?book=K-${300 + i}`);
    if (SIZES.includes(i)) size.set(i, res.cookieBytes);
  }
  const counters = await fetch(`http://127.0.0.1:${node.port}/counters`).then((y) => y.json());
  await member.call(node, '/logout');
  let passed = 0;
  for (let i = 0; i < 5; i += 1) if ((await member.call(node, '/cart')).cart > 0) passed += 1;
  stop([node]);
  return { name, ...counters, size, passed };
}

const o = [await measure('process'), await measure('store'), await measure('token'),
           await measure('token', { REVOKE_LIST: 'on' }, 'token+list')];

console.log(`work done across ${REQUESTS} requests, and five requests accepted after logout`);
console.log('placement       store reads  sign checks    passed after logout');
for (const s of o) {
  console.log(`${s.name.padEnd(15)} ${String(s.storeReads).padEnd(11)} ${String(s.signChecks).padEnd(15)} ${s.passed} / 5`);
}
console.log('');
console.log('bytes carried in the cookie, by books in the cart');
console.log(`placement       ${SIZES.map((n) => `${n} book${n === 1 ? '' : 's'}`.padEnd(10)).join('')}`);
for (const s of o) console.log(`${s.name.padEnd(15)} ${SIZES.map((n) => String(s.size.get(n)).padEnd(10)).join('')}`);

work done across 200 requests, and five requests accepted after logout
placement       store reads  sign checks    passed after logout
process         0           0               0 / 5
store           199         0               0 / 5
token           0           199             5 / 5
token+list      199         199             0 / 5

bytes carried in the cookie, by books in the cart
placement       1 book    8 books   32 books  128 books
process         44        44        44        44
store           44        44        44        44
token           133       208       464       1488
token+list      133       208       464       1488

Two hundred requests counted 199 reads or 199 checks; the one missing is the first, which arrives with no cookie. The numbers are run-independent: the store does one read per request, the token one check per request. Their costs are not paid in the same place — a store read travels over the network and piles onto the store as node count grows, while a signature check scales along with the node.

The cookie column is the token’s bill. In the two placements that settle for an identifier, the cookie stays at 44 bytes no matter how large the cart grows; with the token, once the cart reaches 128 books the cookie reaches 1488 bytes, traveling in both directions on every request.

The last column is where the decision gets made. Of the five requests sent after logout, the shared store accepted zero, the signed token all five: the node verifies the signature, the signature is valid, the session lives on. A token cannot be revoked on its own. Turning on REVOKE_LIST makes revocation work — but the token+list row shows both 199 checks and 199 store reads: revocability hands back the token’s advantage of avoiding the store.

The Silence of a Wrong Key

The token placement’s correct setting is that every node carries the same signing key. What happens when one of three nodes is given a different key can be counted.

// wrong-key.mjs — what happens if one of three nodes is given a different signing key
import { startNode, stop, cleanup, client, roundRobin } from './dispatcher.mjs';

async function measure(thirdKey) {
  const d = [await startNode('token', 'a', 8716), await startNode('token', 'b', 8717),
             await startNode('token', 'c', 8718, { SIGN_KEY: thirdKey })];
  const member = client('U-0001');
  let reset = 0, errorCode = 0;
  const trace = [];
  for (let i = 0; i < 12; i += 1) {
    const res = await member.call(roundRobin(d, i), `/cart?book=K-${400 + i}`);
    if (res.isNew && i > 0) reset += 1;
    trace.push(`${res.node}:${res.cart}`);
  }
  stop(d);
  return { reset, errorCode, trace: trace.join(' ') };
}

for (const [label, key] of [['correct setting', 'library-sign-key'],
                             ['wrong key      ', 'old-sign-key']]) {
  const s = await measure(key);
  console.log(`${label}: silently reset sessions ${s.reset}/11, requests returning an error code ${s.errorCode}/12`);
  console.log(`  node:cart  ${s.trace}`);
}
cleanup();

correct setting: silently reset sessions 0/11, requests returning an error code 0/12
  node:cart  a:1 b:2 c:3 a:4 b:5 c:6 a:7 b:8 c:9 a:10 b:11 c:12
wrong key      : silently reset sessions 7/11, requests returning an error code 0/12
  node:cart  a:1 b:2 c:1 a:1 b:2 c:1 a:1 b:2 c:1 a:1 b:2 c:1

The trace row gives the difference at a glance: with the correct setting the cart counts from one to twelve; with the wrong key it never gets past two, resetting on every third request. All twelve requests succeeded in both runs; the error-code count is zero. The node with the different key reports no error — it treats a token it cannot verify as unrecognized, opens a new session, then sends back a cookie signed with its own key, which the other two nodes do not recognize either. The setting error shows up as a lost cart.

Where the Setting Lives

The placement decision is not a single setting; how many places it spreads to depends on the placement.

Setting	Layer	Where it stands
Session placement	application code	one per node
Stickiness rule	request dispatcher	a single place
Store address	environment variable	one per node
Signing key	environment variable	one per node

In a four-node setup, process memory needs five setting points (four nodes plus the dispatcher’s stickiness rule), the shared store eight, the signed token eight, the token with a revoke list twelve; at twenty nodes: twenty-one, forty, forty, and sixty. What matters is not the total but the “one per node” rows: a value staying different on a single node produces the silent result above. This is why the signing key and store address get distributed from a single source, not typed in by hand per node.

Summary

• The cart lives in the session; the session is placed in one of three spots: process memory, shared store, signed token on the client. On a single node, all three behave alike.
• Eight additions dispatched round robin never accumulated a cart in process memory: all eight requests succeeded, the final read showed zero, none reported an error.
• When node count rose from two to three, seven of twelve members moved; those seven carts emptied in process memory, zero in the other two placements.
• The shared store does one read per request and supports revocation; the signed token does no reads but grows the cookie from 44 to 1488 bytes and accepts all five requests sent after logout. Adding a revoke list brings the store read back.
• When one of three nodes was given a different signing key, seven of eleven requests silently opened a new session; the error-code count stayed at zero.

Next Step

This lesson’s last measurement raised node count from two to three and counted what that did to sessions. Scaling is not the only event that changes the node set: every release also stops nodes, starts new ones, and runs both together for a while. The next lesson measures the release itself: how many requests drop when a version changes at a fixed request rate, how many seconds of downtime that equals, how much draining open connections lowers that number, and how many steps a rollback takes.