Session Stickiness

The five lessons so far never questioned one thing: the balancer’s choice was free. A request could go to any replica, because the replicas were each other’s equals. Even consistent hashing, while it wanted the choice to stay stable, did not make it mandatory — if a key landed on a different replica, the response would still be correct, only the cache would run cold.

The picture changes if state left over from a client’s previous request sits in a particular replica’s memory. That client’s next request can no longer go to just any replica; choice stops being a preference and becomes a constraint. This binding is called session stickiness. The sticky routing from the Caching, Queues and Asynchronous Processing course is the same thing: the client is always routed to the same instance. Session state itself and its management are the Server-Side Fundamentals course’s subject and are not re-derived here; this lesson’s job is to measure stickiness’s scaling bill.

Measuring the Binding

The model takes a client mix and compares three rules under the same mix. Under the non-sticky rule, the decision is made per request. Under sticky-modulo, the client identity’s hash is divided by the replica count. Under sticky-consistent, the binding is read from the consistent hashing ring from the previous lesson.

The client mix carries the shape of a real parcel tracking service: one major seller integration sends many queries from a single session, nine medium sellers send fewer, ninety minor sellers very few. The numbers are model parameters.

// sticky/model.mjs — the effect of stickiness on distribution, session loss, and scale-up.
// This is a MODEL: the client mix and request counts are model parameters.
const CLIENTS = [                  // [name prefix, client count, requests per client]
  ["major", 1, 2000], ["medium", 9, 400], ["minor", 90, 50],
];
const VIRTUAL = 200;               // consistent hashing virtual node count

function hash(text) {              // FNV-1a plus a final mix step; avalanche effect defined in
  let h = 2166136261;               // the Hash Tables lesson: similar keys must give distant values
  for (let i = 0; i < text.length; i += 1) h = Math.imul(h ^ text.charCodeAt(i), 16777619) >>> 0;
  h ^= h >>> 15; h = Math.imul(h, 2246822507) >>> 0; h ^= h >>> 13;
  return h >>> 0;
}

const clients = CLIENTS.flatMap(([name, count, requests]) =>
  Array.from({ length: count }, (_, i) => ({ name: `${name}-${i}`, requests })));
const TOTAL = clients.reduce((a, c) => a + c.requests, 0);

function ring(replicas) {
  const h = [];
  for (const k of replicas) for (let v = 0; v < VIRTUAL; v += 1) h.push([hash(`${k}#${v}`), k]);
  return h.sort((x, y) => x[0] - y[0]);
}
function onRing(a, h) {
  const c = hash(a);
  if (c > h[h.length - 1][0]) return h[0][1];
  let lo = 0, hi = h.length - 1;
  while (lo < hi) { const m = (lo + hi) >> 1; if (h[m][0] < c) lo = m + 1; else hi = m; }
  return h[lo][1];
}

function binding(rule, replicas) {   // client -> replica mapping (no mapping for non-sticky)
  if (rule === "sticky-modulo") return new Map(clients.map((c) => [c.name, replicas[hash(c.name) % replicas.length]]));
  if (rule === "sticky-consistent") { const h = ring(replicas); return new Map(clients.map((c) => [c.name, onRing(c.name, h)])); }
  return null;
}

function load(rule, replicas) {      // requests per replica
  const y = new Map(replicas.map((k) => [k, 0]));
  const b = binding(rule, replicas);
  if (b === null) {                  // per-request round robin: each request decided separately
    let s = 0;
    for (const c of clients) for (let i = 0; i < c.requests; i += 1) { const k = replicas[s++ % replicas.length]; y.set(k, y.get(k) + 1); }
  } else for (const c of clients) y.set(b.get(c.name), y.get(b.get(c.name)) + c.requests);
  return y;
}

const imbalance = (y, n) => (Math.max(...y.values()) / TOTAL) * n;
const THREE = ["k1", "k2", "k3"], FOUR = [...THREE, "k4"];

console.log(`${clients.length} clients, ${TOTAL} requests; largest client's share = ` +
  `${(Math.max(...clients.map((c) => c.requests)) / TOTAL).toFixed(4)}\n`);
console.log("rule               3 replicas load        imbalance   4 replicas load             imbalance   busiest replica drop");
for (const rule of ["non-sticky", "sticky-modulo", "sticky-consistent"]) {
  const u = load(rule, THREE), d = load(rule, FOUR);
  const pu = Math.max(...u.values()) / TOTAL, pd = Math.max(...d.values()) / TOTAL;
  console.log(`${rule.padEnd(18)} ${[...u.values()].join("/").padEnd(20)} ${imbalance(u, 3).toFixed(3).padStart(11)}   ` +
    `${[...d.values()].join("/").padEnd(25)} ${imbalance(d, 4).toFixed(3).padStart(11)}   ` +
    `${(1 - pd / pu).toFixed(4).padStart(21)}`);
}

console.log("\nclients whose binding changes when a replica is added, and their request share:");
for (const rule of ["sticky-modulo", "sticky-consistent"]) {
  const bu = binding(rule, THREE), bd = binding(rule, FOUR);
  const moved = clients.filter((c) => bu.get(c.name) !== bd.get(c.name));
  console.log(`  ${rule.padEnd(18)} clients ${String(moved.length).padStart(3)}/${clients.length}  ` +
    `request share ${(moved.reduce((a, c) => a + c.requests, 0) / TOTAL).toFixed(4)}`);
}

console.log("\nsessions lost when a replica drops, and that replica's request share (3 replicas):");
for (const rule of ["non-sticky", "sticky-modulo", "sticky-consistent"]) {
  const b = binding(rule, THREE);
  if (b === null) { console.log(`  ${"non-sticky".padEnd(18)} sessions   0/${clients.length}  request share 0.0000 (no state sits on the replica)`); continue; }
  const dropped = clients.filter((c) => b.get(c.name) === "k1");
  console.log(`  ${rule.padEnd(18)} sessions ${String(dropped.length).padStart(3)}/${clients.length}  ` +
    `request share ${(dropped.reduce((a, c) => a + c.requests, 0) / TOTAL).toFixed(4)}`);
}

100 clients, 10100 requests; largest client's share = 0.1980

rule               3 replicas load        imbalance   4 replicas load             imbalance   busiest replica drop
non-sticky         3367/3367/3366             1.000   2525/2525/2525/2525             1.000                  0.2501
sticky-modulo      2700/2250/5150             1.530   4450/2400/1800/1450             1.762                  0.1359
sticky-consistent  2200/3150/4750             1.411   1250/2850/4300/1700             1.703                  0.0947

clients whose binding changes when a replica is added, and their request share:
  sticky-modulo      clients  70/100  request share 0.7475
  sticky-consistent  clients  27/100  request share 0.1683

sessions lost when a replica drops, and that replica's request share (3 replicas):
  non-sticky         sessions   0/100  request share 0.0000 (no state sits on the replica)
  sticky-modulo      sessions  40/100  request share 0.2673
  sticky-consistent  sessions  30/100  request share 0.2178

Three Separate Bills

The tables show stickiness being paid for in three separate items.

Distribution. Under the non-sticky rule, imbalance is 1.000 — requests split by count and client identity plays no part. Under sticky-modulo it is 1.530, under sticky-consistent 1.411. The deviation’s source is that clients do not send equal request counts: the major seller’s 2000 requests land on a single replica, and the balancer has no authority to correct that. This is the same connection skew as the second lesson, except there the binding was a connection’s lifetime, here it is a session.

Replica drop. Under the non-sticky rule, a dropped replica carries no session at all; requests route to the remaining replicas and nothing is lost. Under the sticky rules, sessions bound to the dropped replica drop too: under sticky-consistent, 30 of 100 sessions and 21.78 percent of requests. The health check from the first lesson does not reduce this loss — dropping a replica from the pool gets requests answered, but it does not bring back the sessions that lived in that replica’s memory. The health check’s 34.26-request window does not apply here; the loss is measured in sessions, not a window.

Replica addition. The third table is stickiness’s most insidious item. Under sticky-modulo, when a fourth replica is added, 70 of 100 clients bind to a different replica, and those clients carry 74.75 percent of the total requests — scaling up breaks nearly three-quarters of the load’s session bindings. Under sticky-consistent the ratio drops to 27 clients and 16.83 percent — the previous lesson’s movement result repeats here in terms of sessions.

The column at the end of the same row gives the actual result. When the fourth replica is added, the busiest replica’s load drops 25.01 percent under the non-sticky rule — the figure a perfect split gives. Under the sticky rules the drop is 13.59 percent and 9.47 percent. Adding a replica does not lower the peak load, because the large session that creates the busiest replica’s load stays put. Sticky-consistent being better at preserving sessions works against it here: keeping the binding means not moving the load.

Back to the Computation

This behavior’s name was already set in C01. The Horizontal and Vertical Scaling lesson named the unsplittable part of a job the serial fraction and wrote the best possible speedup with $n$ units as:

$$S(n) = \frac{1}{s + \dfrac{1 - s}{n}}$$

Stickiness’s counterpart to the serial fraction is direct: a single session cannot be split. The largest session’s ratio to the total load stays on the busiest replica no matter how many replicas are added.

// sticky/scale.mjs — converting stickiness to a serial fraction and its counterpart in C01's peak rate
const CLIENTS = [["major", 1, 2000], ["medium", 9, 400], ["minor", 90, 50]];   // same as model.mjs
const PEAK_EDGE = 513.89;          // C01 Back-of-the-Envelope Estimation: peak edge requests/s
const SATURATION = 400;            // lesson 01 assumption Y1
const SAFE = 200;                  // lesson 01: Y1 x Y2
const MEASURED = { 3: 1.411, 4: 1.703 };   // model.mjs output: sticky-consistent imbalance

const total = CLIENTS.reduce((a, [, count, req]) => a + count * req, 0);
const s = Math.max(...CLIENTS.map(([, , req]) => req)) / total;   // the part that cannot be split

const amdahl = (n) => 1 / (s + (1 - s) / n);   // C01 Horizontal and Vertical Scaling lesson's relation
console.log(`total requests = ${total}, largest session's share (serial fraction s) = ${s.toFixed(4)}`);
console.log(`upper bound 1/s = ${(1 / s).toFixed(2)}: no matter the replica count, peak load cannot be split beyond this factor\n`);

console.log("replicas  non-sticky speedup  sticky speedup  busiest replica lower bound req/s  utilization now");
for (const n of [3, 4, 8, 16, 64]) {
  const lowerBound = PEAK_EDGE / amdahl(n);
  console.log(`${String(n).padStart(8)}  ${String(n).padStart(18)}  ${amdahl(n).toFixed(2).padStart(14)}  ` +
    `${lowerBound.toFixed(2).padStart(34)}  ${(lowerBound / SATURATION).toFixed(3).padStart(15)}`);
}
console.log(`${"infinite".padStart(8)}  ${"infinite".padStart(18)}  ${(1 / s).toFixed(2).padStart(14)}  ` +
  `${(PEAK_EDGE * s).toFixed(2).padStart(34)}  ${((PEAK_EDGE * s) / SATURATION).toFixed(3).padStart(15)}\n`);

console.log("replicas  measured imbalance  busiest replica req/s  utilization  replicas needed");
for (const [n, d] of Object.entries(MEASURED)) {
  const rate = (d * PEAK_EDGE) / n;
  console.log(`${n.padStart(8)}  ${d.toFixed(3).padStart(18)}  ${rate.toFixed(2).padStart(22)}  ` +
    `${(rate / SATURATION).toFixed(3).padStart(11)}  ${String(Math.ceil((d * PEAK_EDGE) / SAFE)).padStart(16)}`);
}
console.log(`\nsensitivity: if the largest session were 10% of the total, the upper bound would be ${(1 / 0.10).toFixed(2)}; ` +
  `at 30% it would be ${(1 / 0.30).toFixed(2)}`);

total requests = 10100, largest session's share (serial fraction s) = 0.1980
upper bound 1/s = 5.05: no matter the replica count, peak load cannot be split beyond this factor

replicas  non-sticky speedup  sticky speedup  busiest replica lower bound req/s  utilization now
       3                   3            2.15                              239.14            0.598
       4                   4            2.51                              204.79            0.512
       8                   8            3.35                              153.28            0.383
      16                  16            4.03                              127.52            0.319
      64                  64            4.75                              108.20            0.270
infinite            infinite            5.05                              101.76            0.254

replicas  measured imbalance  busiest replica req/s  utilization  replicas needed
       3               1.411                  241.70        0.604                 4
       4               1.703                  218.79        0.547                 5

sensitivity: if the largest session were 10% of the total, the upper bound would be 10.00; at 30% it would be 3.33

These numbers are in the computation class. The serial fraction is 0.1980 and the upper bound is 5.05: even if the replica count goes to infinity, the peak load cannot be split more than fivefold. The gap widens next to the non-sticky column — at 16 replicas the non-sticky rule splits the load 16-fold, the sticky rule 4.03-fold. Most of the extra replicas added do nothing for the busiest replica.

The utilization column ties this to a threshold. The first lesson chose a target utilization of 0.50 per replica (Y2). Under the sticky regime, three replicas do not hold that target: the busiest replica carries at least 239.14 req/s and 0.598 utilization. At four replicas, 204.79 and 0.512 — still above target. Reaching below target takes eight replicas, and at that point eight replicas do the work of three. Stickiness more than doubles the replica count needed, and no capacity arrives in return for the increase.

The second table runs the same computation with the measured imbalance. At three replicas the busiest replica is 241.70 req/s and 0.604 utilization; at four, 218.79 and 0.547. Adding one replica lowers the busiest replica’s load by only 9.4 percent. Holding the imbalance measured at three replicas against the safe per-replica rate implies 4 replicas are actually needed; the imbalance measured at four implies 5. The point of comparison is the first lesson’s number: without stickiness 3 replicas were enough; the second lesson’s connection skew raised that to 5; here it comes out to 4 or 5 — the session binding’s own share.

The result is this layer’s counterpart to C01’s horizontal-scaling rule. That lesson said horizontal scaling does not happen just by increasing the number of units, but because the job can be split in a way that requires no state shared between units, and it measured index partitioning’s serial fraction at 28.0 percent. Session stickiness is the traffic layer’s version of the same statement: as long as state stays in a replica’s memory, the balancer cannot split the load no matter how smart its rule is. Removing stickiness is not a balancing decision — it is a decision to move the state out of the process.

Summary

• Session stickiness binds a client’s requests to a particular replica and turns the distribution choice into a constraint; sticky routing from the Caching, Queues and Asynchronous Processing course is the same binding.
• Under the same client mix, imbalance came out to 1.000 for non-sticky, 1.530 for sticky-modulo, and 1.411 for sticky-consistent; the deviation’s source is that clients do not send equal request counts.
• When a replica drops, sticky-consistent loses 30 of 100 sessions and 21.78 percent of requests; the health check saves the request, not the session.
• When a fourth replica is added, the busiest replica’s load drops 25.01 percent under non-sticky, 13.59 percent under sticky-modulo, and 9.47 percent under sticky-consistent; keeping the binding means not moving the load.
• The largest session’s share, 0.1980, is a serial fraction and holds the scaling upper bound at 5.05: at 16 replicas the non-sticky rule splits the load 16-fold, the sticky rule 4.03-fold.
• The 0.50 utilization target set by Y2 does not hold under the sticky regime at three replicas (0.598) or at four (0.512); the measured imbalance implies 4 replicas are needed at the three-replica reading, versus 3 without stickiness.

Next Step

This topic settled which replica a request lands on. The balancer is a routing device: it makes its decision at the connection level or the content level, picks its rule by turn, queue length, or key, drops a failed replica from the pool through the health check, and loses its power to choose once state stays in a replica’s memory. The work piling up at the edge does not end there. The same point authenticates identity, applies rate limits, translates into the shape a client expects, and gathers responses from several services to satisfy a single user request. Whether this work belongs to the balancer, to the application, or to a separate layer was left open — the third option has a name, and the next topic starts with it: what a gateway, where edge responsibilities gather, should take on, and what it should not.