Load Shedding

The previous lesson built backpressure and ended on a limit: backpressure works in the system’s internal links, it cannot be carried to the client sitting at the outer end. A server handling requests cannot hold a request indefinitely because the background queue is full.

So what does it do under overload. The answer is often skipped because it is hard to accept: reject the request. Rejecting is not a malfunction, it is a design decision. A server that does not reject keeps accepting requests, every one of their latencies grows, and past a certain point no request is served on time. This state is called congestion collapse, and it is far worse than rejecting: a rejected request was at least rejected quickly. This lesson builds how rejection is done in a controlled way and compares the latency of the two states by measuring it.

Admission Control

Load shedding is the system refusing work that exceeds its capacity instead of accepting it. The structure that makes this decision is called admission control, and it runs before a job starts.

Admission control differs from rate limiting on two points. Rate limiting bounds how many requests a specific client can make in a specific time; its criterion is the client, and its purpose is fair sharing. Admission control’s criterion is the system’s current state; its purpose is protecting the timely completion of jobs already accepted. The same client is accepted when the system is idle and rejected when it is saturated.

What the Threshold Is Set On

Setting the threshold on requests per second is common but misleading. Requests per second does not account for job duration: the same rate is comfortable if a job takes twenty milliseconds and overwhelming if it takes three hundred.

The right criterion is concurrency: the number of jobs being processed at once. The $L = \lambda W$ relation used in the previous lesson gives this directly: if the number of jobs in the system, $L$, is fixed, the average duration $W$ is fixed as well. Fixing concurrency fixes latency.

The server below carries two thresholds: the number of jobs that can be processed at once, and the number of jobs that may wait in the queue. Their sum is the largest load the system accepts.

// admission.mjs — job server; runs in two modes
//   node admission.mjs --unbounded    every incoming request is queued
//   node admission.mjs --controlled   once the threshold is crossed the request is turned back with 503
import { createServer } from "node:http";

const CONTROLLED = process.argv.includes("--controlled");
const MAX_CONCURRENT = 2;          // number of jobs processed at once
const MAX_QUEUE = 2;               // number of jobs that may wait in the queue
const JOB_DURATION = 300;          // duration of a job (ms)

let running = 0;
const queue = [];

const doWork = () => new Promise((c) => setTimeout(c, JOB_DURATION));

async function start(job) {
  running++;
  await doWork();
  running--;
  job.response.writeHead(200, { "content-type": "text/plain" });
  job.response.end(`${job.status}\n`);
  const next = queue.shift();
  if (next) start(next);
}

createServer((request, response) => {
  response.sendDate = false;
  if (running < MAX_CONCURRENT) return start({ response, status: "accepted" });
  if (!CONTROLLED || queue.length < MAX_QUEUE)
    return queue.push({ response, status: "queued" });
  // Threshold crossed: the request is not held; when to retry is communicated instead.
  response.writeHead(503, { "content-type": "text/plain", "retry-after": "1" });
  response.end("shed\n");
}).listen(8352, "127.0.0.1", () => console.log("listening: 127.0.0.1:8352"));

Measurement

The script below brings the server up in both modes, sends ten requests at the exact same time in each, and groups the results by status code and hundred-millisecond latency slices. Port 8352 is arbitrary and must be free.

#!/usr/bin/env bash
# Starts admission.mjs in two modes, sends ten requests at once in each mode, summarizes the result.
A=http://127.0.0.1:8352

measure() {
  node admission.mjs "$1" > /dev/null & server=$!
  sleep 1
  pid=()
  for i in $(seq 1 10); do
    curl -sS -o /dev/null -w '%{http_code} %{time_total}\n' "$A/job" & pid+=($!)
  done > raw.txt 2>&1
  wait "${pid[@]}"
  echo "--- $2 ---"
  awk '{ printf "status=%s  latency=%2d slice\n", $1, int($2 * 10) }' raw.txt | sort | uniq -c
  kill "$server"
  sleep 0.5
}

measure --unbounded  "no admission control: every request is queued"
measure --controlled "admission control on: 503 once the threshold is crossed"

--- no admission control: every request is queued ---
   2 status=200  latency= 3 slice
   2 status=200  latency= 6 slice
   2 status=200  latency= 9 slice
   2 status=200  latency=12 slice
   2 status=200  latency=15 slice
--- admission control on: 503 once the threshold is crossed ---
   2 status=200  latency= 3 slice
   2 status=200  latency= 6 slice
   6 status=503  latency= 0 slice

The slice count is a hundred-millisecond unit; because a job takes three hundred milliseconds, the first pair gets a response in three slices, the second pair in six. On a slower machine the slice counts may shift up by one.

Both modes do the same job, but their latency distributions are opposite.

In the uncontrolled mode no request was rejected, and nobody was satisfied: the last two requests waited a second and a half. If the load had been a hundred requests instead of ten, the last request’s latency would have been fifteen seconds. Every accepted request carries the latency of everyone accepted before it.

In the controlled mode four requests were served, six were rejected in slice zero, meaning without being held at all. The accepted requests’ latency stayed at three and six slices; the system kept the promise it made to the requests it accepted.

The difference in one sentence: admission control turns an uncertain wait into a certain rejection. This is an improvement for the client, because it has a behavior in response to rejection — retrying, deferring the job, informing the user. It has no behavior in response to an uncertain wait.

The Shape of the Rejection

How the rejection is communicated matters as much as the decision to reject. The script below drives the server to saturation and shows the raw response to one request.

#!/usr/bin/env bash
# Shows the headers of the rejection response from a server that has reached saturation.
node admission.mjs --controlled > /dev/null & server=$!
sleep 1
A=http://127.0.0.1:8352

pid=()
for i in 1 2 3 4; do curl -sS -o /dev/null "$A/job" & pid+=($!); done   # 2 running + 2 queued
sleep 0.1
echo "--- response to a request arriving at saturation ---"
curl -sS -D - "$A/job"
wait "${pid[@]}"
kill "$server"

--- response to a request arriving at saturation ---
HTTP/1.1 503 Service Unavailable
content-type: text/plain
retry-after: 1
Connection: keep-alive
Keep-Alive: timeout=5
Transfer-Encoding: chunked

shed

Three details are deliberate choices. The status code is 503, not 500: 500 says the server is at fault, 503 says it is temporarily unable to serve. This difference determines the client’s behavior — a client receiving 500 should not retry, one receiving 503 should. The Retry-After header states, in seconds, when that retry should happen; the client does not need to guess. And the body is short: a saturated server producing a long error body is exactly the work it is trying to avoid.

The connection headers depend on the runtime’s default settings.

What to Shed

Up to here, the request rejected was the one crossing the threshold. This is the simplest policy, and it has one flaw: it treats all jobs as equal. Yet in the library service a loan return and a monthly report request are not equal.

Two criteria give this distinction. In shedding by priority, every job is given a class, and at saturation the lowest class is rejected first; a concurrency share reserved for the critical class is always kept free. In shedding by deadline, every job arrives with a latency budget, and a job whose budget expires while waiting in the queue is dropped, unprocessed, when its turn comes. The second is especially valuable because it catches the case where the job at the head of the queue has already become meaningless — the client may have disconnected.

In both policies, the shedding decision must be fed by a measured indicator, not a fixed threshold: time spent in the queue, worker saturation, the latency of an external dependency. A fixed threshold sheds either too early or too late once the shape of the load changes.

Retry Storm

Shedding itself produces a new hazard. Six clients rejected at the same time come back at the same time once their Retry-After duration expires. This produces a loop that keeps the server saturated, and it is called a retry storm.

The antidote was already built earlier in this curriculum: the exponential backoff and jitter introduced in the messaging topic. The Retry-After value is a base duration; the client adds a random jitter to that base and grows the duration on every failed attempt. A client-side upper attempt limit is also required, or the rejected job comes back forever.

There is a countermeasure on the server side too: logging a rejected request must be cheaper than logging an accepted one. Logging every rejection in detail under saturation spends the capacity load shedding just gained on the logging work itself.

Summary

• Load shedding is refusing work that exceeds capacity instead of accepting it; admission control, which makes this decision, runs before a job starts.
• The threshold is set on concurrency, not requests per second; per $L = \lambda W$, fixing the number of jobs in the system fixes latency.
• The measurement showed latency spreading from 3 slices to 15 slices in the uncontrolled server, and staying at 3–6 slices for accepted jobs in the controlled server, with the rest rejected in slice zero.
• Admission control turns an uncertain wait into a certain rejection; 503 with Retry-After tells the client what to do, while 500 reports an error that should not be retried.
• Shedding produces a retry storm; the antidote is exponential backoff with jitter on the client side and keeping rejection cheap on the server side.

Next Step

Everything built up to here concerned the job’s server-side durability: a job falls to a worker, fires on time, has its progress recorded, does not overflow the queue, and is rejected in a controlled way under overload. One end of this chain is still open. The client that started the job learned it was accepted but does not know when the result will be ready. The progress percentage of a long report, the change in position in a reservation queue, the update of a branch inventory counter — all of it is ready on the server side and invisible on the client side. The next topic covers how this information reaches the client, starting with the simplest route: the client asking over and over.