Taints and Tolerations

All of the previous lesson’s constraints looked in the same direction: the pod wanted something from the node. The reverse direction was never written. In the regional measurement network (the running fiction), the nightly batch job saturates the disk and the memory; a service copy that lands on the same node slows down for the whole night. The way to set these nodes aside for the batch job is not to write a “do not go to these nodes” constraint into each of the eighteen pods — that solution has to be rewritten for every new service, and one forgotten declaration breaks the separation.

A taint is placed on the node and flips the default: the node rejects every pod that does not say otherwise. A toleration is written on the pod and lifts only that rejection. For a pod to arrive, both declarations must exist, and yet the two together are not enough — a toleration permits, it does not attract. What still carries a pod to the set-aside node is the previous lesson’s label selection.

What is measured is a model; no real cluster runs.

The Condition the Node Sets

The model is nine nodes, eighteen general pods, and six batch pods. The tainted-node count is scanned from zero to five, and at each step three things are counted: how many nodes the general workload fits into, how many pods fail to place, and how much of the set-aside capacity sits idle.

// scheduling/taint.mjs — node separation: taints and tolerations. Not a real cluster, a model.
// A taint is placed on the node, a toleration is written on the pod; a toleration permits, it does not attract.
export const NODE_COUNT = 9, CAPACITY = 1500;   // SS13: nine nodes, 1500 units per node
export const TAINT_ORDER = [0, 3, 6, 1, 4];     // tainting order: one from each zone first

export const makeNodes = (tainted, effect = "block") =>
  Array.from({ length: NODE_COUNT }, (_, i) => ({
    name: `d${i}`, capacity: CAPACITY, used: 0, pods: [],
    taint: TAINT_ORDER.slice(0, tainted).includes(i) ? { key: "batch", effect } : null,
  }));

// SS14: general workload and the nightly batch job (regional measurement network, the running fiction)
export const GENERAL = [["reading-collector", 9, 420], ["verifier", 6, 650],
                        ["billing", 3, 300]];
export const BATCH = [["nightly-batch", 6, 700]];
export const makePods = (list, extra = {}) => list.flatMap(([name, n, unit]) =>
  Array.from({ length: n }, (_, i) => ({ name: `${name}-${i + 1}`, service: name, unit, ...extra })));

// The taint's first effect: a pod with no toleration cannot be placed on a tainted node.
// The third effect (deprioritize) pushes a tainted node to last resort; it does not forbid it.
export function place(podList, nodes) {
  const unplaced = [], misplaced = [];
  for (const k of podList) {
    const eligible = nodes.filter((d) => {
      if (d.used + k.unit > d.capacity) return false;
      if (k.target && d.taint?.key !== k.target) return false;
      return d.taint === null || k.toleration === true || d.taint.effect === "deprioritize";
    });
    const preferred = eligible.filter((d) => d.taint === null || d.taint.effect !== "deprioritize" ||
      k.toleration === true);
    const d = (preferred.length > 0 ? preferred : eligible)[0];
    if (d === undefined) { unplaced.push(k); continue; }
    d.used += k.unit; d.pods.push(k);
    if (d.taint !== null && k.service !== "nightly-batch") misplaced.push(k);
  }
  return { nodes, unplaced, misplaced };
}

export const occupied = (nodes) => nodes.filter((d) => d.pods.length > 0).length;
export const TOTAL_GENERAL = GENERAL.reduce((t, [, n]) => t + n, 0);

if (process.argv[1].endsWith("taint.mjs")) {
  const gb = GENERAL.reduce((t, [, n, b]) => t + n * b, 0);
  const tb = BATCH.reduce((t, [, n, b]) => t + n * b, 0);
  console.log(`model: ${NODE_COUNT} nodes x ${CAPACITY} units = ${NODE_COUNT * CAPACITY}; ` +
    `general workload ${gb} units / ${TOTAL_GENERAL} pods, nightly batch job ${tb} units / 6 pods`);
  console.log("SS15: the batch job runs 4 hours a day; a set-aside node waits the day's full 24 hours\n");
  const s = (x, n = 16) => String(x).padStart(n);
  console.log(`${"tainted".padEnd(10)}${s("general placed")}${s("unplaced")}` +
    `${s("general nodes")}${s("batch placed")}${s("reserved h/day")}${s("idle ratio")}`);
  for (let k = 0; k <= 5; k++) {
    const d = makeNodes(k);
    const g = place(makePods(GENERAL), d);
    const generalNodes = occupied(d);
    const t = place(makePods(BATCH, { toleration: true, target: "batch" }), d);
    const batchNodes = d.filter((x) => x.pods.some((y) => y.service === "nightly-batch")).length;
    const reserved = k * 24, used = batchNodes * 4;
    console.log(`${String(k).padEnd(10)}${s(TOTAL_GENERAL - g.unplaced.length)}` +
      `${s(g.unplaced.length)}${s(generalNodes)}${s(6 - t.unplaced.length)}${s(reserved)}` +
      `${s(k === 0 ? "-" : (100 * (1 - used / reserved)).toFixed(1) + "%")}`);
  }
}

model: 9 nodes x 1500 units = 13500; general workload 8580 units / 18 pods, nightly batch job 4200 units / 6 pods
SS15: the batch job runs 4 hours a day; a set-aside node waits the day's full 24 hours

tainted     general placed        unplaced   general nodes    batch placed  reserved h/day      idle ratio
0                       18               0               7               0               0               -
1                       18               0               7               2              24           83.3%
2                       18               0               7               4              48           83.3%
3                       15               3               6               6              72           83.3%
4                       13               5               5               6              96           87.5%
5                       11               7               4               6             120           90.0%

Two Bills for the Separation

The first row gives, in numbers, the difference between a toleration and label selection. With no tainted node, none of the batch job’s six pods get placed, because the batch pods carry a target that seeks a tainted node, and no such node exists. Their tolerations sit unused; a toleration alone carries no pod anywhere.

The first bill belongs to the general workload. With one or two nodes set aside, the general workload still fits into seven nodes and all eighteen pods get placed; once three nodes are set aside, it is squeezed into six and three pods fail to place. Once five nodes are set aside, the unplaced count rises to seven. Setting nodes aside is an isolation decision, and its price is paid directly from the general workload’s capacity.

The second bill is in the set-aside capacity itself. The batch job runs four hours a day, while a set-aside node waits the full twenty-four hours: the idle ratio is 83.3 percent in the best case. This ratio is not the inefficiency of separation but its definition — separation sets aside a node, not time. Once more than three nodes are set aside, the ratio only worsens (87.5 percent, then 90 percent), because the batch job never uses more than three nodes and every extra set-aside node spends its entire day idle. With three nodes set aside, 72 node-hours a day are reserved while 12 node-hours are used.

These two bills must be read together with the declaration cost of the separation. Building the same separation on the pod side requires writing a constraint into each of the eighteen general pods; building it with a taint takes three node taints and six tolerations — nine items in all. The difference is not only in the count but in who can forget: in the pod-side setup, a new service that forgets to write the constraint silently breaks the separation; in the node-side setup, a service that forgets it only fails to enter the set-aside nodes. A separation built in the wrong direction can be broken again by every new service.

The idle ratio also comes down to a single number. Because a set-aside node waits the entire day, the ratio follows directly from the width of the window: 83.3 percent for a four-hour window, 66.7 percent for an eight-hour window, 95.8 percent for a one-hour window. The decision to set a node aside is therefore not a resource decision but a window decision; the shorter the batch job’s duration, the larger the set-aside node’s cost grows in proportion.

The Taint’s Three Effects and a Wrong Toleration

A taint does not do just one thing. The same taint can be placed with three separate effects: blocking a new pod from being placed, evicting pods that are already running too, or only lowering the preference order. The three are measured separately, and then what happens when a toleration is given incorrectly is scanned. The distinction among the three effects is not merely a setting: the first binds future placements, the second binds pods already running, the third binds only ordering. The same separation intent yields three separate outcomes, and the difference between them is only visible once it is counted.

// scheduling/toleration.mjs — the taint's three effects, one at a time, and the cost of a wrongly given toleration.
import { GENERAL, BATCH, TAINT_ORDER, TOTAL_GENERAL, makeNodes, makePods, place } from "./taint.mjs";

const TAINTED = 3;                     // SS16: three nodes are set aside for the nightly batch job
const generalPods = (tolerating = 0) =>
  makePods(GENERAL).map((k, i) => (i < tolerating ? { ...k, toleration: true } : k));
const batchPods = () => makePods(BATCH, { toleration: true, target: "batch" });
const s = (x, n = 16) => String(x).padStart(n);

// Second effect: if the taint is placed AFTER pods are already running, those without a toleration are evicted.
function taintAfterward(tolerating = 0) {
  const d = makeNodes(0);
  place(generalPods(tolerating), d);
  const set = new Set(TAINT_ORDER.slice(0, TAINTED));
  const evicted = [];
  d.forEach((x, i) => {
    if (!set.has(i)) return;
    x.taint = { key: "batch", effect: "evict" };
    evicted.push(...x.pods.filter((k) => k.toleration !== true));
    x.pods = x.pods.filter((k) => k.toleration === true);
    x.used = x.pods.reduce((t, k) => t + k.unit, 0);
  });
  const requeued = place(evicted, d);
  const retained = d.filter((x, i) => set.has(i))
    .reduce((t, x) => t + x.pods.filter((k) => k.toleration === true).length, 0);
  return { evicted: evicted.length, unplaced: requeued.unplaced.length, retained };
}

console.log(`the taint's three effects, one at a time (${TAINTED} nodes set aside, toleration given correctly):`);
console.log(`${"effect".padEnd(24)}${s("general placed")}${s("misplaced")}${s("evicted")}` +
  `${s("unplaced")}`);
for (const effect of ["block", "deprioritize"]) {
  const d = makeNodes(TAINTED, effect);
  const g = place(generalPods(), d);
  console.log(`${(effect === "block" ? "placement block" : "deprioritization").padEnd(24)}` +
    `${s(TOTAL_GENERAL - g.unplaced.length)}${s(g.misplaced.length)}${s(0)}${s(0)}`);
}
const t2 = taintAfterward();
console.log(`${"eviction block".padEnd(24)}` +
  `${s(TOTAL_GENERAL - t2.unplaced)}${s(0)}${s(t2.evicted)}${s(t2.unplaced)}`);

// SS17: a toleration spreads by copy-paste. The count of wrongly given tolerations is scanned.
console.log(`\nwrongly given toleration (${TAINTED} nodes set aside, placement block):`);
console.log(`${"general tolerating".padEnd(20)}${s("misplaced")}${s("general placed")}` +
  `${s("batch placed")}${s("retained")}`);
for (const n of [0, 2, 4, 6, 9]) {
  const d = makeNodes(TAINTED);
  const g = place(generalPods(n), d);
  const t = place(batchPods(), d);
  const retained = taintAfterward(n);
  console.log(`${String(n).padEnd(20)}${s(g.misplaced.length)}` +
    `${s(TOTAL_GENERAL - g.unplaced.length)}${s(6 - t.unplaced.length)}` +
    `${s(retained.retained)}`);
}

// SS18: declarations read for the decision. The taint sits on the node, the toleration on the pod; both are declarations.
const items = TAINTED + TOTAL_GENERAL + 6;
console.log(`\ndeclarations read: ${TAINTED} node taints + ${TOTAL_GENERAL + 6} pod toleration ` +
  `fields = ${items}; no item measures their correctness`);

the taint's three effects, one at a time (3 nodes set aside, toleration given correctly):
effect                    general placed       misplaced         evicted        unplaced
placement block                       15               0               0               0
deprioritization                      18               3               0               0
eviction block                        15               0               8               3

wrongly given toleration (3 nodes set aside, placement block):
general tolerating         misplaced  general placed    batch placed        retained
0                                  0              15               6               0
2                                  2              18               4               2
4                                  3              18               4               3
6                                  3              18               4               3
9                                  3              18               4               3

declarations read: 3 node taints + 24 pod toleration fields = 27; no item measures their correctness

The Three Effects, One at a Time

The placement block binds only the future: fifteen general pods get placed, not a single general pod sits on the set-aside nodes, and nothing is evicted. The separation is clean, and its price is three unplaced pods.

The eviction block binds today. When the taint is placed after the pods are already running, eight pods are evicted from their nodes; when they try to place back into the remaining capacity, three of them find no room. The same final state (fifteen pods standing) is reached by both paths, but the second path costs eight pods a restart. When a taint is placed matters as much as which effect it is placed with.

Deprioritization blocks nothing. All eighteen of the eighteen pods get placed, but three general pods land on the set-aside nodes. A soft taint works only until capacity gets tight, and it dissolves on its own exactly when capacity gets tight — exactly when the separation is needed. A separation needs to be an enforcement, not a note.

Read together, the three effects’ numbers also clarify the scope of the delegated decision. The authority handed to the system covers not only “where to place” but also “whom to evict” and “whom to deprioritize.” All three rest on a single field — the effect of the taint on the node — and when this field is chosen wrongly, the result is either a separation that stays on paper (deprioritization) or a sudden wave of eight pod restarts (eviction block). What produces the difference between the two is not the state of the cluster but a single-word declaration.

A Wrong Toleration Makes the Error Look Like an Improvement

The final table counts the price of a toleration that spreads by copy-paste. Once an unnecessary toleration is written onto two general pods, those pods land on the set-aside nodes and two of the nightly batch job’s six pods become unplaceable. Once four or more pods carry the toleration, the number of foreign pods on the set-aside nodes saturates at three.

This table’s dangerous column is the middle one. Once the wrong toleration is given, the general placed count rises from fifteen to eighteen. No indicator that looks at the placement decision reports a problem: more pods have been placed, no pod is left unsolvable, the cluster’s occupancy has gone up. The only thing broken is the separation itself, and it only becomes visible at night, when the batch job runs with pods missing. The last column adds a second loss: once the eviction block is placed, these pods are not evicted, because their tolerations protect them. A wrong declaration also stands in the way of the correction mechanism.

The decision rests on twenty-seven items of information: three node taints and the toleration field of twenty-four pods. None of these items is a measurement, and there is no single counter that tests whether a toleration is genuinely necessary. The system places the wrong pod on the set-aside node not because it fails to know something is wrong, but because no item exists that it could know it from.

Summary

• A taint is written on the node, a toleration on the pod; a toleration permits, it does not attract. With no tainted node, none of the batch job’s six pods get placed, because what carries them is label selection.
• The separation’s first bill is on the general workload: once three nodes are set aside, the general workload squeezes into six nodes and three pods fail to place; once five nodes are set aside, seven pods fail to place.
• The second bill is in the set-aside capacity: reserving twenty-four hours for a four-hour window puts the idle ratio at 83.3 percent in the best case, rising to 90 percent as the set-aside node count grows.
• The three effects measure three separate things: the placement block leaves zero foreign pods on the set-aside node, the eviction block evicts eight pods and three of them fail to re-place, deprioritization does not block the three foreign pods at all.
• An unnecessary toleration lets three foreign pods onto the set-aside nodes and makes two of the nightly batch job’s pods unplaceable; because the placed count rises from fifteen to eighteen, the error looks like an improvement.
• The decision rests on twenty-seven declaration items, and none measures whether a toleration is necessary; a wrong toleration also blocks the correction, because it protects the pod under the eviction effect.

Next Step

Every decision so far has been made at the moment of placement: a pod lands on a node and stays there. Resource pressure, by contrast, is born after placement. Once a node’s memory fills up, something has to be evicted, and the decision-maker here is not the scheduler but the node itself. Which pod gets evicted depends on how far below or above its request it is running, and on a priority class written onto the pod. The next lesson models the eviction order, and counts how many pods a priority addition affects when making room, and how many times the chain grows longer.