Jobs and Scheduled Jobs

A replica set and a stateful set share the same assumption: pods do not finish. The desired replica count is a target, and if a pod terminates this is a drift, which the reconciler re-creates. The measurement network’s nightly batch job does not fit this assumption. That job starts every night, verifies the day’s readings across twelve meter sets, produces the billing entry, and finishes. Finishing is the expected behavior; if it were given to a replica set, the reconciler would restart it endlessly every night.

This is why a second object type is needed: the job. A job’s target is not the number of running pods, it is the number of completed runs. Termination is not drift, it is the target itself. Adding a schedule on top produces a cron job: an object that produces job objects at specific times.

The devolved decision here has three parts: whether the job is finished, whether it is retried if not, and when the next run happens and what happens if the previous one is still going. Four things are counted: how much completion the attempt limit buys, what round count and what waste parallelism gives, what each of the three overlap policies sacrifices, and whether a missed run can be made up.

Work Expected to Finish

The reconciler runs the same loop in both objects — desired state, actual state, gap, correction — but “desired state” is a different thing. In a replica set, desired state is about a moment: three pods should be running right now. In a job, desired state is about a history: twelve slices should have completed. The first case is checked continuously, the second is satisfied once and never breaks again.

• CO43 — The job, attempt counter, parallelism slots, and nightly schedule are a model built in node; no real cluster runs.
• CO44 — The measurement network is fictional. The nightly batch job runs for 30 nights, and each run is split into twelve meter sets (slices).
• CO45 — An attempt’s failure probability is 0.5; the model has a single failure class (input file not ready). The attempt limit is swept at 1, 3, and 6.
• CO46 — Each slice takes one round; parallelism is the number of concurrent slots. If a slice falls to a second attempt, one round is added to the total time.
• CO47 — The cron job’s interval is 24 hours. Run durations come from the generator: roughly a quarter of the time 30–54 hours, otherwise 3–7 hours.
• CO48 — The controller is down for three nights. The startup window is swept at 0, 1, and 3 nights. Seed 20260813.

// measure/jobs.mjs — a job object's completion guarantee and a cron job's overlap policies.
// MODEL: the job object, attempt counter, parallelism slots, and nightly schedule are a model built
// in node; no real cluster runs. The numbers below are the model's numbers.
const NIGHTS = 30, SLICES = 12, INTERVAL = 24, SEED = 20260813, FAIL_PROB = 0.5;
let s = SEED % 2147483647;
const random = () => (s = (s * 48271) % 2147483647) / 2147483647;

// --- 1. completion guarantee: attempt limit ---
console.log(`${"attempt limit".padEnd(14)}${"completed nights".padStart(17)}${"failed nights".padStart(15)}` +
  `${"total attempts".padStart(16)}${"wasted attempts".padStart(17)}${"leftover pods".padStart(15)}`);
for (const limit of [1, 3, 6]) {
  s = SEED % 2147483647;
  let completed = 0, attempts = 0, wasted = 0;
  for (let g = 0; g < NIGHTS; g += 1) {
    let d = 0, ok = false;
    while (d < limit && !ok) { d += 1; attempts += 1; if (random() >= FAIL_PROB) ok = true; else wasted += 1; }
    if (ok) completed += 1;
  }
  console.log(`${String(limit).padEnd(14)}${String(completed).padStart(17)}${String(NIGHTS - completed).padStart(15)}` +
    `${String(attempts).padStart(16)}${String(wasted).padStart(17)}${String(wasted).padStart(15)}`);
}

// --- 2. parallelism and completion count ---
console.log(`\n${"parallelism".padEnd(12)}${"completions".padStart(13)}${"rounds".padStart(8)}${"pod-rounds".padStart(12)}` +
  `${"empty slots".padStart(13)}${"one slice needs 2 tries".padStart(25)}${"increase".padStart(10)}`);
for (const p of [1, 3, 5, 6, 12]) {
  const rounds = Math.ceil(SLICES / p), delayed = rounds + 1;
  console.log(`${String(p).padEnd(12)}${String(SLICES).padStart(13)}${String(rounds).padStart(8)}${String(SLICES).padStart(12)}` +
    `${String(p * rounds - SLICES).padStart(13)}${String(delayed).padStart(25)}${`%${Math.round((delayed / rounds - 1) * 100)}`.padStart(10)}`);
}

// --- 3. overlap: three policies ---
s = SEED % 2147483647;
const DURATION = Array.from({ length: NIGHTS }, () => (random() < 0.25 ? 30 + Math.floor(random() * 25) : 3 + Math.floor(random() * 5)));
console.log(`\n${NIGHTS} nights' run durations (hours): ${DURATION.join(",")}`);

function policy(label) {
  const start = new Array(NIGHTS).fill(null), end = new Array(NIGHTS).fill(null);
  let finish = 0, started = 0, skipped = 0, delay = 0;
  for (let g = 0; g < NIGHTS; g += 1) {
    const t = g * INTERVAL;
    if (label === "skip" && finish > t) { skipped += 1; continue; }
    const b = label === "queue" ? Math.max(t, finish) : t;
    start[g] = b; end[g] = b + DURATION[g]; finish = end[g]; delay += b - t; started += 1;
  }
  let maxConcurrent = 0, maxQueue = 0;
  for (let g = 0; g < NIGHTS; g += 1) {
    const t = g * INTERVAL;
    maxConcurrent = Math.max(maxConcurrent, start.filter((b, i) => b !== null && b <= t && end[i] > t).length);
    maxQueue = Math.max(maxQueue, start.filter((b, i) => b !== null && b > t && i * INTERVAL <= t).length);
  }
  const last = NIGHTS * INTERVAL;
  return { started, skipped, delay, maxConcurrent, maxQueue,
    carryover: start.filter((b) => b !== null && b >= last).length,
    podHours: start.reduce((a, b, i) => a + (b === null ? 0 : end[i] - b), 0) };
}
const P = {};
console.log(`${"policy".padEnd(14)}${"started".padStart(9)}${"skipped".padStart(9)}${"max concurrent".padStart(17)}` +
  `${"max queue".padStart(12)}${"avg delay (hours)".padStart(20)}${"pod-hours".padStart(12)}`);
for (const label of ["skip", "queue", "run parallel"]) {
  const r = policy(label); P[label] = r;
  console.log(`${label.padEnd(14)}${String(r.started).padStart(9)}${String(r.skipped).padStart(9)}${String(r.maxConcurrent).padStart(17)}` +
    `${String(r.maxQueue).padStart(12)}${(r.delay / r.started).toFixed(2).padStart(20)}${String(r.podHours).padStart(12)}`);
}
console.log(`unprocessed nights: skip ${NIGHTS - P["skip"].started}, queue ${NIGHTS - P["queue"].started}, ` +
  `run parallel ${NIGHTS - P["run parallel"].started}; runs carrying past the horizon ${P["queue"].carryover}`);

// --- 4. missed run: the controller is down from night 10 to 12 ---
const MISSED = 3;
console.log(`\n${"startup window".padEnd(16)}${"made up".padStart(11)}${"dropped".padStart(11)}` +
  `${"ran under skip".padStart(17)}${"ran under queue".padStart(18)}${"ran under parallel".padStart(21)}`);
for (const window of [0, 1, 3]) {
  const madeUp = Math.min(window, MISSED);
  console.log(`${String(window).padEnd(16)}${String(madeUp).padStart(11)}${String(MISSED - madeUp).padStart(11)}` +
    `${String(Math.min(madeUp, 1)).padStart(17)}${String(madeUp).padStart(18)}${String(madeUp).padStart(21)}`);
}
console.log(`information items: 4 per night (scheduled time, run duration, attempt counter, active-run count) = ${NIGHTS * 4} (seed ${SEED})`);

attempt limit  completed nights  failed nights  total attempts  wasted attempts  leftover pods
1                            19             11              30               11             11
3                            27              3              49               22             22
6                            29              1              54               25             25

parallelism   completions  rounds  pod-rounds  empty slots  one slice needs 2 tries  increase
1                      12      12          12            0                       13        %8
3                      12       4          12            0                        5       %25
5                      12       3          12            3                        4       %33
6                      12       2          12            0                        3       %50
12                     12       1          12            0                        2      %100

30 nights' run durations (hours): 5,4,7,5,6,5,5,4,7,31,3,3,5,3,50,4,51,6,5,3,5,7,47,5,3,37,6,5,3,5
policy          started  skipped   max concurrent   max queue   avg delay (hours)   pod-hours
skip                 25        5                1           0                0.00         266
queue                30        0                1           2                4.23         335
run parallel         30        0                2           0                0.00         335
unprocessed nights: skip 5, queue 0, run parallel 0; runs carrying past the horizon 0

startup window      made up    dropped   ran under skip   ran under queue   ran under parallel
0                         0          3                0                 0                    0
1                         1          2                1                 1                    1
3                         3          0                1                 3                    3
information items: 4 per night (scheduled time, run duration, attempt counter, active-run count) = 120 (seed 20260813)

The Completion Guarantee: Attempt Limit

With a single attempt the job completes 19 of 30 nights, and 11 nights fail. When the attempt limit rises to three, completed nights rise to 27; at six, to 29. The guarantee can be bought, but not linearly: the first increase gains 8 nights, the second only 2.

The cost is in the other columns. Total attempts rise from 30 to 49, then to 54; wasted attempts from 11 to 22, then to 25. So raising the limit from 3 to 6 gains 2 nights while adding 3 wasted attempts — one and a half wasted runs for every night gained. These runs consume resources and eat into the cron job’s window too.

The last column is a silent cost. A failed attempt’s pod is not deleted; it is kept so its log can be read. At a limit of six, 25 leftover pods remain, and nobody collects them. The job needs to declare a retention period, or by the end of thirty nights the cluster fills up with failed pods that have long lost their meaning.

What “failed night” means also has to be stated separately. At a limit of 1, eleven nights’ data was never verified; the job declares this in a status field, but declaring is not processing. The devolved decision is only the “retry” decision; the owner of the “attempts exhausted, data stayed incomplete” decision is still outside.

Parallelism and Completion Count

The second table shows what the two settings give together. The completion count is fixed at 12 — all twelve slices have to be processed. As parallelism rises from 1 to 12, round count drops from 12 to 1, and the total pod-rounds never changes: 12 in every case. Parallelism does not reduce total work, it only spreads it out over time.

Two things change. The first is empty slots: at a parallelism of 5, in the last of three rounds only 12 of 15 slots are filled, and 3 slots sit empty. Parallelism values that do not divide the completion count produce waste. The second, and more important, is the last two columns: if a slice falls to a second attempt, total time stretches by one round, and this one round’s share grows together with parallelism. At a parallelism of 1 the increase is 8%, at 12 it is 100%. At full parallelism, the work equals the duration of the slowest slice; a single slow slice doubles the total time.

When completion count is not declared, the object switches to a different mode: pods drain a work queue and terminate themselves once it empties. The round count stays the same, but the finish decision sits inside the pod. The cluster only sees the pod terminate; it does not know how many slices were processed. This is where the devolved decision is taken back here: if the completion count is declared, the cluster gives the guarantee; if not, the application does.

Overlap: Three Policies

The third table is the nightly job’s real question. Among the thirty nights’ run durations, five nights run 31, 37, 47, 50, and 51 hours — longer than the 24-hour interval. A new run arrives before the previous one has finished.

The skip policy does not start a conflicting run: 25 runs started, 5 nights were skipped. Concurrency stays at 1, there is no queue, no delay, and resource consumption is lowest at 266 pod-hours. The cost is direct data coverage: five nights’ readings were never verified, and this policy does not bring them back.

The queue policy skips no night at all: 30 runs, 0 unprocessed nights. In exchange runs start an average of 4.23 hours late from their scheduled time, and at one point two runs wait in the queue at once. Concurrency is still 1 — the single-runner guarantee is preserved. The critical point is that the delay does not accumulate: because long nights are spread out, the queue empties, and no run carries past the thirtieth night. If long runs came back to back, this column would grow without bound.

The run parallel policy also starts all 30 runs, with zero delay, but concurrency rises to 2. Two runs read the same meter sets and write the same billing entry at the same time. From the cluster’s point of view there is no problem — both pods are healthy; the problem is in the application’s data layer, and the cluster cannot see it. On the resource side, the same 335 pod-hours are spent as under queue.

What all three policies have in common is this: none of them knows the run duration. The policy is chosen before the run starts; the duration is learned after it. Because the system does not know, one sacrifices data coverage, one sacrifices timing, and one sacrifices the singleton guarantee.

The correct choice for the measurement network can be read from the table. The billing entry cannot tolerate a missing night, so skip’s 5 nights are unacceptable. Two runs writing the same night at once is also unacceptable, because the billing entry is written additively, and two runs would add the same readings twice. What is left is queue, and its cost is visible in the table: an average delay of 4.23 hours, a peak queue of two runs. This cost can only be paid as long as the queue empties; if long nights arrive back to back, the delay accumulates, and the nightly job spills over into the day at some point. So the choice of policy is not a setting made once and left alone, it is a decision that has to be revisited as the run-duration distribution changes.

The Missed Run

The last table measures the case where the controller is down for three nights. With a startup window of 0 nights, none of the three missed runs are made up; with a window of 1 night, one runs and two are dropped; with a window of 3 nights, all three go into the queue.

Being made up does not mean running. At window 3, three runs become eligible at the same moment, and the overlap policy kicks in right there: under the skip policy only one runs and two drop again; under queue, all three run one after another; under run parallel, all three start at once. So the two settings can cancel each other out — if a wide window is paired with a strict overlap policy, catch-up is open on paper and closed in practice.

This lesson’s devolution balance sheet is this: the decision rests on four items per night — scheduled time, run duration, attempt counter, active-run count; 120 items for thirty nights. Three of these come from measurement; one (run duration) is only known after the run finishes. This is the one item the system does not know, and the entire difference between the three policies comes from it.

Summary

• A job’s target is not the number of running pods but the number of completed runs; termination is not drift, it is the target itself.
• As the attempt limit rises 1 → 3 → 6, completed nights go 19 → 27 → 29 and wasted attempts 11 → 22 → 25. The last increase gains 2 nights and adds 3 wasted runs; 25 leftover pods remain.
• Parallelism does not change total work (12 pod-rounds in every case), it brings rounds down from 12 to 1; at a parallelism of 5, 3 empty slots appear, and the cost of a single slow slice rises from 8% to 100%.
• If the completion count is not declared, the guarantee moves from the cluster to the application: the cluster only sees termination, it does not know how many slices were processed.
• The three overlap policies sacrifice three separate things: skip leaves 5 nights unprocessed (266 pod-hours), queue delays by an average of 4.23 hours, run parallel raises concurrency to 2 and lets two writers loose on the same data (335 pod-hours).
• With a startup window of 0 → 1 → 3 nights, catch-up is 0 → 1 → 3; but how many of the made-up runs actually run is decided by the overlap policy (1 under skip).

Next Step

The nightly job became a cluster object: its finish is defined, its attempts are limited, its schedule is declared. Every number measured was on a single application’s own account — 335 pod-hours, two concurrent runs, twelve parallel slices. Yet the cluster does not belong to the measurement network alone. Two other applications run on the same nodes, and the question of what happens to their requests when the nightly job opens twelve slices at once was never asked. Pod, replica set, version object, service object, volume, job — all of them sit in a single naming space and all of them eat from the same resource pool. The next lesson gathers three tenants into a single cluster and counts: how many requests one tenant starves another out of without a quota, how many requests are rejected once a quota is set, how many units of the cluster the sum of the quotas leaves idle, and how default request values shift these three numbers.