Scheduled Triggering

The whole flow in the previous lesson started with an event: a reading arrived, validation was triggered, the chain ran. Some of the measurement network’s jobs have no such event. The nightly batch invoicing, the daily reconciliation, and the weekly meter health scan depend on no reading’s arrival; the only thing that starts them is the clock.

Scheduled work is not new by itself; missed-run and overlap policies were measured earlier and that comparison is not repeated here. This lesson’s question is narrow: what changes when the clock is held by a service?

The Handed-Over Decision and the Kind of Guarantee

In a self-run scheduler, the clock is a specific process running on a specific machine. If that machine dies, no trigger fires; if two machines run at once, the trigger fires twice. Both are visible, both are fixable — because whatever holds the clock is in your own hands.

In a scheduler taken as a service, this responsibility is handed over. The constraint that comes back in exchange fits one sentence, and it is this lesson’s axis: the guarantee is at least once. The service undertakes that the trigger will happen, not that it will happen exactly once. Two triggers can fire in the same minute; this is not a malfunction, it is declared behavior. This is why the job being idempotent is not a good practice here — it is the service’s precondition.

What is handed over is countable: keeping the process that holds the clock alive, keeping that process single-instance, not losing the schedule when the machine changes, keeping a record of triggers. What remains is again a small surface — schedule expression, time zone, target, the input passed to the target, and the retry policy applied if triggering fails: five items. This is the same size of surface as the queue configuration in the earlier lessons, the same kind of constraint.

The time zone item is a case on its own. When a schedule is written against local time, two nights a year are not 24 hours long — one is 23, the other 25. A once-a-day job can run twice on one of those nights and not at all on the other. So this measurement’s two flaws — double runs and missed runs — can also come from the time zone, with nothing to do with the trigger’s own accuracy.

The second constraint runs the other way. If the target is not standing at trigger time — the function is throttled, at its account limit, mid-deployment — that run is missed and not compensated for. The scheduler only knows “now”; it does not reproduce a past moment. The two constraints together form an odd pair: the same job sometimes runs twice, sometimes not at all.

The measurement runs on the fictional regional measurement network’s nightly batch invoicing job.

• ED19. A thirty-night run; 1,200 subscribers are invoiced every night, expected total 36,000 invoices.
• ED20. In one of eleven nights the trigger arrives twice in the same minute, in one of thirteen it does not arrive at all; the days are fixed, there is no randomness in the model.
• ED21. The idempotency key is derived from period plus subscriber ID; if the key already exists, the second write is skipped.
• ED22. The trigger moment drifts between −60 and +60 units from the intended moment; the drift is fixed per day.
• ED23. A day is 1,440 units and readings are spread evenly across the day.
• ED24. The model does not build the internal locking of overlapping runs; only the invoices written and the readings processed are counted.

Two Triggers in the Same Minute

// triggering.mjs — scheduled triggering model: at-least-once guarantee and a missed run
// Model: not a real scheduler, but the counted effect of the guarantee type on invoice count.
const DAYS = 30, SUBSCRIBERS = 1200;
const DOUBLE = (g) => g % 11 === 4;    // on these nights the trigger arrived twice in the same minute
const MISSED = (g) => g % 13 === 9;    // on these nights the trigger never arrived

function run(hasKey) {
  const written = new Set();
  let invoices = 0, duplicate = 0, missing = 0, triggers = 0;
  for (let g = 0; g < DAYS; g++) {
    if (MISSED(g)) { missing += SUBSCRIBERS; continue; }        // no compensation: the day never comes back
    for (let t = 0; t < (DOUBLE(g) ? 2 : 1); t++) {
      triggers++;
      for (let a = 0; a < SUBSCRIBERS; a++) {
        const key = `p${g}:s${a}`;                              // idempotency key: period + subscriber
        if (hasKey && written.has(key)) continue;
        written.add(key); invoices++;
        if (t > 0) duplicate++;
      }
    }
  }
  return { triggers, invoices, duplicate, missing };
}

console.log("idempotency key".padEnd(21) + "triggers".padStart(11) +
  "invoices written".padStart(19) + "duplicate".padStart(11) + "missing".padStart(9));
for (const [name, v] of [["no", false], ["yes", true]]) {
  const s = run(v);
  console.log(name.padEnd(21) + String(s.triggers).padStart(11) + String(s.invoices).padStart(19) +
    String(s.duplicate).padStart(11) + String(s.missing).padStart(9));
}
console.log(`days: ${DAYS}, subscribers: ${SUBSCRIBERS}, expected invoices: ${DAYS * SUBSCRIBERS}`);

node triggering.mjs

idempotency key         triggers   invoices written  duplicate  missing
no                            31              37200       3600     2400
yes                           31              33600          0     2400
days: 30, subscribers: 1200, expected invoices: 36000

The table processes the same thirty nights twice; the only difference is whether the idempotency key is checked while writing. 31 triggers arrived across the thirty nights. The number is not thirty, because two nights had no trigger at all and three nights had two each: 28 + 3. On its own this number looks harmless; its effect on invoicing is in the first row.

Without an idempotency key, 3,600 duplicate invoices were written. On each of three nights, 1,200 subscribers were billed twice. In the measurement network this means 3,600 subscribers got a second invoice for the same period, a second debt record was created, and — following the chain from earlier lessons — probably a second notification was sent as well. There is no error anywhere; both runs finished successfully.

Alongside the duplicate invoices there is a second number: 3,600 extra function calls. Work per subscriber doubled over those three nights, twice the space was held from the concurrency pool, and the invoicing window stretched. And because the two runs started in the same minute, they overlap: two runs write to the same subscriber’s record at once. If the write does not rest on a uniqueness constraint, both rows land in the table and both count as “successful.” The cost of a double trigger is therefore not only the duplicate record but also a race at the record level.

The second row shows what the key does. Trigger count is still 31 — the key does not deduplicate the trigger, the service keeps sending the second trigger regardless. What changes is the write: if the key derived from period and subscriber ID has already been written, the second write is skipped and duplicate invoices drop to zero. This is the same distinction built in the delivery-semantics measurement — what gets deduplicated is not the delivery, it is the effect.

What the key is derived from is decisive here. In the measurement, the key was built from period plus subscriber ID; the second trigger carried the same period, so it produced the same key and the write was skipped. If the key had been derived from the run’s own ID — since every trigger arrives with its own ID — the two runs would have produced two different keys and the table would have fallen back to its first row. The key defines by which measure a repeat counts as “the same job”; in scheduled work this measure is always the job’s scope, not the run itself. In the measurement network the scope is “which period, which subscriber”; the same period cannot be billed twice, because the second invoice is not a new fact, it is a copy of the first.

The Column the Key Doesn’t Close

The table’s last column is the same in both rows: 2,400 missing invoices. The idempotency key does not touch this number, because the key only answers “was this job done before”; there is no such question to ask about a job that was never done at all.

The missing column’s real difficulty is how long it takes to notice. A duplicate invoice shows itself: a second notification, a doubled charge, a call to the support line. A missing invoice produces no symptom — one never generated bothers no one. 2,400 invoices are 6.7 percent of the thirty-day period, and the gap is only visible at end-of-period reconciliation against revenue expectations. The cost of the two missed nights here is therefore not just 2,400 invoices but also that they are cut a month late.

A missed run is the other face of the service’s guarantee, and fixing it is not the trigger’s responsibility but the job definition’s. The difference sits in one sentence: if the job is written as “invoice today,” a missed day never comes back; if it is written as “invoice unbilled periods,” the next night closes both days at once. The second form is also resilient to double triggering, because the second run finds no unbilled period.

This form has two costs. First, the job must find its own scope: every run executes a “which periods went unprocessed” query, and it runs uselessly more often the more rarely a run is missed. Second, as missed days pile up, a single run’s workload multiplies — two days missed back to back means the third night writes 3,600 invoices, which must finish within the function’s time limit. Scope-based work turns a missed run into a time-limit problem.

A third option is to mark, in a separate place, that a run happened, and build an auditor that looks for missing days. This makes the miss visible, but it is itself a scheduled job and is subject to the same guarantee; there is no third layer to catch the auditor’s own missed run.

The choice among the three is made by looking at the nature of the job. For work whose scope can be read from the data, like invoicing, the second form applies directly. For work with no scope — say, a job that warms a cache or deletes temporary files — a missed night has no counterpart to compensate for; there, a miss is not a defect but an accepted condition. The test for the distinction is one question: when a run is missed, is there a missing record left behind? If yes, scope-based writing is mandatory; if not, the scheduler’s guarantee is accepted as is.

Clock Drift and Where the Window Comes From

Triggering also has a moment. The service says “at a given minute every night,” but the moment that actually happens drifts around that minute: queue wait, scaling delay, clock synchronization difference. The drift is small; the problem is not the size of the drift but where the job’s window comes from. The run below processes the same thirty nights with two window definitions: one counts the window backward from the trigger moment, the other starts it from where the previous run left off. The drift is the same in both.

// drift.mjs — clock drift: is the work window built on the trigger moment, or on the data
const DAYS = 30, DAILY = 1200, DAY_MINUTES = 1440;
const drift = (g) => ((g * 37) % 121) - 60;      // -60..+60 units, fixed per day
const trigger = (g) => (g + 1) * DAY_MINUTES + drift(g);

const READINGS = [];                              // readings are spread evenly across the day
for (let g = 0; g < DAYS; g++)
  for (let i = 0; i < DAILY; i++) READINGS.push(g * DAY_MINUTES + Math.floor((i * DAY_MINUTES) / DAILY));

function run(scoped) {
  const counter = new Map(READINGS.map((z, i) => [i, 0]));
  let last = 0;
  for (let g = 0; g < DAYS; g++) {
    const T = trigger(g);
    const start = scoped ? last : T - DAY_MINUTES;   // scoped: from the last processed moment
    READINGS.forEach((z, i) => { if (z >= start && z < T) counter.set(i, counter.get(i) + 1); });
    last = T;
  }
  let never = 0, double = 0;
  for (const n of counter.values()) { if (n === 0) never++; else if (n > 1) double += n - 1; }
  return { never, double };
}

console.log("window".padEnd(22) + "never processed".padStart(17) + "double processed".padStart(19));
for (const [name, k] of [["from trigger time", false], ["from last processed", true]]) {
  const s = run(k);
  console.log(name.padEnd(22) + String(s.never).padStart(17) + String(s.double).padStart(19));
}
console.log(`readings: ${READINGS.length}, drift range: -60..+60 units`);

node drift.mjs

window                  never processed   double processed
from trigger time                   648                560
from last processed                   0                  0
readings: 36000, drift range: -60..+60 units

When the window is counted backward from the trigger moment, 648 of the 36,000 readings were never processed and 560 were processed twice. No error happened and no trigger was missed; every night the job ran and finished successfully. The only thing that drifted was the window’s two edges. When one night triggers early and the next triggers late, the slice between them falls inside both windows; the reverse leaves it inside neither.

648 readings is 1.8 percent of 36,000, and the ratio is misleading — the loss is not spread at random. The missed slice always sits at the day boundary, so it always lands on meters that report at the same fixed times. In a fleet reporting on a fixed schedule, this means the same subset of subscribers goes unbilled every month — not a 1.8 percent loss overall, but close to a 100 percent loss for a handful of subscribers. The 560 double-processed readings sit in the same subset, and the two errors hide each other: an audit of total consumption sees the 648 missing and 560 extra partly cancel out and reads the deviation as small.

The second row produces zero with the same drift. The difference is one line: the window starts not at T - 1440 but from the last processed moment. The window is now derived not from the clock but from the job’s own progress; drift can still move where the window ends, but its start is wherever the previous run left off, so neither a gap nor an overlap remains.

From here the lesson’s rule follows: scheduled triggering should say when the job starts, not what it processes. What it processes should be read from the data. Once this distinction is set, three constraints close at once — a double trigger finds no new work, a missed run gets closed by the next run, and clock drift does not break the window. The cost is that every run has to write its own progress marker permanently, and that write has to commit at the same boundary as the work; if the marker is written before the work, a missed slice is silently skipped, and if written after, the same slice is processed twice.

Summary

• Once scheduled triggering is taken as a service, the guarantee is at least once: two triggers can fire in the same minute, and this is declared behavior, not a malfunction.
• Across thirty nights, 31 triggers arrived; without an idempotency key, 3,600 duplicate invoices were written and both runs finished without error.
• With the key added, trigger count stayed at 31 but duplicate invoices dropped to zero: what gets deduplicated is not the trigger, it is the write.
• A missed run is not compensated for; 2,400 missing invoices stayed the same in both rows. What closes this is not the key but writing the job as “invoice unbilled periods” instead of “invoice today.”
• With the window derived from trigger time, 648 of 36,000 readings were never processed and 560 were processed twice; deriving it from the last processed moment turned both numbers to zero under the same drift. The loss is not random either — the missed slice always sits at the day boundary, landing on meters that report at the same time, and an audit of the total sees the 648 missing and 560 extra partly cancel out.
• Scheduled triggering says when a job starts; reading what it processes from the data closes double triggering, missed runs, and clock drift all at once.

Next Step

Up to this point, everything that woke a function came from inside: a rule, a queue, a flow step, a clock. The measurement network also has an outward-facing side — the field crew’s app, the branch screens, and the interface subscribers use to query their own consumption. These requests cannot reach the function directly; a layer that authenticates identity, maps the route, and throttles stands in between. The next lesson measures that layer as a service: the number of configuration items, the cost per request, the effect of doing authorization at the gateway on function call count, and what the gateway does and does not do about cold start.