On-Demand Environment Provisioning

The previous topic defined what a platform is and drew its boundaries: which decision is taken over, which stays with the team, who the internal customer is. The definition holds, but it still has not said what a team concretely asks the platform for when it starts work in the morning. The first thing asked for is almost always the same: an environment to work in.

Environment on demand is an environment that a team starts and finishes itself, instead of writing its environment setup into a request form and waiting. How the environment is provisioned — the path from declaration to reality, the drift between record and reality — was measured in M22/K01’s environment provisioning lesson and in M22/K05; it is not repeated here. What this lesson measures is not the provisioning mechanics but the environment’s life cycle: how long it waited, how long it lived, and how much of that lifespan went unused.

The measurement runs on the regional measurement network: fictional software that collects water-meter readings, validates them, and opens field work orders. No real infrastructure is provisioned; what is measured is a model written in node.

PC1. Five teams produce the network’s software: meter reading, validation, billing, field, reporting. The teams and request rates are fictional. PC2. The observation window is 90 days. The request flow is drawn from a self-written generator; the seed is visible. PC3. Manually setting up an environment is 6 hours of human labor, and the platform team’s manual setup capacity is one environment per day. PC4. Generation on the on-demand path takes 0.5 hours; the pool of concurrently open environments is 16. PC5. An environment delivered on the queue path is never closed — waiting again is expensive. On the on-demand path the team closes it once its use is done; the cases where it forgets to close are written in advance, with no randomness. PC6. The platform offers two environment shapes: small and medium. A field-equipped shape is not offered. An abandoned environment is one that stays open for at least seven days after its last day of use.

Request Queue and On Demand

// data.mjs — ninety-day environment demand for five teams; model, generator self-written, seed visible.
export const TEAM = ["meter-reading", "validation", "billing", "field", "reporting"];
export const DAYS = 90, SEED = 20260801;
export const generator = (t) => () => ((t = (t * 1103515245 + 12345) % 2147483648) / 2147483648);

const RATE = { "meter-reading": 0.24, validation: 0.16, billing: 0.10, field: 0.30, reporting: 0.06 };
const SHAPE = { "meter-reading": "small", validation: "medium", billing: "medium",
  field: "field-equipped", reporting: "medium" };   // the platform offers only small and medium
export const OFFERED = ["small", "medium"];

export const requests = () => {                     // independent draw, day by day, per team
  const r = generator(SEED), t = [];
  for (let g = 0; g < DAYS; g++)
    for (const e of TEAM) {
      const [a, b, c] = [r(), r(), r()];
      if (a < RATE[e]) t.push({ team: e, day: g, shape: SHAPE[e],
        usage: 1 + Math.floor(b * 9),                // how many days it will actually be used
        stateful: c < 0.45 });                        // does generated data persist in the environment
    }
  return t;
};

The same request flow is run through two paths. On the queue path, a human meets every request. On the platform path, requests in an offered shape are generated on demand; those in an unoffered shape stay outside the golden path, and that team builds its own environment.

// path.mjs — the same request flow is run through two paths: the request queue and on-demand.
import { requests, DAYS, TEAM, OFFERED } from "./data.mjs";

const CAPACITY = 1, POOL = 16, PROVISION_HOURS = 0.5, MANUAL_HOURS = 6, ABANDON = 7;
const T = requests();
const avg = (d) => (d.length ? d.reduce((a, b) => a + b, 0) / d.length : 0).toFixed(1);

const queue = (list) => {                            // a human builds it: at most one env per day
  const waiting = [], env = [];
  for (let g = 0; g < DAYS; g++) {
    waiting.push(...list.filter((t) => t.day === g));
    for (let k = 0; k < CAPACITY && waiting.length; k++) {
      const t = waiting.shift();
      env.push({ ...t, opened: g, wait: (g - t.day) * 24 + MANUAL_HOURS, closed: DAYS });
    }                                                 // on the queue path an env is never released
  }
  return { env, undelivered: waiting.length, full: 0 };
};

const onDemand = (list) => {                          // generated by one request, pool-limited
  const waiting = [], env = [], open = [];
  let full = 0;
  for (let g = 0; g < DAYS; g++) {
    for (let i = open.length - 1; i >= 0; i--) if (open[i].closed <= g) open.splice(i, 1);
    waiting.push(...list.filter((t) => t.day === g));
    while (waiting.length && open.length < POOL) {
      const t = waiting.shift();
      const abandoned = (t.day * 7 + t.usage) % 4 === 0;   // written in advance, not random
      const o = { ...t, opened: g, wait: (g - t.day) * 24 + PROVISION_HOURS, abandoned,
        closed: abandoned ? DAYS : Math.min(DAYS, g + t.usage) };
      env.push(o); open.push(o);
    }
    if (open.length >= POOL) full++;
  }
  return { env, undelivered: waiting.length, full };
};

const selfBuilt = (list) => ({ full: 0, undelivered: 0,   // outside the golden path, the team builds and never closes it
  env: list.map((t) => ({ ...t, opened: t.day, wait: MANUAL_HOURS, closed: DAYS })) });

const measure = (label, { env, undelivered, full }) => {
  const envDays = env.reduce((a, o) => a + (o.closed - o.opened), 0);
  const used = env.reduce((a, o) => a + Math.min(o.usage, o.closed - o.opened), 0);
  const abandonedCount = env.filter((o) => o.closed - o.opened - o.usage >= ABANDON).length;
  console.log(`${label.padEnd(22)} delivered ${String(env.length).padStart(2)}` +
    `  undelivered ${undelivered}  wait ${avg(env.map((o) => o.wait)).padStart(5)} h` +
    `  env-days ${String(envDays).padStart(4)}  idle ${String(envDays - used).padStart(4)}` +
    ` (${String(Math.round((envDays - used) / envDays * 100)).padStart(2)}%)  abandoned ${abandonedCount}`);
  return { envDays, used, abandonedCount, full, n: env.length };
};

const inScope = T.filter((t) => OFFERED.includes(t.shape));
const outOfScope = T.filter((t) => !OFFERED.includes(t.shape));
console.log(`requests ${T.length}  window ${DAYS} days  teams ${TEAM.length}  manual setup ` +
  `${MANUAL_HOURS} h, ${CAPACITY}/day  pool ${POOL} concurrent  provision ${PROVISION_HOURS} h`);
const A = measure("request queue (all)", queue(T));
const B = measure("on demand (in scope)", onDemand(inScope));
const C = measure("outside golden path", selfBuilt(outOfScope));
console.log(`days the pool was full ${B.full}/${DAYS}; while the pool is full incoming requests wait,` +
  ` ${B.full ? "hidden capacity leaks back" : "no leak"}`);
console.log(`outside golden path ${outOfScope.length}/${T.length} requests, sole reason field-equipped shape` +
  ` (${[...new Set(outOfScope.map((t) => t.team))].join(", ")}); none of them benefit from the platform service`);
console.log(`human labor: queue ${A.n * MANUAL_HOURS} h   platform ${C.n * MANUAL_HOURS} h` +
  ` (outside path only)   saved ${(A.n - C.n) * MANUAL_HOURS} h`);

requests 90  window 90 days  teams 5  manual setup 6 h, 1/day  pool 16 concurrent  provision 0.5 h
request queue (all)    delivered 83  undelivered 7  wait  83.2 h  env-days 3528  idle 3133 (89%)  abandoned 73
on demand (in scope)   delivered 60  undelivered 3  wait  10.1 h  env-days  877  idle  580 (66%)  abandoned 12
outside golden path    delivered 27  undelivered 0  wait   6.0 h  env-days 1212  idle 1090 (90%)  abandoned 22
days the pool was full 16/90; while the pool is full incoming requests wait, hidden capacity leaks back
outside golden path 27/90 requests, sole reason field-equipped shape (field); none of them benefit from the platform service
human labor: queue 498 h   platform 162 h (outside path only)   saved 336 h

The wait gap is the most visible number: an average of 83.2 hours on the queue path, 10.1 hours on the on-demand path. On the queue path, 7 of the 90 requests are still waiting when the window closes. But automation alone is not what lowers the wait; the queue path’s capacity is one environment per day and demand is roughly one request per day, so the queue already runs at its saturation limit. The decision handed over is this: the team, not a human, decides when the environment comes into existence and when it goes away.

The second number is more interesting than the first. On the queue path, 3,133 of 3,528 environment-days are idle — environments go unused for 89 percent of their lifespan. The reason lies in the waiting itself: because getting an environment takes days, nobody releases the one they got, and 73 of 83 environments turn into abandoned ones. Wait time and idle resources are not two independent problems; one produces the other.

Lifespan Limit: The Tradeoff of a Short Life

// lifespan.mjs — the lifespan limit is swept: cost, rebuilds, data loss; then the threshold is computed.
import { requests, DAYS, TEAM, OFFERED } from "./data.mjs";

const POOL = 16, MANUAL_HOURS = 6, ABANDON = 7, SETUP = 320, OPEX_MONTH = 8, MONTHS = 3;
const T = requests(), inScope = T.filter((t) => OFFERED.includes(t.shape));

const sweep = (lifespan) => {              // when the lifespan runs out the env is deleted; unfinished work is requested again
  const waiting = [], open = [];
  let totalDays = 0, provisioned = 0, rebuilds = 0, loss = 0, abandonedCount = 0;
  for (let g = 0; g < DAYS; g++) {
    for (let i = open.length - 1; i >= 0; i--) {
      const o = open[i], age = g - o.opened, finished = o.remaining === 0 && !o.abandoned;
      if (!finished && age < lifespan) continue;
      open.splice(i, 1);
      totalDays += age;
      if (age - (o.usage - o.remaining) >= ABANDON) abandonedCount++;
      if (o.remaining > 0) { rebuilds++; loss += o.stateful ? 1 : 0; waiting.push(o); }
    }
    waiting.push(...inScope.filter((t) => t.day === g).map((t) => ({ ...t, remaining: t.usage,
      abandoned: (t.day * 7 + t.usage) % 4 === 0 })));
    while (waiting.length && open.length < POOL) {
      open.push({ ...waiting.shift(), opened: g }); provisioned++;
    }
    for (const o of open) if (o.remaining > 0) o.remaining--;
  }
  for (const o of open) {                            // still open at the end of the window
    totalDays += DAYS - o.opened;
    if (DAYS - o.opened - (o.usage - o.remaining) >= ABANDON) abandonedCount++;
  }
  return { totalDays, provisioned, rebuilds, loss, abandonedCount, undelivered: waiting.length };
};

console.log("lifespan  env-days  provisioned  rebuilds  data loss  abandoned  undelivered");
for (const s of [1, 2, 3, 5, 7, 14, 90]) {
  const r = sweep(s);
  console.log(`${(s === 90 ? "none" : s + (s === 1 ? " day" : " days")).padEnd(10)}${String(r.totalDays).padStart(6)}` +
    `${String(r.provisioned).padStart(12)}${String(r.rebuilds).padStart(11)}` +
    `${String(r.loss).padStart(11)}${String(r.abandonedCount).padStart(11)}${String(r.undelivered).padStart(13)}`);
}

const benefit = Object.fromEntries(TEAM.map((e) =>              // human-hours saved per team
  [e, inScope.filter((t) => t.team === e).length * MANUAL_HOURS]));
const cost = SETUP + OPEX_MONTH * MONTHS;
let k = 0, threshold = 0;
const cumulative = TEAM.map((e) => { k += benefit[e]; if (k >= cost && !threshold) threshold = TEAM.indexOf(e) + 1;
  return `${e} ${benefit[e]} (${k})`; });
console.log(`\nplatform cost ${cost} person-hours (setup ${SETUP} + ${MONTHS} months opex)` +
  `   five teams' total gain ${k} person-hours`);
console.log(`cumulative gain per team: ${cumulative.join("  ")}`);
console.log(`threshold ${threshold ? "crossed at team " + threshold : "not crossed in this period"}` +
  `; teams contributing ${TEAM.filter((e) => benefit[e] > 0).length}/${TEAM.length}`);

lifespan  env-days  provisioned  rebuilds  data loss  abandoned  undelivered
1 day        314         314        251        113          0            0
2 days       322         175        112         50          0            0
3 days       331         127         64         27          0            0
5 days       343          90         27         12          0            0
7 days       355          76         13          8          0            0
14 days      425          63          0          0         10            0
none         877          60          0          0         12            3

platform cost 344 person-hours (setup 320 + 3 months opex)   five teams' total gain 378 person-hours
cumulative gain per team: meter-reading 174 (174)  validation 78 (252)  billing 90 (342)  field 0 (342)  reporting 36 (378)
threshold crossed at team 5; teams contributing 4/5

The sweep shows the two curves moving at different speeds. With no lifespan limit, 877 environment-days are spent; pulling the limit to fourteen days drops that to 425 — under half — and in exchange produces zero rebuilds. Past this point it stops being equally cheap: dropping from fourteen days to seven days only pulls the cost from 425 to 355, but produces 13 rebuilds and 8 data-loss incidents. Dropping to one day lowers the cost to 314 — a twelve-percent gain over seven days — and in exchange produces 251 rebuilds and 113 data-loss incidents. As the cost curve flattens, the rebuild curve steepens.

The limit also has an unseen benefit. With no lifespan limit, 3 requests are never met at all, because abandoned environments fill the pool; the moment the limit is set, undelivered requests drop to zero. The cost of an unused environment is paid not only on the ledger, but in other teams’ waiting.

The Cost of the Hidden Layer

The hidden layer is the environment’s setup steps, and the gain is 336 person-hours: 498 hours of work on the queue path drops to 162 hours on the platform path. The threshold comes from the same computation. The platform’s cost over one quarter is 344 person-hours, and the five teams’ combined gain is 378 person-hours; the threshold is crossed only at the fifth team, and the team that crosses it is reporting, the team with the lowest annual demand. With three teams, the same platform does not pay for itself. The number does not point in only one direction: the same model also says the platform should not be built for a network where few teams operate.

The abstraction leaks in two places. The first is the pool: on 16 of 90 days the pool is full, and on those days waiting comes back the way it did on the queue path — the team assumes it will get an environment the moment it asks, but in reality it hits the lower layer’s capacity. The second is the lifespan limit: the fact that the environment is not permanent becomes visible to the team only at a data-loss incident, and that knowledge is learned not by reading documentation but by losing work.

Those who stay outside do so for a single reason. All 27 requests asking for a field-equipped environment belong to the field team, and this team benefits from none of the platform service: it builds its own environment, the lifespan limit does not apply, 90 percent of its environment-days sit idle, and 22 of its environments are abandoned. There is no partial benefit; if a request’s shape does not fit one of the two offered shapes, that request spends its entire life cycle outside the platform.

Summary

• The same 90 requests waited an average of 83.2 hours on the queue path and 10.1 hours on the on-demand path; on the queue path, 7 requests were never met before the window closed.
• Waiting produced idle resources: on the queue path, 89% of 3,528 environment-days went unused, and 73 of 83 environments turned into abandoned ones.
• Pulling the lifespan limit to 14 days dropped the cost from 877 environment-days to 425 and produced zero rebuilds; pulling it to 1 day dropped the cost only to 314 but produced 251 rebuilds and 113 data-loss incidents.
• The lifespan limit also emptied the pool: the 3 requests that went unmet on the unlimited path dropped to zero once the limit was set.
• The platform’s quarterly cost was 344 person-hours and the five teams’ gain was 378 person-hours; the threshold was crossed only at the fifth team, and the field team never contributed to that gain.
• The abstraction leaked in two places: the pool filled on 16 of 90 days and waiting came back; the lifespan limit is existence became visible to the team only through data loss.

Next Step

The environment now exists the moment it is requested, but what goes inside it is still written by hand. When a sixth service needs to be added to the measurement network, the team starts from an empty directory and makes every decision over again, from log format to health endpoint, from the build step to metric publishing — and the five teams make these decisions without knowing what the others chose. The next lesson takes up the flow for starting a new service: real files are generated, the generated output is compared against what was written by hand, and once the template is updated after generation, the number of services left behind is counted.