Instrumentation Standards

In the previous lesson, the span records of five services merged into a single tree. The reason for the merge was written into the code: every service used the same field names, the same unit, and the same trace id. That assumption has no counterpart in a real network. Five teams write five services; each team names its own record by its own habit, writes duration in its own unit, marks error the way it knows. The records are correct one by one, and each team sees the correct numbers on its own dashboard. What breaks is the cross-service question.

This lesson has the five services of the regional measurement network (fiction) write in their own local schemas, asks the same ten questions over these schemas first, then defines a shared telemetry convention and asks them again. What a convention is is explained by its shared structure; the schema is defined within the lesson.

Five Records of the Same Request

// convention/record.mjs — MODEL of five services writing the same request in their own local schema.
// The real values are generated first (ground truth), then each service writes a record in its own
// field names and units. MD7: the local schemas are fiction. MD8: read and billing write the same
// "duration" name in different units. MD9: billing carries its own request number, not the id from
// the chain. MD10: workorder is a component brought in from outside; its record cannot be changed.
export const SEED = 20260803, REQUESTS = 6000;
export const SERVICES = ["gateway", "read", "validate", "billing", "workorder"];
export const BASE = { gateway: 6, read: 21, validate: 28, billing: 47, workorder: 19 };
export const FLOW = "billing_query";

export function real() {
  let s = SEED;
  const r = () => ((s = (s * 1664525 + 1013904223) >>> 0) / 2 ** 32);
  const output = [];
  for (let i = 0; i < REQUESTS; i += 1) {
    const t0 = Date.UTC(2026, 7, 3) + Math.floor(r() * 3600000), skipped = r() < 0.11, spans = [];
    for (const name of SERVICES) {
      if (name === "validate" && skipped) continue;
      spans.push({ service: name, duration_ms: Math.max(1, Math.round(BASE[name] * (0.6 + r() * 0.9))),
        status: r() < ({ billing: 0.014, workorder: 0.0005 }[name] ?? 0.004) ? "error" : "ok" });
    }
    output.push({ trace: `trace-${(i + 1).toString(36).padStart(4, "0")}`, t0, skipped, spans });
  }
  return output;
}

// Local schemas: the same information written with five different names and three different units.
export const LOCAL = {
  gateway: (z, a) => ({ time: new Date(z.t0).toISOString(), service: "gateway", operation: FLOW,
    duration_ms: a.duration_ms, status: a.status, trace: z.trace }),
  read: (z, a) => ({ t: Math.floor(z.t0 / 1000), unit: "read", action: "query",
    duration: a.duration_ms * 1000, result: a.status === "error" ? 1 : 0, tracing: z.trace }),
  validate: (z, a) => ({ ts: z.t0, component: "validate", operation: "check",
    latency: a.duration_ms / 1000, error: a.status === "error", trace_id: z.trace }),
  billing: (z, a) => ({ time: new Date(z.t0).toISOString(), app: "billing", endpoint: "compute",
    duration: a.duration_ms, status_code: a.status === "error" ? 500 : 200, request_no: `f-${z.trace.slice(6)}` }),
  workorder: (z, a) => ({ time: z.t0, service: "workorder", procedure: "open_order",
    operation_duration: a.duration_ms, error_code: a.status === "error" ? 7 : 0 }),
};
export const localFlow = (G) => G.flatMap((z) => z.spans.map((a) => LOCAL[a.service](z, a)));

if (import.meta.filename === process.argv[1]) {
  const G = real(), z = G[0];
  console.log(`seed ${SEED}, ${REQUESTS} requests, ${SERVICES.length} services, ` +
    `${G.reduce((x, y) => x + y.spans.length, 0)} records`);
  console.log(`\nrecords written for one request (${z.trace}) across five services:`);
  for (const a of z.spans) console.log(`  ${JSON.stringify(LOCAL[a.service](z, a))}`);
  const fields = new Map();
  for (const a of z.spans) for (const k of Object.keys(LOCAL[a.service](z, a)))
    fields.set(k, (fields.get(k) || 0) + 1);
  console.log(`\n${fields.size} distinct field names in total, common to all five services: ` +
    `${[...fields].filter(([, n]) => n === 5).map(([k]) => k).join(", ") || "none"}`);
}

seed 20260803, 6000 requests, 5 services, 29362 records

records written for one request (trace-0001) across five services:
  {"time":"2026-08-03T00:21:34.006Z","service":"gateway","operation":"billing_query","duration_ms":5,"status":"ok","trace":"trace-0001"}
  {"t":1785716494,"unit":"read","action":"query","duration":30000,"result":0,"tracing":"trace-0001"}
  {"ts":1785716494006,"component":"validate","operation":"check","latency":0.035,"error":false,"trace_id":"trace-0001"}
  {"time":"2026-08-03T00:21:34.006Z","app":"billing","endpoint":"compute","duration":54,"status_code":200,"request_no":"f-0001"}
  {"time":1785716494006,"service":"workorder","procedure":"open_order","operation_duration":25,"error_code":0}

24 distinct field names in total, common to all five services: none

In the five records of a single request, three separations show up at once. Field name: the same concept appears under the names duration_ms, duration, latency, operation_duration across five services; a total of 24 distinct field names are counted, and there is not one field common to all five services. Unit: duration is written in three different units — milliseconds, microseconds, seconds — and the field name does not say which. Value space: error is marked across the five services as the text "error", the number 1, the boolean true, the code 500, and the code 7. The fourth separation is in identity: billing carries its own request number, not the trace id coming from the chain, and workorder carries no id at all.

The Convention’s Shared Structure

A telemetry convention is not a tool, it is an agreement: it fixes which field is written under which name, in which unit, in which value space, and how the request’s id is carried from call to call. The six-field schema used in this lesson is as follows:

The alternative to a convention is a mapping table hand-written on the query side: “if you see this field name, count it as duration; if you see this value, count it as an error.” The comparison below measures both approaches with the same set of questions.

Ten Questions, Two Stages

// convention/query.mjs — ten questions are asked in two stages: first over the local schemas with a
// hand-written mapping table (MD11), then over records moved to the shared convention. Every answer
// is compared with the real value: "wrong" if the deviation exceeds 5%, "none" if no field reads.
import { real, LOCAL, SERVICES, FLOW, REQUESTS } from "./record.mjs";
const G = real();
// MD11: the mapping table uses the first field it finds; it does not know the value space.
const MAPPING = { trace: ["trace", "tracing", "trace_id", "request_no"], time_ms: ["time_ms", "time", "ts", "t"],
  service: ["service", "unit", "component", "app"], flow: ["flow", "operation", "action", "procedure", "endpoint"],
  duration_ms: ["duration_ms", "duration", "latency", "operation_duration"],
  status: ["status", "error", "result", "status_code", "error_code"] };
const lookup = (k, field) => { for (const a of MAPPING[field]) if (a in k) return k[a]; return null; };
const num = (v) => (typeof v === "number" && Number.isFinite(v) ? v : null);
const isError = (v) => v === "error" || v === true;      // the two forms the alias table's author knows
// MD12: the pass to the convention happens where the record is generated; workorder (MD10) stays local.
const CONTRACT = ["trace", "time_ms", "service", "flow", "duration_ms", "status"], INCOMPATIBLE = ["workorder"];
const conventional = (z, a) => (INCOMPATIBLE.includes(a.service) ? LOCAL[a.service](z, a)
  : { trace: z.trace, time_ms: z.t0, service: a.service, flow: FLOW, duration_ms: a.duration_ms, status: a.status });
const records = (f) => G.flatMap((z) => z.spans.map((a) => f(z, a)));
const [LOCAL_K, CONV_K] = [records((z, a) => LOCAL[a.service](z, a)), records(conventional)];

const avg = (x) => (x.length ? x.reduce((a, b) => a + b, 0) / x.length : null);
const p95 = (x) => (x.length ? x.slice().sort((a, b) => a - b)[Math.floor(x.length * 0.95)] : null);
const groupBy = (K, field) => { const m = new Map();
  for (const k of K) { const g = lookup(k, field); if (g !== null) m.set(g, [...(m.get(g) ?? []), k]); }
  return m; };
const durationOf = (k) => num(lookup(k, "duration_ms")) ?? 0;
const traceTotal = (K) => [...groupBy(K, "trace").values()].map((v) => v.reduce((x, k) => x + durationOf(k), 0));
const serviceDurations = (K, s) => (groupBy(K, "service").get(s) ?? []).map((k) => num(lookup(k, "duration_ms")))
  .filter((v) => v !== null);
const slowestService = (K) => SERVICES.map((s) => [s, p95(serviceDurations(K, s)) ?? -1]).sort((a, b) => b[1] - a[1])[0];
const realService = (s) => G.flatMap((z) => z.spans.filter((a) => a.service === s).map((a) => a.duration_ms));
const allSpans = G.flatMap((z) => z.spans), endToEnd = G.map((z) => z.spans.reduce((x, a) => x + a.duration_ms, 0));

const QUESTIONS = [
  ["slowest service", "billing", (K) => slowestService(K)[0]],
  ["slowest service's p95 (ms)", p95(realService("billing")), (K) => slowestService(K)[1]],
  ["service producing the most errors", "billing", (K) => SERVICES.map((s) => [s,
    (groupBy(K, "service").get(s) ?? []).filter((k) => isError(lookup(k, "status"))).length])
    .sort((a, b) => b[1] - a[1])[0][0]],
  ["error rate (%)", allSpans.filter((a) => a.status === "error").length / allSpans.length * 100,
    (K) => K.filter((k) => isError(lookup(k, "status"))).length / K.length * 100],
  ["requests per hour", REQUESTS, (K) => ((g) => (g.length ? g.length : null))(
    (groupBy(K, "service").get("gateway") ?? []).filter((k) => num(+lookup(k, "time_ms")) !== null))],
  ["end-to-end average duration (ms)", avg(endToEnd), (K) => avg(traceTotal(K))],
  ["records merged per request", avg(G.map((z) => z.spans.length)),
    (K) => avg([...groupBy(K, "trace").values()].map((v) => v.length))],
  ["requests with validation skipped", G.filter((z) => z.skipped).length,
    (K) => [...groupBy(K, "trace").values()].filter((v) => v.every((k) => lookup(k, "service") !== "validate")).length],
  [`${FLOW} flow's total duration (ms)`, endToEnd.reduce((a, b) => a + b, 0),
    (K) => (groupBy(K, "flow").get(FLOW) ?? []).reduce((x, k) => x + durationOf(k), 0) || null],
  ["workorder span's share of end-to-end (%)", avg(realService("workorder")) / avg(endToEnd) * 100,
    (K) => { const m = groupBy(K, "trace"), i = [...m.values()].flat()
      .filter((k) => lookup(k, "service") === "workorder" && lookup(k, "trace") !== null);
      return i.length ? avg(i.map(durationOf)) / avg(traceTotal(K)) * 100 : null; }],
];
const format = (v) => (v === null ? "-" : typeof v === "number" ? v.toFixed(v < 100 ? 2 : 0) : String(v));
const verdict = (g, v) => (v === null ? "none" : typeof g === "number"
  ? (Math.abs(v - g) / g <= 0.05 ? "correct" : "wrong") : (v === g ? "correct" : "wrong"));

console.log(`${"question".padEnd(42)}${"real".padStart(9)}${"local schema".padStart(14)}` +
  `${"status".padStart(8)}${"convention".padStart(12)}${"status".padStart(9)}`);
const tally = { local: {}, conv: {} };
for (const [label, g, f] of QUESTIONS) {
  const [y, s] = [f(LOCAL_K), f(CONV_K)], [dy, ds] = [verdict(g, y), verdict(g, s)];
  tally.local[dy] = (tally.local[dy] || 0) + 1; tally.conv[ds] = (tally.conv[ds] || 0) + 1;
  console.log(`${label.padEnd(42)}${format(g).padStart(9)}${format(y).padStart(14)}${dy.padStart(8)}` +
    `${format(s).padStart(12)}${ds.padStart(9)}`);
}
const summarize = (o) => `correct ${o.correct || 0}, wrong ${o.wrong || 0}, none ${o.none || 0}`;
console.log(`\nlocal schema: ${summarize(tally.local)}\nconvention  : ${summarize(tally.conv)}`);
const fieldChanges = SERVICES.map((s) => [s, INCOMPATIBLE.includes(s) ? 0
  : CONTRACT.filter((a) => !(a in LOCAL[s](G[0], G[0].spans[0]))).length]);
console.log(`convention ${CONTRACT.length} fields; changed fields: ${fieldChanges.map(([s, n]) => `${s} ${n}`).join(", ")}` +
  `; total ${fieldChanges.reduce((x, [, n]) => x + n, 0)} field changes, ` +
  `incompatible ${INCOMPATIBLE.length}/${SERVICES.length} services`);

question                                       real  local schema  status  convention   status
slowest service                             billing          read   wrong     billing  correct
slowest service's p95 (ms)                    68.00         31000   wrong       68.00  correct
service producing the most errors           billing      validate   wrong     billing  correct
error rate (%)                                 0.56          0.18   wrong        0.54  correct
requests per hour                              6000             -    none        6000  correct
end-to-end average duration (ms)                124         11141   wrong         104    wrong
records merged per request                     4.89          1.95   wrong        3.89    wrong
requests with validation skipped                638          6638   wrong         638  correct
billing_query flow's total duration (ms)     744212         37943   wrong      625249    wrong
workorder span's share of end-to-end (%)      15.99             -    none           -     none

local schema: correct 0, wrong 8, none 2
convention  : correct 6, wrong 3, none 1
convention 6 fields; changed fields: gateway 2, read 6, validate 6, billing 6, workorder 0; total 20 field changes, incompatible 1/5 services

Over the local schemas, none of the ten questions was answered correctly. The real finding is in the distribution: only two stayed unanswered, eight were answered silently wrong. The mapping table mistook the read service’s microsecond value for milliseconds and found a p95 of 31,000 instead of 68, showing read instead of billing as the slowest service. The error rate came out as 0.18% instead of 0.56%, because the table only recognizes two of the encodings. The question about which service produces the most errors was answered “validate” instead of “billing”. The end-to-end duration appeared as 11,141 ms instead of 124 ms. None of these answers carries a warning; the query runs, produces a number, and is drawn on the dashboard.

The two unanswered questions are cheaper for exactly this reason: requests per hour could not be computed at all, because the gateway writes time as text and it could not be converted to a number. The absence of an answer is visible; a wrong answer is not.

Once the convention was applied, six questions were answered correctly. All four that remain trace back to a single source: the service that could not be moved to the convention. The end-to-end duration came out as 104 ms instead of 124, records merged per request as 3.89 instead of 4.89, the flow total as 625,249 instead of 744,212 ms — all three are the gap left by one missing span. The workorder span’s share of the end-to-end total, though, still goes unanswered: that service’s record carries no trace id, so the span cannot be tied to any request.

The Convention’s Cost

The convention consists of six fields. Four services moved to these fields, and that move amounts to 20 field changes in total: gateway added two fields, the other three services each renamed or converted the unit of six fields. The fifth service could not be changed because it is a component brought in from outside, and it stayed incompatible.

The ratio of these three numbers carries the decision. The gain per field is high: 20 field changes bought a rise from zero correct answers to six, meaning one correct answer cost an average of about 3.3 field changes. Against that, the gain is not linear: 80% of the services conformed to the convention, but only 60% of the questions were answered, because a cross-service question depends on the weakest link. A single incompatible span in a request chain breaks every question that includes that span.

The cost of the unanswered question is two-layered here. The first layer is direct: because the workorder span’s share is unknown, that service’s slowdown does not show up in the end-to-end measurement. The second layer is more expensive: before the convention, eight questions were answered wrong, and decisions were made on those answers — for as long as read is shown as the slowest service, the effort to improve it is spent on the wrong service.

Summary

• When five services write in their own naming, a single request scatters across 24 distinct field names, and no field remains common to all five services.
• Over the local schemas, zero of the ten questions were answered correctly; two stayed unanswered, eight were answered silently wrong — a wrong answer is more expensive than no answer, because it is not visible.
• The separation shows up in three layers at once: field name, unit (ms, µs, s), and value space (text, number, boolean, code); an identity separation on top of these blocks merging entirely.
• The six-field convention raised correct answers to six in exchange for 20 field changes across four services.
• 80% of the services conformed, but only 60% of the questions were answered: the single incompatible service breaks all four questions that include its span.

Next Step

Once the convention is in place, questions become answerable, but where the answer is read remains open. A person does not write a query during an incident; they look at the panels in front of them. Which question a panel answers, though, is usually never asked — dashboards fill up over time with panels that get added and that nobody deletes. The next lesson audits a dashboard against a set of questions: how many panels answer a question, how many panels answer none, how many questions remain without a panel, and how many panels it takes to browse through to find an answer to a question.