---
title: Correlation
source: 'https://academia.sh/en/courses/observability/correlation'
course: 'Observability and Reliability'
language: en
updated: '2026-08-23T07:00:28+00:00'
license: 'CC BY-SA 4.0'
---

# Correlation

Combining the three signals into a single event by correlation id: how many steps and how many signals the same question takes to answer, combined and uncombined, the cost of combining that gets paid once, and the sampling rate's effect on how many events carry all three signals.

All three signals are produced in code, none is bound to a collector, and all of them carry the
same correlation id. But they still live in three separate places. When a question is asked, the
log is read separately, the metric separately, the trace separately, and the link between them
is rebuilt every time. This lesson builds that link once, stores it, and compares the cost of
the two paths.

The question being asked has three parts, and each part falls to a different signal: **why** did
this request fail (log), **where** did it slow down (trace), **is** this slowness normal
(metric). The target is a request that both comes off the slow shelf and belongs to a member
with an unpaid fee; all three questions apply to it at once.

## Three Signals, One Id

The service writes all three signals, and every one of them carries the same correlation id. The
second lesson's uncovered call site is closed here: the notification call now carries the
context too, so no broken trace remains.

**TL6 — the quantity the histogram buckets is the request's reported work, not its wall-clock
duration.** Rationale: wall-clock duration varies from run to run, and requests near a bucket
boundary shift from one run to the next; reported work is the same on every run, so the bucket
shares stay comparable. Wall-clock duration still lives in the span, and that is what answers
the "where did it slow down" question.

```js
// service6.mjs — <name> <port> [downstream...]; all three signals carry the same correlation id
import { createServer } from "node:http";
import { appendFileSync, writeFileSync } from "node:fs";
const [name, port, ...downstream] = process.argv.slice(2);
const BOUNDS = [10, 45, 150, 400];                  // reported-work buckets (ms)
const bucket = new Array(BOUNDS.length + 1).fill(0);
const counter = { requests: 0, errors: 0 };
let rate = 1, no = 0;
const hash = (s) => { let h = 2166136261; for (const c of s) { h ^= c.charCodeAt(0); h = Math.imul(h, 16777619) >>> 0; } return h; };
const wait = (ms) => new Promise((c) => setTimeout(c, ms));
const url = (p, u) => `http://127.0.0.1:${p}/work?member=${u.get("member")}&copy=${u.get("copy")}`;
const reset = () => { writeFileSync(`log-${name}.txt`, ""); writeFileSync(`trace-${name}.txt`, ""); };
reset();

createServer(async (req, res) => {
  const u = new URL(req.url, "http://y").searchParams;
  if (req.url.startsWith("/rate")) {
    rate = Number(u.get("value")); counter.requests = 0; counter.errors = 0; bucket.fill(0); no = 0;
    reset(); res.end("done"); return;
  }
  if (req.url === "/metrics") { res.end(JSON.stringify({ service: name, ...counter, bounds: BOUNDS, bucket })); return; }
  const member = u.get("member"), copy = Number(u.get("copy"));
  const incoming = req.headers["x-trace"];
  const [traceId, parent, s] = incoming ? incoming.split("|") : [`loan-${copy}`, "", ""];
  const sampled = incoming ? s === "1" : hash(traceId) % 1000 < rate * 1000;
  const span = `${name}${(no += 1)}`, t0 = performance.now();
  counter.requests += 1;
  const ownWork = name === "catalog" && copy % 5 === 0 ? 80 : 3;   // reported work: independent of the run
  await wait(ownWork);
  let status = name === "fee" && member === "4023" ? 402 : 200, downstreamWork = 0;
  for (const p of downstream) {
    const y = await fetch(url(p, u), { headers: { "x-trace": `${traceId}|${span}|${sampled ? 1 : 0}` } });
    const body = await y.json(); downstreamWork += body.work;
    if (y.status !== 200) status = y.status;
  }
  const work = ownWork + downstreamWork, duration = Math.round(performance.now() - t0);
  if (status !== 200) counter.errors += 1;
  let k = 0; while (k < BOUNDS.length && work > BOUNDS[k]) k += 1; bucket[k] += 1;
  appendFileSync(`log-${name}.txt`, JSON.stringify({ correlationId: traceId, service: name, member, copy, status }) + "\n");
  if (sampled) appendFileSync(`trace-${name}.txt`,
    JSON.stringify({ trace: traceId, span, parent, service: name, duration, work, bucket: k }) + "\n");
  res.statusCode = status; res.end(JSON.stringify({ work, status }));
}).listen(Number(port));
```

## Two Paths

The query tool answers the same question twice. In the first path, the three stores are read
separately, each is filtered by the target id, and the three results are aligned by hand. In the
second path, every record is first gathered into a single event by correlation id, and then the
question is asked of that one store. The step definition is the same as it has been throughout
the topic: reading a file or an endpoint, parsing, filtering, or aligning are each one step.

```js
// query6.mjs — <rate> [mode]; the same question answered first from three stores, then from the combined event
import { readFileSync } from "node:fs";
const SERVICES = ["loan", "membership", "catalog", "fee", "notification"];
const PORT = { loan: 9601, membership: 9602, catalog: 9603, fee: 9604, notification: 9605 };
const TARGET = "loan-15";                          // both a slow shelf copy and an unpaid fee
const [rate, mode] = process.argv.slice(2);
let step = 0; const count = () => { step += 1; };
const read = (prefix, field) => {                  // 5 file reads + 1 parsing pass
  const k = SERVICES.flatMap((s) => { count(); return readFileSync(`${prefix}-${s}.txt`, "utf8").split("\n").filter(Boolean); });
  count(); return k.map((l) => JSON.parse(l)).map((r) => ({ ...r, key: r[field] }));
};
const readMetrics = async () => { const o = {}; for (const s of SERVICES) { count(); o[s] = await (await fetch(`http://127.0.0.1:${PORT[s]}/metrics`)).json(); } return o; };
const label = (m, k) => `[${k === 0 ? 0 : m.bounds[k - 1]},${k < m.bounds.length ? m.bounds[k] : "+"})`;

// uncombined: each signal is read separately, results are aligned by hand
step = 0;
const g = read("log", "correlationId"), trace = read("trace", "trace"), m = await readMetrics();
const gh = g.filter((r) => r.key === TARGET); count();
const th = trace.filter((r) => r.key === TARGET); count();
const source = gh.find((r) => r.status !== 200 && r.service !== "loan" && r.service !== "membership");
const longest = th.filter((r) => r.service !== "loan").sort((a, b) => b.duration - a.duration)[0];
const rootSpan = th.find((r) => r.service === "loan");
const share = Math.round((100 * m.loan.bucket[rootSpan.bucket]) / m.loan.requests);
count();
const answer = `member ${gh[0].member}, status ${gh[0].status} (source ${source.service}); longest downstream ` +
  `${longest.service}; work in the ${label(m.loan, rootSpan.bucket)} bucket, %${share} of requests`;
if (mode === "question") {
  console.log(`question: why did the ${TARGET} request fail, where did it slow down, is this work normal`);
  console.log(`${"method".padEnd(17)}${"step".padStart(5)}${"signal".padStart(8)}  answer`);
  console.log(`${"uncombined".padEnd(17)}${String(step).padStart(5)}${"3".padStart(8)}  ${answer}`);
}

// combined: the three signals are gathered into a single event by correlation id
step = 0;
const g2 = read("log", "correlationId"), trace2 = read("trace", "trace"), m2 = await readMetrics();
const event = new Map();
for (const r of g2) {
  if (!event.has(r.key)) event.set(r.key, { correlationId: r.key, signals: 2, spans: 0 });
  const e = event.get(r.key);
  if (r.service === "loan") { e.member = r.member; e.status = r.status; }
  if (r.status !== 200 && r.service !== "loan" && r.service !== "membership") e.source = r.service;
}
for (const r of trace2) {
  const e = event.get(r.key); e.spans += 1;
  if (e.signals === 2) e.signals = 3;
  if (r.service === "loan") { e.bucket = label(m2.loan, r.bucket); e.share = Math.round((100 * m2.loan.bucket[r.bucket]) / m2.loan.requests); }
  else if (!e.longest || r.duration > e.longestDuration) { e.longest = r.service; e.longestDuration = r.duration; }
}
count();
const combining = step;
step = 0; count(); const e = event.get(TARGET); count();
if (mode === "question") {
  console.log(`${"combined".padEnd(17)}${String(step).padStart(5)}${"1".padStart(8)}  ` +
    `member ${e.member}, status ${e.status} (source ${e.source}); longest downstream ${e.longest}; ` +
    `work in the ${e.bucket} bucket, %${e.share} of requests`);
  console.log(`combining cost: ${combining} steps, ${g2.length + trace2.length} records -> ` +
    `${event.size} events (paid once, not on every query)`);
  console.log(`\n${"rate".padStart(5)}${"count".padStart(6)}${"3 signals".padStart(12)}${"2 signals".padStart(12)}  missing signal`);
}
const three = [...event.values()].filter((x) => x.signals === 3).length;
console.log(`${rate.padStart(5)}${String(event.size).padStart(6)}${String(three).padStart(12)}` +
  `${String(event.size - three).padStart(12)}  ${event.size - three === 0 ? "-" : "trace (unsampled request)"}`);
```

```bash
cat > run.mjs <<'EOF'
const MEMBER = ["4021", "4022", "4023", "4024"];      // 4023 is the member with an unpaid fee
const PORT = [9601, 9602, 9603, 9604, 9605];
const rate = process.argv[2];
for (const p of PORT) await fetch(`http://127.0.0.1:${p}/rate?value=${rate}`);
for (let i = 1; i <= 100; i += 1)
  await fetch(`http://127.0.0.1:9601/work?member=${MEMBER[(i - 1) % 4]}&copy=${i}`);
EOF
node service6.mjs fee 9604 &
node service6.mjs catalog 9603 &
node service6.mjs notification 9605 &
node service6.mjs membership 9602 9604 &
node service6.mjs loan 9601 9602 9603 9605 &
sleep 1

node run.mjs 1.00 && node query6.mjs 1.00 question
node run.mjs 0.50 && node query6.mjs 0.50
pkill -f "node service6.mjs"
```

```
question: why did the loan-15 request fail, where did it slow down, is this work normal
method            step  signal  answer
uncombined          20       3  member 4023, status 402 (source fee); longest downstream catalog; work in the [45,150) bucket, %20 of requests
combined             2       1  member 4023, status 402 (source fee); longest downstream catalog; work in the [45,150) bucket, %20 of requests
combining cost: 18 steps, 1000 records -> 100 events (paid once, not on every query)

 rate count   3 signals   2 signals  missing signal
 1.00   100         100           0  -
 0.50   100          46          54  trace (unsampled request)
```

## Twenty Steps versus Two

The two answers are word for word the same. The difference is in the cost: the uncombined path
spent twenty steps and went to three signals separately, the combined path spent two steps and
looked at a single event. The breakdown of the twenty steps is instructive too — fifteen are
reads alone (five log files, five trace files, five metric endpoints), two are parsing, three are
filtering and alignment. Because the question has three parts, all three stores had to be
visited.

Combining does not eliminate this work, it does it **once**. The eighteen-step cost is printed
separately, marked with the line: paid once, not on every query. A thousand records were read
and reduced to a hundred events. When a second question is asked, the uncombined path will again
spend twenty steps, the combined path again two. Correlation's payoff does not grow within a
single query, it grows with the number of queries.

The one thing that makes combining possible is that all three signals carry the same id. That id
entered the log in the second lesson, entered the trace context in the fourth; the metric does
not carry an id and cannot — the third lesson measured exactly this, the id getting lost inside
the total. That is why the metric joins the event differently: a single request cannot be tied
to the metric, but **which bucket** the request fell into, and that bucket's share of the
population, can be. That is the combined event's last field, and it is the only thing that
answers "is this slowness normal": request 15's work falls in the bucket that holds twenty
percent of requests. The trace alone says "eighty milliseconds were spent in catalog," the
metric alone says "twenty percent of requests are in this bucket"; combined, they say "this
request is one of the normal twenty percent that runs slow."

The bottom table gives the limit of combining. At full sampling, all hundred of the hundred
events carry three signals. When the sampling rate is halved, the event count does not change —
the log is written for every request, so a hundred events still stand — but fifty-four of them
carry only two signals: their traces were never written at all. On these events "why" and "is
this normal" can still be asked, "where" cannot. The fourth lesson's sampling measure shows up
here a second time: what gets thinned out is the trace record's volume, what gets lost is the
combined event's resolution.

## Summary

- Once all three signals carried the same correlation id, they could be combined into a single
  event: 1000 records reduced to 100 events.
- The same question spent 20 steps and 3 signals uncombined, 2 steps and 1 signal combined; the
  two answers came out word for word the same.
- Combining's 18-step cost is paid once; the payoff accumulates not within a single query but
  across the number of queries.
- Because the metric carries no id, it joined the event not as an individual value but as the
  bucket the request fell into and that bucket's share — the one field that answers "is this
  slowness normal."
- Sampling limits combining's resolution: when the rate dropped to 0.50, 54 of 100 events were
  left without the trace signal, and the "where" question was closed off on those events.

## Next Step

The topic closes here. A request's path is now visible: who asked, where it slowed down, which
service gave birth to the error, where this request falls in the population — all from a single
record, in two steps. There is no such thing anymore as a request lost among five processes.

What remains is this: none of these records says whether the request was **good or bad**. The
measurement says request 15's work falls in the bucket that holds twenty percent of requests; it
does not say whether that twenty percent is acceptable. It says the fee service returned a 402;
it does not say whether twenty-five errors in a hundred requests is normal or a disaster.
Telemetry produces **observation**, not **judgment**; what a judgment would rest on has not been
built yet. The next lesson closes that gap: an indicator is defined from the measured signal, an
objective is attached to the indicator, and the objective is tied to the promise made to the
outside world.