Data Sources

The previous two lessons asked about the type and scale of the fields, but both took the 1,400 rows as given. This lesson asks where the rows came from, and measures a single quantity — one month’s average consumption — from three separate sources. All three numbers were collected correctly, and all three are different.

The difference does not come from a calculation error. Each source carries a decision about who gets into the record, and that decision does not arrive with the data; it is stored in the way the source was built. A survey contains only those who responded, a telemetry stream only the devices that can transmit, a summary report only the published breakdown. The measure at the end of this lesson is the bias a source carries: the difference between the number the source gives and the truth, and which excluded records that difference comes from.

• SD10. The set is the 1,400-row subscriber record of the previous lessons. The set is fictional, and the privilege of this lesson is the following: the model’s true value is known. The true average consumption is 21.31 m³, and no source sees this number.
• SD11. Primary data is data the party doing the analysis collected itself: the household survey filled in during a field visit. Its coverage is determined by two decisions — who was visited and who responded.
• SD12. Secondary data is data someone else collected for another purpose and published: the billing system’s tariff tier summary. There are no raw rows, only tier counts.
• SD13. Generated data is data that accumulates by itself during the operation of a mechanism: the telemetry the meters send. Nobody fills it in, but a device failing to transmit is also a recording decision.
• SD14. In the model, the probability of responding rises with consumption, the probability of transmitting falls with consumption, and the telemetry sends between 20 and 30 days of data each month. These three tendencies are given to the model as input; they represent the structure of the source.

Three Sources, Three Numbers

// sources.mjs — a model that measures the same quantity from three sources.
export const SEED = 20260218, RECORD_COUNT = 1400, TARIFF = [10, 25, 40];
const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507);
  x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; };
const rng = (t) => { let s = mix(t);
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const pick = (u, w) => { let t = 0;
  for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; };
export const REGION = [
  { name: "north", share: 0.28, base: 21, sat: [0.05, 0.10, 0.25, 0.40, 0.20] },
  { name: "south", share: 0.22, base: 17, sat: [0.02, 0.08, 0.30, 0.45, 0.15] },
  { name: "east", share: 0.18, base: 26, sat: [0.20, 0.08, 0.05, 0.12, 0.55] },
  { name: "west", share: 0.14, base: 14, sat: [0.10, 0.15, 0.25, 0.30, 0.20] },
  { name: "central", share: 0.18, base: 23, sat: [0.06, 0.12, 0.28, 0.34, 0.20] },
];
export const RECORDS = Array.from({ length: RECORD_COUNT }, (_, i) => {
  const r = rng(SEED + i);
  const b = REGION[pick(r(), REGION.map((x) => x.share))];
  const t = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62));
  const consumption = Math.round(t * 100) / 100;
  const meters = pick(r(), [0.62, 0.24, 0.10, 0.04]) + 1;
  const visits = pick(r(), [0.38, 0.32, 0.16, 0.08, 0.04, 0.02]);
  const satisfaction = pick(r(), b.sat) + 1;
  const temperature = Math.round((6 + r() * 22) * 10) / 10;
  const elapsed = Math.floor(Math.pow(r(), 1.9) * 210);
  // The source draws are made AFTER the base fields; the base set does not change.
  const responded = r() < 0.30 + 0.014 * consumption;   // heavy consumers respond more
  const transmitted = r() < 0.95 - 0.010 * consumption;  // transmission drops more at high flow
  const coveredDays = 30 - Math.floor(r() * 11); // days the device reported within the month
  return { subscriber_id: 10001 + i, region: b.name, consumption_m3: consumption, meter_count: meters,
    visit_count: visits, satisfaction, temperature_c: temperature, days_elapsed: elapsed,
    responded, transmitted, coveredDays,
    tier: consumption < TARIFF[0] ? 1 : consumption < TARIFF[1] ? 2 : consumption < TARIFF[2] ? 3 : 4 };
});
export const mean = (a) => a.reduce((x, y) => x + y, 0) / a.length;
const TRUTH = mean(RECORDS.map((k) => k.consumption_m3));

// Primary: the household survey collected during the field visit (those visited and those who responded).
export const primary = RECORDS.filter((k) => k.visit_count >= 1 && k.responded);
// Secondary: the mean rebuilt from the billing summary using tier counts.
export const MIDPOINT = [5, 17.5, 32.5, 50];
export const secondaryMean = (represent) => {
  let t = 0;
  for (let b = 1; b <= 4; b++) t += RECORDS.filter((k) => k.tier === b).length * represent[b - 1];
  return t / RECORD_COUNT;
};
// Generated: meter telemetry; only the devices that transmit, missing days as they are.
export const generated = RECORDS.filter((k) => k.transmitted);
const rawTelemetry = (k) => k.consumption_m3 * k.coveredDays / 30;

const gen = (B) => (h) => h.map((v, j) => (j ? String(v).padStart(B[j])
  : String(v).padEnd(B[0]))).join("");

if (import.meta.url === `file://${process.argv[1]}`) {
  console.log(`model (seed ${SEED}): ${RECORD_COUNT} subscribers; true value of the model: ` +
    `mean ${TRUTH.toFixed(2)} m3`);
  const y = gen([28, 9, 10, 12, 11]);
  console.log("\n" + y(["source", "records", "coverage", "mean", "deviation"]));
  const lines = [
    ["primary: field survey", primary.length, mean(primary.map((k) => k.consumption_m3))],
    ["secondary: billing summary", RECORD_COUNT, secondaryMean(MIDPOINT)],
    ["generated: meter telemetry", generated.length, mean(generated.map(rawTelemetry))],
  ];
  for (const [name, n, o] of lines) {
    console.log(y([name, n, `${(100 * n / RECORD_COUNT).toFixed(1)}%`, o.toFixed(2),
      (o - TRUTH >= 0 ? "+" : "") + (o - TRUTH).toFixed(2)]));
  }
  console.log(`\nsubscriber count also varies by source: ${RECORD_COUNT} in the record system, ` +
    `${RECORDS.filter((k) => k.visit_count >= 1).length} in the field list, ` +
    `${generated.length} devices in telemetry`);
  const three = lines.map((s) => s[2]);
  console.log(`the largest gap among the three sources is ${(Math.max(...three) - Math.min(...three)).toFixed(2)} m3`);
}

model (seed 20260218): 1400 subscribers; true value of the model: mean 21.31 m3

source                        records  coverage        mean  deviation
primary: field survey             544     38.9%       22.42      +1.11
secondary: billing summary       1400    100.0%       21.59      +0.28
generated: meter telemetry       1041     74.4%       17.01      -4.30

subscriber count also varies by source: 1400 in the record system, 886 in the field list, 1041 devices in telemetry
the largest gap among the three sources is 5.41 m3

All three sources are built to cover the same 1,400 subscribers, and none of them does. The field survey reaches 544 records, the telemetry 1,041 devices; the billing summary covers all of them in count but shows no raw rows. The deviation column shows three different directions: the survey 1.11 m³ up, the summary 0.28 m³ up, the telemetry 4.30 m³ down. The largest gap is 5.41 m³, and this is close to a quarter of the true value.

The last line is more unsettling. Even the subscriber count changes with the source: 1,400 in the record system, 886 in the field list, 1,041 in telemetry. The question “how many subscribers do we have” has no single answer; each number is the answer to a coverage definition. If a report states a subscriber count without saying which source was counted, that number is one of three different quantities and which one cannot be read.

The 4.30 m³ deviation of the telemetry comes from two separate causes, and both belong to the structure of the source. Devices report not the whole month but between 20 and 30 days, so the raw total falls below a month’s consumption. In addition, the probability of transmitting falls at high consumption, which means the devices that are missing were not chosen at random. When a source delivers missing data, the real question is not how much is missing but which ones are missing.

When the Decision Changes

Coverage is the source’s decision. There is also a decision belonging to the person doing the analysis: how to extract the number from the records at hand. The block below changes a single decision in each source.

// decision.mjs — a single decision is changed in each source, two numbers printed side by side.
import { RECORDS, RECORD_COUNT, mean, primary, generated, secondaryMean, MIDPOINT } from "./sources.mjs";
const TRUTH = mean(RECORDS.map((k) => k.consumption_m3));
const share = (b) => RECORDS.filter((k) => k.tier === b).length / RECORD_COUNT;
// The tier shares are known from the billing summary; the sample is reweighted back to these shares.
const weighted = (l) => { let t = 0;
  for (let b = 1; b <= 4; b++) { const sub = l.filter((k) => k.tier === b);
    if (sub.length) t += share(b) * mean(sub.map((k) => k.consumption_m3)); }
  return t; };
const lines = [
  ["primary", "A: raw mean of respondents", mean(primary.map((k) => k.consumption_m3))],
  ["primary", "B: reweighting to tier shares", weighted(primary)],
  ["secondary", "A: each tier at its midpoint", secondaryMean(MIDPOINT)],
  ["secondary", "B: each tier at its lower bound", secondaryMean([0, 10, 25, 40])],
  ["generated", "A: missing days summed as they are",
    mean(generated.map((k) => k.consumption_m3 * k.coveredDays / 30))],
  ["generated", "B: scaled to the month by covered days", mean(generated.map((k) => k.consumption_m3))],
];
const y = (h) => String(h[0]).padEnd(11) + String(h[1]).padEnd(41)
  + String(h[2]).padStart(9) + String(h[3]).padStart(11);
console.log(y(["source", "decision", "number", "deviation"]));
for (const [k, a, s] of lines) {
  console.log(y([k, a, s.toFixed(2), (s - TRUTH >= 0 ? "+" : "") + (s - TRUTH).toFixed(2)]));
}
const s = lines.map((x) => x[2]);
console.log(`\ntrue value of the model ${TRUTH.toFixed(2)} m3; none of the six numbers has a ` +
  `calculation error`);
console.log(`largest difference within a source ${Math.max(...[0, 2, 4].map((i) =>
  Math.abs(s[i + 1] - s[i]))).toFixed(2)} m3, range of the six numbers ` +
  `${Math.min(...s).toFixed(2)} - ${Math.max(...s).toFixed(2)} m3`);
console.log(`mean of the A decisions of the three sources ` +
  `${mean([s[0], s[2], s[4]]).toFixed(2)} m3 — there is no single decision behind it`);

// The source of the bias: which records fell outside the coverage.
console.log(`\ncoverage tier by tier`);
const y2 = (h) => String(h[0]).padEnd(10) + h.slice(1).map((v) => String(v).padStart(14)).join("");
console.log(y2(["tier", "whole set", "in survey", "survey %", "in telemetry", "telemetry %"]));
for (const b of [1, 2, 3, 4]) {
  const t = RECORDS.filter((k) => k.tier === b).length;
  const a = primary.filter((k) => k.tier === b).length;
  const u = generated.filter((k) => k.tier === b).length;
  console.log(y2([b, t, a, `${(100 * a / t).toFixed(1)}%`, u, `${(100 * u / t).toFixed(1)}%`]));
}

source     decision                                    number  deviation
primary    A: raw mean of respondents                   22.42      +1.11
primary    B: reweighting to tier shares                21.52      +0.21
secondary  A: each tier at its midpoint                 21.59      +0.28
secondary  B: each tier at its lower bound              14.11      -7.19
generated  A: missing days summed as they are           17.01      -4.30
generated  B: scaled to the month by covered days       20.48      -0.83

true value of the model 21.31 m3; none of the six numbers has a calculation error
largest difference within a source 7.48 m3, range of the six numbers 14.11 - 22.42 m3
mean of the A decisions of the three sources 20.34 m3 — there is no single decision behind it

coverage tier by tier
tier           whole set     in survey      survey %  in telemetry   telemetry %
1                     48            15         31.3%            43         89.6%
2                    970           351         36.2%           754         77.7%
3                    348           159         45.7%           224         64.4%
4                     34            19         55.9%            20         58.8%

The six numbers spread between 14.11 and 22.42 m³, and all of them describe the same subscribers, the same month, the same quantity. The largest difference within a source is 7.48 m³ — larger than the difference between sources. This is the most important number of the lesson: how you process the source moves the number as much as which source you choose.

The two decisions on the secondary source do not answer the same question, and unless this distinction is written down the two numbers cannot be compared. Using the tier midpoint is an estimate; using the tier lower bound is a lower bound and answers “at least how much do subscribers consume”. The reported 14.11 m³ is not wrong, but placed next to 21.59 the reader thinks they are seeing two estimates. Whether a number is an estimate or a bound cannot be read from the number itself.

On the primary source, reweighting brings the deviation down from 1.11 to 0.21 m³. The price of the correction is this: the weights come from another source, the billing summary. A second source was needed to remove the bias, and that source has its own decisions. On the telemetry, scaling by covered days brings the deviation down from 4.30 to 0.83 m³; the remaining 0.83 m³ comes from the second cause, the high consumers not transmitting, and the day scaling does not touch it. Correcting one decision does not remove the whole bias; the gap each decision closes is separate.

The last table writes where the bias comes from. In the survey, 31.3% of the subscribers in the first tier are present, against 55.9% of those in the fourth: high consumers are overrepresented in the sample, so the mean shifts upward. In the telemetry the direction reverses — 89.6% of the first tier transmits while only 58.8% of the fourth does. Both sources are fed from the same network, in the same month, from the same subscribers, and are biased in opposite directions. Bias is not a property of the data but a consequence of the source’s coverage decision; without the coverage table written, no deviation can be explained.

Taking the average of the three sources — 20.34 m³ — is ignoring this table. There is no single decision behind that number; it is the arithmetic mean of three separate coverage decisions, and which question it answers is unclear.

Summary

• The same quantity gives three numbers in three sources: field survey 22.42, billing summary 21.59, telemetry 17.01 m³; the model’s true value is 21.31 m³ and no source sees it.
• The subscriber count also changes with the source — 1,400, 886, 1,041 — and without writing which one was counted, this number cannot be read.
• Within a source, the gap opened by a processing decision reaches 7.48 m³ and is larger than the gap between sources.
• An estimate and a lower bound are written in the same form and cannot be told apart: 21.59 is an estimate, 14.11 is a lower bound, and both come out of the same summary.
• Bias comes from the coverage decision: the survey has 55.9% of the fourth tier, the telemetry 58.8%, and the two sources are biased in opposite directions.

Next Step

This lesson measured which records the source leaves outside, but accepted without question the records let inside: all 544 survey rows were counted as filled in correctly, all 1,041 telemetry records as consistent. In a real set this does not hold. A reading may be physically impossible, a field may be left empty, two fields may contradict each other, a record may have been written months ago. The next lesson turns these four questions into four measurable dimensions and produces a number for each; then it tightens the thresholds and shows how many rows are eliminated and in which direction the eliminated rows push the reported number.