--- title: Outliers source: 'https://academia.sh/en/courses/data-collection-and-preparation/outliers' course: 'Data Collection and Preparation' language: en updated: '2026-10-07T04:31:11+00:00' license: 'CC BY-SA 4.0' objectives: - 'Applies the interquartile range and standard score criteria to the same set and counts the non-overlapping part of the sets they flag' - 'Measures that the standard score hides its own outliers: shows how the standard deviation and the number of flagged rows change once the outliers are set aside' - 'Calculates how much setting aside the flagged rows moves the mean, the median, and the total' - 'Writes the row accounting of the step and shows that the rows set aside move to the quarantine set instead of being deleted' --- # Outliers On the 1,880 inherited readings the interquartile range flags 64 rows and the standard score flags 14; the second set is a subset of the first and the non-overlapping part is 50 rows. The difference comes not from the sensitivity of the methods but from the standard score hiding its own outliers: on 1,880 rows the standard deviation is 19.68 m³, it falls to 6.35 m³ once the 64 flagged rows are set aside, and the same three-sigma criterion this time flags zero rows. Of the 15 readings in the model whose decimal point shifted, the interquartile range catches 15 and the standard score catches 14. Setting the flagged rows aside pulls the mean from 22.22 m³ to 20.14 m³ (2.08 m³, 9.4%), the median from 20.31 to 19.85, and the total from 41,774.8 m³ to 36,578.3 m³. The step enters with 1,880 rows and leaves with 1,816 rows. The previous lesson converted all the fields to the right type, and all 1,880 readings are now numbers. Being a number is not being correct: in a number of readings the decimal point has shifted and the value was written ten times too large, and the conversion faithfully converted them. These values sit at the extremes of the set, violate no rule, and are strange only **relative to their neighbors**. This lesson scans those extremes with two methods. The **interquartile range** rests on ordering and draws a fence outside by a coefficient multiple of the spread; the **standard score** rests on the mean and the standard deviation. Both ask the same question and give different numbers. The number of the lesson is the **non-overlapping part** of these two sets and **how much setting aside the flagged values moves** the reported number. - **CL13.** The set is inherited from the previous two lessons: 188 rows merged, and the reading field was converted to numbers with the format-aware conversion at zero loss. The set is **modeled** with `node` and is **fiction**. - **CL14.** The interquartile range is $Q_3 - Q_1$; the fences stand at $Q_1 - k(Q_3-Q_1)$ and $Q_3 + k(Q_3-Q_1)$. The coefficient $k$ is a decision, it does not come out of the data. - **CL15.** The standard score is $(x - \mu)/\sigma$. Both of its inputs — the mean and the standard deviation — are **computed from the very values to be flagged**; the criterion is affected by what it is looking for. - **CL16.** Flagging an outlier is not a deletion decision. In this lesson the flagged rows move to a **quarantine set**; they leave the set but do not vanish, and their count is written. ## Two Methods, Two Sets ```js // outliers.mjs — the same raw set is a MODEL. The set is fiction: 1400 subscribers of the // regional measurement network and their field visit records; same seed, same rows. The // correct conversion of the previous lesson is applied and the reading field becomes a number. export const SEED = 20260305, SUBSCRIBERS = 1400; const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507); x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; }; export const rng = (t) => { let s = mix(t); return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; }; export const pick = (u, w) => { let t = 0; for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; }; // Last field: the rate at which a visit produces no reading. Regions differ by this rate. export const REGION = [["north", "NR", 0.28, 21, 0.06], ["south", "SU", 0.22, 17, 0.03], ["east", "ET", 0.18, 26, 0.27], ["west", "WK", 0.14, 14, 0.03], ["central", "CM", 0.18, 23, 0.08]] .map(([name, code, share, base, missing]) => ({ name, code, share, base, missing })); const STREET = ["Çınar", "Söğüt", "Işık", "Ihlamur", "Zeytin", "Menekşe", "Palmiye", "Ceviz"]; const TARIFF = ["Mesken", "mesken", "MESKEN", " Mesken ", "Konut", "konut", "Ticari", "ticari", "TİCARİ", "Is yeri", "İş Yeri", "isyeri", "Sanayi", "sanayi", "Tarimsal"]; export const RAW = []; let prev = null; for (let i = 0; i < SUBSCRIBERS; i++) { const r = rng(SEED + i); const b = REGION[pick(r(), REGION.map((x) => x.share))]; const base = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62)); const meter = `${b.code}-${String(1000 + i).padStart(4, "0")}`; const shared = prev && prev.region === b.name && r() < 0.03; // second meter in the same building const street = shared ? prev.street : STREET[Math.floor(r() * STREET.length)]; const door = shared ? prev.door : 1 + Math.floor(r() * 60); const unit = shared ? prev.unit : r() < 0.26 ? `/${1 + Math.floor(r() * 9)}` : ""; prev = { region: b.name, street, door, unit }; // address fields are used in lesson 06 const index0 = 8000 + Math.floor(r() * 11000); for (const p of r() < 0.34 ? ["2026-01", "2026-02"] : ["2026-01"]) { const q = rng(SEED + 7000 + RAW.length), u = q(); const truth = Math.round(base * (p === "2026-02" ? 0.94 : 1) * 100) / 100; // Missingness shows up in three forms: empty cell, whitespace only, "none" written. const miss = u < b.missing ? pick(q(), [0.52, 0.19, 0.29]) : -1; const shifted = miss < 0 && q() < 0.008; // decimal shift: reading written ten times too large const written = shifted ? Math.round(truth * 1000) / 100 : truth; const reading = miss === 0 ? "" : miss === 1 ? " " : miss === 2 ? "none" : q() < 0.31 ? written.toFixed(2).replace(".", ",") : written.toFixed(2); const day = 1 + Math.floor(q() * 27), month = p.slice(5); const dateStyle = pick(q(), [0.55, 0.21, 0.24]); const date = dateStyle === 0 ? `${p}-${String(day).padStart(2, "0")}` : dateStyle === 1 ? `${String(day).padStart(2, "0")}.${month}.2026` : `${String(day).padStart(2, "0")}/${month}/2026`; const index = index0 + Math.round(truth * (p === "2026-02" ? 2 : 1)); const dotted = q() < 0.22; // The address field is not used in this lesson; only the written style is kept. const addrStyle = pick(q(), [0.30, 0.18, 0.18, 0.16, 0.18]); RAW.push({ subscriber_id: 10001 + i, meter_id: q() < 0.011 ? "" : meter, region: b.name, period: p, reading_text: reading, index_text: dotted ? `${Math.floor(index / 1000)}.${String(index % 1000).padStart(3, "0")}` : `${index}`, date_text: date, tariff_text: TARIFF[pick(q(), Array(15).fill(1 / 15))], address_style: addrStyle, true_m3: truth, written_m3: miss < 0 ? written : null }); if (q() < 0.09) { // the same visit entered a second time const orig = RAW[RAW.length - 1], style = pick(q(), [0.34, 0.28, 0.38]); RAW.push({ ...orig, meter_id: style === 0 ? ` ${meter}` : style === 1 ? meter.toLowerCase() : meter, address_style: addrStyle === 2 ? 4 : 2, tariff_text: TARIFF[pick(q(), Array(15).fill(1 / 15))] }); } } } // Converts the reading to a number. The decimal separator may be a comma; the traps of // the conversion itself are measured in the third lesson of this topic, here they are tolerated. export const toNumber = (t) => { const s = String(t).trim().replace(",", "."); return s === "" || !/^\d+(\.\d+)?$/.test(s) ? null : Number(s); }; export const median = (a) => { const l = [...a].sort((x, y) => x - y); return l.length % 2 ? l[(l.length - 1) / 2] : (l[l.length / 2 - 1] + l[l.length / 2]) / 2; }; export const fmt = (w) => (h) => h.map((v, j) => (j ? String(v).padStart(w[j]) : String(v).padEnd(w[0]))).join(""); // Decision of the first lesson: rows with an empty identifying field were dropped, rows with // a missing reading were flagged and filled with the region median. 2068 rows enter this step. const identified = RAW.filter((s) => s.meter_id.trim() !== ""); const measured = identified.filter((s) => toNumber(s.reading_text) !== null); const regionMedian = new Map(REGION.map((b) => [b.name, median(measured .filter((s) => s.region === b.name).map((s) => toNumber(s.reading_text)))])); export const INCOMING = identified.map((s) => toNumber(s.reading_text) === null ? { ...s, reading_text: regionMedian.get(s.region).toFixed(2) } : s); // Decision of the previous two lessons: deduplication (188 rows merged) and format-aware // conversion (0 lost, 0 corrupted records). 1880 rows enter this step. const dedupe = (l, k) => { const m = new Map(); for (const s of l) if (!m.has(k(s))) m.set(k(s), s); return [...m.values()]; }; export const READINGS = dedupe(INCOMING, (s) => `${s.subscriber_id}|${s.period}|${s.meter_id.trim().toUpperCase()}`) .map((s) => ({ ...s, reading_m3: Number(s.reading_text.trim().replace(",", ".")) })); export const THRESHOLD = 0.005; // half of the two reported decimal places export const sum = (a) => a.reduce((x, z) => x + z, 0); export const mean = (a) => sum(a) / a.length; // The quartile is computed by linear interpolation on the sorted array. export const quartile = (a, p) => { const l = [...a].sort((x, z) => x - z); const h = (l.length - 1) * p, i = Math.floor(h); return l[i] + (h - i) * (l[Math.min(i + 1, l.length - 1)] - l[i]); }; export const stdDev = (a) => { const m = mean(a); return Math.sqrt(mean(a.map((v) => (v - m) ** 2))); }; if (import.meta.url === `file://${process.argv[1]}`) { const d = READINGS.map((s) => s.reading_m3); const q1 = quartile(d, 0.25), q3 = quartile(d, 0.75), iqr = q3 - q1; const lo = q1 - 1.5 * iqr, hi = q3 + 1.5 * iqr; const m = mean(d), sd = stdDev(d); console.log(`model (seed ${SEED}): set inherited from the previous steps is ${READINGS.length} rows; ` + `mean ${m.toFixed(2)} m3, standard deviation ${sd.toFixed(2)} m3`); console.log(`quartiles: Q1 ${q1.toFixed(2)}, Q3 ${q3.toFixed(2)}, ` + `interquartile range ${iqr.toFixed(2)}; fences ${lo.toFixed(2)} and ${hi.toFixed(2)} m3`); // Two methods, two flagged sets. A: interquartile range. B: standard score. const A = new Set(), B = new Set(); d.forEach((v, i) => { if (v < lo || v > hi) A.add(i); if (Math.abs((v - m) / sd) > 3) B.add(i); }); const overlap = [...A].filter((i) => B.has(i)).length; const fmt1 = fmt([34, 11, 10, 12]); console.log("\n" + fmt1(["method", "flagged", "overlap", "only this"])); console.log(fmt1(["A interquartile range (1.5x)", A.size, overlap, A.size - overlap])); console.log(fmt1(["B standard score (3 sigma)", B.size, overlap, B.size - overlap])); console.log(`the non-overlapping part of the two sets is ${A.size + B.size - 2 * overlap} rows; ` + `set B is a subset of set A`); // In the model, readings written ten times too large through a decimal shift are flagged. const shifted = READINGS.map((s, i) => [s, i]) .filter(([s]) => s.written_m3 !== null && Math.abs(s.written_m3 - s.true_m3) > THRESHOLD); console.log(`\nthe model has ${shifted.length} readings with a decimal shift; ` + `A flags ${shifted.filter(([, i]) => A.has(i)).length} of them, ` + `B flags ${shifted.filter(([, i]) => B.has(i)).length}`); } ``` ``` model (seed 20260305): set inherited from the previous steps is 1880 rows; mean 22.22 m3, standard deviation 19.68 m3 quartiles: Q1 15.38, Q3 24.59, interquartile range 9.21; fences 1.55 and 38.41 m3 method flagged overlap only this A interquartile range (1.5x) 64 14 50 B standard score (3 sigma) 14 14 0 the non-overlapping part of the two sets is 50 rows; set B is a subset of set A the model has 15 readings with a decimal shift; A flags 15 of them, B flags 14 ``` On the same set the two methods flagged 64 and 14 rows. The second set is a **subset** of the first: the overlap is 14, the non-overlapping part is **50 rows**, and all 50 of those rows are seen only by the interquartile range. One criterion sees a fifth of what the other sees. The difference does not come from sensitivity tuning. Outliers hardly move the quartiles, because even if they travel from one end of the ordering to the other, they do not change where a quartile sits. The two inputs of the standard score, by contrast, are computed directly from those values — the standard deviation of 19.68 m³ is itself a product of the readings written ten times too large. The criterion has folded what it is looking for into its own base. The last line measures this: of the **15 readings** with a shifted decimal point, the interquartile range catches 15 and the standard score catches 14. ## When the Threshold Changes ```js // threshold.mjs — the same set, two questions: as the threshold changes, how many rows each // method flags, and how much setting aside the flagged rows moves the reported number. import { READINGS, quartile, mean, stdDev, sum, fmt } from "./outliers.mjs"; const d = READINGS.map((s) => s.reading_m3); const median = (a) => quartile(a, 0.5); const iqrFlag = (a, k) => { const q1 = quartile(a, 0.25), q3 = quartile(a, 0.75); const spread = q3 - q1; return a.map((v, i) => [v, i]) .filter(([v]) => v < q1 - k * spread || v > q3 + k * spread).map(([, i]) => i); }; const zFlag = (a, k) => { const m = mean(a), s = stdDev(a); return a.map((v, i) => [v, i]).filter(([v]) => Math.abs((v - m) / s) > k).map(([, i]) => i); }; const fmt1 = fmt([24, 11, 11, 12, 13, 11]); console.log(fmt1(["method", "threshold", "flagged", "rows left", "mean m3", "median m3"])); for (const [name, f, k] of [["interquartile range", iqrFlag, 1.5], ["interquartile range", iqrFlag, 3.0], ["standard score", zFlag, 2.0], ["standard score", zFlag, 2.5], ["standard score", zFlag, 3.0]]) { const flagged = new Set(f(d, k)), kept = d.filter((_, i) => !flagged.has(i)); console.log(fmt1([name, k.toFixed(1), flagged.size, kept.length, mean(kept).toFixed(2), median(kept).toFixed(2)])); } console.log(`\nraw set mean ${mean(d).toFixed(2)} m3, median ` + `${median(d).toFixed(2)} m3; between the widest and the narrowest flagging of the two methods ` + `there is a difference of ${iqrFlag(d, 1.5).length - zFlag(d, 3.0).length} rows`); // The standard score hides its own outliers: the deviation is computed from the values to be flagged. const A = new Set(iqrFlag(d, 1.5)), clean = d.filter((_, i) => !A.has(i)); console.log(`\noutliers inflate the deviation themselves: standard deviation is ` + `${stdDev(d).toFixed(2)} m3 over ${d.length} rows; once set A is set aside it falls to ` + `${stdDev(clean).toFixed(2)} m3 over ${clean.length} rows, and the same 3 sigma criterion this time flags ` + `${zFlag(clean, 3.0).length} rows`); // Effect of setting aside on the reported number. const remaining = READINGS.filter((_, i) => !A.has(i)); console.log(`setting aside moves the mean from ${mean(d).toFixed(2)} m3 to ` + `${mean(clean).toFixed(2)} m3 (${(mean(d) - mean(clean)).toFixed(2)} m3, ` + `${(100 * (mean(d) - mean(clean)) / mean(d)).toFixed(1)}%), the median from ` + `${median(d).toFixed(2)} to ${median(clean).toFixed(2)}; the total falls from ` + `${sum(d).toFixed(1)} m3 to ${sum(clean).toFixed(1)} m3`); console.log(`\nrow accounting: in ${READINGS.length}, out ${remaining.length}; ` + `${A.size} rows were set aside into the quarantine set (not deleted, kept for review), ` + `0 rows silently lost`); ``` ``` method threshold flagged rows left mean m3 median m3 interquartile range 1.5 64 1816 20.14 19.85 interquartile range 3.0 18 1862 20.71 20.30 standard score 2.0 15 1865 20.76 20.31 standard score 2.5 14 1866 20.78 20.31 standard score 3.0 14 1866 20.78 20.31 raw set mean 22.22 m3, median 20.31 m3; between the widest and the narrowest flagging of the two methods there is a difference of 50 rows outliers inflate the deviation themselves: standard deviation is 19.68 m3 over 1880 rows; once set A is set aside it falls to 6.35 m3 over 1816 rows, and the same 3 sigma criterion this time flags 0 rows setting aside moves the mean from 22.22 m3 to 20.14 m3 (2.08 m3, 9.4%), the median from 20.31 to 19.85; the total falls from 41774.8 m3 to 36578.3 m3 row accounting: in 1880, out 1816; 64 rows were set aside into the quarantine set (not deleted, kept for review), 0 rows silently lost ``` The table shows that the threshold is a decision and that the two methods react to it differently. In the interquartile range, raising the coefficient from 1.5 to 3.0 brings the flagged rows from 64 down to 18. In the standard score, moving the threshold from 2.0 to 3.0 brings it from 15 to 14: **one row**. The criterion does not react to the threshold, because its denominator has grown so large that no reasonable threshold can divide it. The second block measures this directly. The standard deviation is 19.68 m³ on the raw set and **6.35 m³** once the 64 flagged rows are set aside, and the same three-sigma criterion this time flags **zero rows**. After doing its own work the criterion makes itself unnecessary; wherever the standard score is used in a single pass, it undercounts. The cost of setting aside is in the last line. The mean falls from 22.22 m³ to **20.14 m³**: 2.08 m³, 9.4%. The median moves only 0.46 m³ — from 20.31 to 19.85 — because the median already rests on ordering. The total falls from 41,774.8 to 36,578.3 m³, by 12.4%. The same 64 rows move the median by 2.3%, the mean by 9.4%, and the total by 12.4%; **which number is reported is also a decision**. The row accounting of the step is this: **1,880 rows in, 1,816 rows out.** The 64 rows in between were set aside into the quarantine set — not deleted, but counted and written — and no row was silently lost. Not all of those set aside are erroneous; some are genuinely high consumption, and moving them to the quarantine set makes that investigable. ## Summary - On the same set of 1,880 rows the interquartile range flags 64 rows and the standard score flags 14; the second set is a subset of the first and the non-overlapping part is 50 rows. - Both inputs of the standard score are computed from the values to be flagged, and the criterion dulls itself against what it is looking for: the standard deviation is 19.68 m³ on the raw set and 6.35 m³ once the flagged rows are set aside, and the same three-sigma criterion this time flags zero rows. - Of the 15 readings in the model whose decimal point shifted, the interquartile range catches 15 and the standard score catches 14. - Setting aside pulls the mean from 22.22 to 20.14 m³ (9.4%), the median from 20.31 to 19.85, and the total from 41,774.8 to 36,578.3 m³. - Row accounting of the step: 1,880 in, 1,816 out; 64 rows were set aside into the quarantine set, 0 silently lost. ## Next Step The numeric fields have now been scanned and their extremes written down. There is one more field in the set that has not been looked at: the tariff class. That field is not a number but hand-typed text, and the same class has been written in several forms. The next lesson counts the spellings, applies the merging steps in order, writes how many categories remain after each step, and measures how many rows merging two categories carries into the **wrong class**.