Type Conversions

The previous lesson reduced the set to 1,880 unique rows. The reading, index, and date fields of these rows are still text and must be converted before they can enter a calculation.

Conversion is the quietest step in this topic. Dropping lowers the row count, and it is visible that something has gone missing. Conversion, on the other hand, leaves no trace: a common number-conversion call extracts a number from every text, raises no error, stops no operation, and the resulting set looks tidy. The number of this lesson is exactly that: how many records were corrupted while looking error-free and how many rows dropped out without any notice.

• CL9. The set is inherited from the previous lesson: 188 rows merged, 1,880 unique rows remain. Same seed, same rows; the set is modeled with node and is fiction.
• CL10. The same number can be written under two separator conventions: in the reading field the decimal separator is a dot or a comma, in the index field the thousands separator is absent or a dot. The same character does two different jobs.
• CL11. Silent coercion is a conversion that produces a wrong value without producing an error. A record is counted as corrupted when the difference between the results of the two conversions exceeds a threshold of 0.005; two floating-point values are not compared with an equality operator.
• CL12. A date that cannot be resolved is not an error but a row that falls out of the filter: it produces no warning, it fails to appear in the next count.

Converting to Numbers

// convert.mjs — the same raw set is a MODEL: every field is text exactly as it came from
// the field form. The set is fiction: 1400 subscribers of the regional measurement network
// and their field visit records; same seed, same rows.
export const SEED = 20260305, SUBSCRIBERS = 1400;
const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507);
  x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; };
export const rng = (t) => { let s = mix(t);
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
export const pick = (u, w) => { let t = 0;
  for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; };
// Last field: the rate at which a visit produces no reading. Regions differ by this rate.
export const REGION = [["north", "NR", 0.28, 21, 0.06], ["south", "SU", 0.22, 17, 0.03],
  ["east", "ET", 0.18, 26, 0.27], ["west", "WK", 0.14, 14, 0.03],
  ["central", "CM", 0.18, 23, 0.08]]
  .map(([name, code, share, base, missing]) => ({ name, code, share, base, missing }));
const STREET = ["Çınar", "Söğüt", "Işık", "Ihlamur", "Zeytin", "Menekşe", "Palmiye", "Ceviz"];
const TARIFF = ["Mesken", "mesken", "MESKEN", " Mesken ", "Konut", "konut", "Ticari",
  "ticari", "TİCARİ", "Is yeri", "İş Yeri", "isyeri", "Sanayi", "sanayi", "Tarimsal"];

export const RAW = [];
let prev = null;
for (let i = 0; i < SUBSCRIBERS; i++) {
  const r = rng(SEED + i);
  const b = REGION[pick(r(), REGION.map((x) => x.share))];
  const base = Math.max(1.2, b.base * Math.exp((r() + r() + r() - 1.5) * 0.62));
  const meter = `${b.code}-${String(1000 + i).padStart(4, "0")}`;
  const shared = prev && prev.region === b.name && r() < 0.03;   // second meter in the same building
  const street = shared ? prev.street : STREET[Math.floor(r() * STREET.length)];
  const door = shared ? prev.door : 1 + Math.floor(r() * 60);
  const unit = shared ? prev.unit : r() < 0.26 ? `/${1 + Math.floor(r() * 9)}` : "";
  prev = { region: b.name, street, door, unit };
  const index0 = 8000 + Math.floor(r() * 11000);
  for (const p of r() < 0.34 ? ["2026-01", "2026-02"] : ["2026-01"]) {
    const q = rng(SEED + 7000 + RAW.length), u = q();
    const truth = Math.round(base * (p === "2026-02" ? 0.94 : 1) * 100) / 100;
    // Missingness shows up in three forms: empty cell, whitespace only, "none" written.
    const miss = u < b.missing ? pick(q(), [0.52, 0.19, 0.29]) : -1;
    const shifted = miss < 0 && q() < 0.008;   // decimal shift: reading written ten times too large
    const written = shifted ? Math.round(truth * 1000) / 100 : truth;
    const reading = miss === 0 ? "" : miss === 1 ? "  " : miss === 2 ? "none"
      : q() < 0.31 ? written.toFixed(2).replace(".", ",") : written.toFixed(2);
    const day = 1 + Math.floor(q() * 27), month = p.slice(5);
    const dateStyle = pick(q(), [0.55, 0.21, 0.24]);
    const date = dateStyle === 0 ? `${p}-${String(day).padStart(2, "0")}`
      : dateStyle === 1 ? `${String(day).padStart(2, "0")}.${month}.2026`
      : `${String(day).padStart(2, "0")}/${month}/2026`;
    const index = index0 + Math.round(truth * (p === "2026-02" ? 2 : 1));
    const dotted = q() < 0.22;
    const addrStyle = pick(q(), [0.30, 0.18, 0.18, 0.16, 0.18]);
    const doorFull = `${door}${unit}`;
    const Cap = `${b.name[0].toUpperCase()}${b.name.slice(1)}`, nameUpper = b.name.toLocaleUpperCase("tr");
    const lower = street.toLocaleLowerCase("tr"), upper = street.toLocaleUpperCase("tr");
    const address = addrStyle === 0 ? `${b.name} mahallesi ${lower} sokak no: ${doorFull}`
      : addrStyle === 1 ? `${Cap} Mahallesi ${street} Sokak No ${doorFull}`
      : addrStyle === 2 ? `${nameUpper} MAHALLESİ ${upper} SOKAK NO ${doorFull}`
      : addrStyle === 3 ? `  ${Cap}  Mahallesi  ${street}  Sokak  No ${doorFull} `
      : `${b.name} mah. ${lower} sk. no:${doorFull}`;
    RAW.push({ subscriber_id: 10001 + i, meter_id: q() < 0.011 ? "" : meter, region: b.name,
      period: p, reading_text: reading,
      index_text: dotted ? `${Math.floor(index / 1000)}.${String(index % 1000).padStart(3, "0")}`
        : `${index}`,
      date_text: date, tariff_text: TARIFF[pick(q(), Array(15).fill(1 / 15))],
      address_text: address, true_m3: truth, written_m3: miss < 0 ? written : null });
    if (q() < 0.09) {   // the same visit entered a second time
      const orig = RAW[RAW.length - 1], style = pick(q(), [0.34, 0.28, 0.38]);
      RAW.push({ ...orig,
        meter_id: style === 0 ? ` ${meter}` : style === 1 ? meter.toLowerCase() : meter,
        address_text: addrStyle === 2 ? `${b.name} mah. ${lower} sk. no:${doorFull}`
          : `${nameUpper} MAHALLESİ ${upper} SOKAK NO ${doorFull}`,
        tariff_text: TARIFF[pick(q(), Array(15).fill(1 / 15))] });
    }
  }
}

// Converts the reading to a number. The decimal separator may be a comma; the traps of
// the conversion itself are measured in the third lesson of this topic, here they are tolerated.
export const toNumber = (t) => { const s = String(t).trim().replace(",", ".");
  return s === "" || !/^\d+(\.\d+)?$/.test(s) ? null : Number(s); };
export const median = (a) => { const l = [...a].sort((x, y) => x - y);
  return l.length % 2 ? l[(l.length - 1) / 2] : (l[l.length / 2 - 1] + l[l.length / 2]) / 2; };
export const fmt = (w) => (h) => h.map((v, j) => (j ? String(v).padStart(w[j])
  : String(v).padEnd(w[0]))).join("");

// Decision of the previous lesson: rows with an empty identifying field were dropped, rows with
// a missing reading were flagged and filled with the region median. 2068 rows enter this step.
const identified = RAW.filter((s) => s.meter_id.trim() !== "");
const measured = identified.filter((s) => toNumber(s.reading_text) !== null);
const regionMedian = new Map(REGION.map((b) => [b.name, median(measured
  .filter((s) => s.region === b.name).map((s) => toNumber(s.reading_text)))]));
export const INCOMING = identified.map((s) => toNumber(s.reading_text) === null
  ? { ...s, reading_text: regionMedian.get(s.region).toFixed(2) } : s);

// Decision of the previous lesson: rows were deduplicated with a key produced from the
// subscriber, the period, and the normalized meter code; 188 rows merged. 1880 rows enter this step.
const dedupe = (l, k) => { const m = new Map();
  for (const s of l) if (!m.has(k(s))) m.set(k(s), s); return [...m.values()]; };
export const DEDUPED = dedupe(INCOMING,
  (s) => `${s.subscriber_id}|${s.period}|${s.meter_id.trim().toUpperCase()}`);

// Two conversion decisions. D1 is the common call: it produces no error and extracts a
// number from every text. D2 recognizes the format first and does not turn text it does
// not recognize into a number.
export const D1 = { reading: (t) => parseFloat(t), index: (t) => parseFloat(t) };
export const D2 = { reading: (t) => Number(t.trim().replace(",", ".")),
  index: (t) => Number(t.replace(".", "")) };
export const sum = (a) => a.reduce((x, z) => x + z, 0);
export const THRESHOLD = 0.005;   // half of the two reported decimal places

if (import.meta.url === `file://${process.argv[1]}`) {
  const count = (f) => DEDUPED.filter(f).length;
  console.log(`model (seed ${SEED}): set inherited from the previous step is ${DEDUPED.length} rows`);
  const fmt1 = fmt([36, 8]);
  console.log("\n" + fmt1(["field format", "rows"]));
  console.log(fmt1(["reading, decimal separator dot", count((s) => !s.reading_text.includes(","))]));
  console.log(fmt1(["reading, decimal separator comma", count((s) => s.reading_text.includes(","))]));
  console.log(fmt1(["index, no thousands separator", count((s) => !s.index_text.includes("."))]));
  console.log(fmt1(["index, thousands separator dot", count((s) => s.index_text.includes("."))]));

  const fmt2 = fmt([32, 10, 9, 13, 12]);
  console.log("\n" + fmt2(["numeric conversion", "corrupted", "error", "total m3", "mean index"]));
  for (const [name, d] of [["D1 common call", D1], ["D2 format-aware", D2]]) {
    const corrupted = DEDUPED.filter((s) => Math.abs(d.reading(s.reading_text)
      - D2.reading(s.reading_text)) > THRESHOLD || Math.abs(d.index(s.index_text)
      - D2.index(s.index_text)) > THRESHOLD).length;
    console.log(fmt2([name, corrupted, 0, sum(DEDUPED.map((s) => d.reading(s.reading_text))).toFixed(1),
      (sum(DEDUPED.map((s) => d.index(s.index_text))) / DEDUPED.length).toFixed(0)]));
  }
  const gap = sum(DEDUPED.map((s) => D2.reading(s.reading_text) - D1.reading(s.reading_text)));
  console.log(`\nD1 produced no error and stopped no operation; it corrupted ` +
    `${count((s) => Math.abs(D1.reading(s.reading_text) - D2.reading(s.reading_text)) > THRESHOLD)} records in the reading field and ` +
    `${count((s) => Math.abs(D1.index(s.index_text) -
      D2.index(s.index_text)) > THRESHOLD)} records in the index field. The reported consumption total ` +
    `comes out ${gap.toFixed(1)} m3 short (the comparison is made with a ${THRESHOLD} m3 threshold, ` +
    `floating-point values are not put through an equality operator)`);
}

model (seed 20260305): set inherited from the previous step is 1880 rows

field format                            rows
reading, decimal separator dot          1338
reading, decimal separator comma         542
index, no thousands separator           1473
index, thousands separator dot           407

numeric conversion               corrupted    error     total m3  mean index
D1 common call                         827        0      41511.2       10558
D2 format-aware                          0        0      41774.8       13481

D1 produced no error and stopped no operation; it corrupted 541 records in the reading field and 407 records in the index field. The reported consumption total comes out 263.6 m3 short (the comparison is made with a 0.005 m3 threshold, floating-point values are not put through an equality operator)

The first table counts how the fields were written: in 542 of the readings the decimal separator is a comma, in 407 of the indexes the thousands separator is a dot; both are mixed within a single field, depending on the habit of the person filling in the form.

The second table is the number of the lesson. D1 is the common call and produces no error — the error column is zero in both rows. Even so, it corrupts 827 rows. In a reading text with a comma, the call reads up to the comma and stops: the value 12,74 becomes 12, and the decimal part is silently discarded. In an index with a thousands separator, the dot is taken for a decimal separator: the value 12.480 becomes 12.48. The first is a small inaccuracy, the second is wrong by a factor of a thousand, and both happen with the same silence.

The totals confirm this. The reported consumption total comes out 263.6 m³ short — a deviation nobody would notice. The mean index falls from 13,481 to 10,558; the index is the field used to compute consumption differences, and any difference computed with a field this corrupted is meaningless. D2 gives zero corrupted records; all it does is recognize the format before converting.

Converting to Dates

// dates.mjs — the same set, this time the date field. Two parsing decisions: one counts
// the first field as the month without knowing the written format, the other recognizes
// the format first.
import { DEDUPED, fmt } from "./convert.mjs";

// T1: format unknown, the first field is counted as the month (a four-digit first field means year).
// T2: format recognized first; in the delimited notation the fields are in day, month, year order.
const T1 = (t) => { const p = t.split(/[-./]/).map(Number);
  const [year, month, day] = p[0] > 999 ? p : [p[2], p[0], p[1]];
  return month >= 1 && month <= 12 && day >= 1 && day <= 31 ? { year, month, day } : null; };
const T2 = (t) => { const p = t.split(/[-./]/).map(Number);
  const [year, month, day] = /^\d{4}-/.test(t) ? p : [p[2], p[1], p[0]];
  return month >= 1 && month <= 12 && day >= 1 && day <= 31 ? { year, month, day } : null; };
const two = (n) => String(n).padStart(2, "0");

const fmt1 = fmt([26, 9, 13, 11, 11]);
console.log(fmt1(["date parsing", "invalid", "wrong month", "correct", "error"]));
for (const [name, f] of [["T1 format unknown", T1], ["T2 format recognized", T2]]) {
  let invalid = 0, wrongMonth = 0, correct = 0;
  for (const s of DEDUPED) { const t = f(s.date_text);
    if (t === null) invalid++; else if (two(t.month) !== s.period.slice(5)) wrongMonth++; else correct++; }
  console.log(fmt1([name, invalid, wrongMonth, correct, 0]));
}

// A number reported from the same set: the number of visits made in the January period.
const january = DEDUPED.filter((s) => s.period === "2026-01").length;
const t1January = DEDUPED.filter((s) => { const t = T1(s.date_text);
  return t !== null && t.month === 1; }).length;
const t1Valid = DEDUPED.filter((s) => T1(s.date_text) !== null).length;
console.log(`\nvisits made in the January period: ${january} by the period field, ` +
  `${t1January} by the dates T1 resolved; T1 could not resolve ${DEDUPED.length - t1Valid} rows at all ` +
  `and these rows silently dropped out of the date filter`);
const fmt2 = fmt([22, 10, 16]);
console.log("\n" + fmt2(["date format", "rows", "T1 resolves it"]));
for (const [name, f, c] of [["year-month-day", (t) => /^\d{4}-/.test(t), "yes"],
  ["day.month.year", (t) => t.includes("."), "if day <= 12"],
  ["day/month/year", (t) => t.includes("/"), "if day <= 12"]])
  console.log(fmt2([name, DEDUPED.filter((s) => f(s.date_text)).length, c]));

console.log(`\nrow accounting: in ${DEDUPED.length}, out ${DEDUPED.length}; 0 rows removed, ` +
  `0 rows silently lost. Had the common conversion been chosen, ${DEDUPED.length - t1Valid} rows ` +
  `would silently drop, 948 fields would be corrupted while looking error-free, and no warning would appear`);

date parsing                invalid  wrong month    correct      error
T1 format unknown               465          361       1054          0
T2 format recognized              0            0       1880          0

visits made in the January period: 1388 by the period field, 805 by the dates T1 resolved; T1 could not resolve 465 rows at all and these rows silently dropped out of the date filter

date format                 rows  T1 resolves it
year-month-day              1022             yes
day.month.year               389    if day <= 12
day/month/year               469    if day <= 12

row accounting: in 1880, out 1880; 0 rows removed, 0 rows silently lost. Had the common conversion been chosen, 465 rows would silently drop, 948 fields would be corrupted while looking error-free, and no warning would appear

The date field is a harsher form of the same problem, because here a wrong value is sometimes no value at all. T1 does not know the written format and applies a common assumption: the first field is the month. Of the 1,880 rows it resolves 1,054 correctly, writes 361 to the wrong month, and cannot resolve 465 at all.

The three outcomes behave differently. A row written to the wrong month stays in the set, carries a well-formed date, and nothing anywhere reveals that it is wrong — a visit made on February 1 appears to have been made on January 2. A row that cannot be resolved produces no error message; it drops off the list at the first operation that filters by date.

The reported number is the sum of the three. The number of visits made in the January period is 1,388 by the period field and 805 by the dates T1 resolved. Of the 583 visits in between, some were written to the wrong month and some became invisible.

The row accounting of the step is this: 1,880 rows in, 1,880 rows out. No row was removed, no row was silently dropped, no record was corrupted. The cost of this step is the number that appears when it is not done: had the common conversion been chosen, 465 rows would silently drop and 948 fields would be corrupted while looking error-free.

Summary

• The same field can be written under two separator conventions: in 542 of the 1,880 rows the decimal separator is a comma, in 407 the thousands separator is a dot.
• The common number-conversion call produces no error and corrupts 827 rows: a reading with a comma loses its decimal part, an index with a thousands separator shrinks a thousandfold.
• The effect varies by field: the consumption total comes out 263.6 m³ short, and the mean index falls from 13,481 to 10,558.
• A date parser that does not recognize the format resolves 1,054 rows correctly, writes 361 to the wrong month, and cannot resolve 465 at all; the January visits are reported as 805 instead of 1,388.
• Row accounting of the step: 1,880 in, 1,880 out; 0 removed, 0 silently lost, 0 records corrupted — had the common conversion been chosen, 465 rows would drop and 948 fields would be corrupted.

Next Step

All the fields are now of the right type, and all 1,880 readings are numbers. Being numbers does not mean they are correct: in some, the decimal point has shifted and the value was written ten times too large, and the conversion faithfully converted them. The next lesson scans these values with the interquartile range and the standard score, counts the part of the sets flagged by the two methods that does not overlap, and measures how much setting the flagged values aside moves the mean.