Categorical Encoding

The derivation produced numeric fields from numeric fields, and the fields that carry no number were left untouched in the dataset. region is a name, satisfaction is a grade, tariff_tier is a tier. The way to calculate on them goes through categorical encoding: a number is assigned to each category.

The operation looks mechanical and does not touch the row count, but the number given carries a claim with it. When one category is given 3 and another 5, it is said that the second is “greater” than the first and that the distance between them is 2. The Measurement Scales lesson in the Introduction to Data Analytics course measured in which fields this claim is legitimate; this lesson does not repeat the scale definitions, it counts how many rows the claim moves in the preparation step.

• TF21. Categorical encoding is the decision that turns a category into a number. It does not change the row count, it adds columns. The dataset is carried over from the previous lesson and is fictional.
• TF22. Ordered encoding gives each category an integer and produces a single column. Unordered encoding opens a separate 0/1 column for each category, produces as many columns as there are categories, and claims no order.
• TF23. Encoding an unordered field in order adds to the data an order that does not exist in it. That order is arbitrary: there are several defensible orderings over the same field, and which one was chosen is not visible in the encoded column.
• TF24. In an ordered field, every increasing encoding carries the same order information; the result of a threshold defined by category is independent of the encoding. In an unordered field, the result of the same threshold is determined entirely by the encoding.

Two Encodings, the Same Row Count

// encode.mjs — MODEL: the 3,199-row dataset of the previous lessons and the encoding of
// categorical fields. The dataset is fictional; no interface is imitated.
export const SEED = 20260218, RAW = 1400, PERIOD = ["2026-01", "2026-02", "2026-03"];
const TARIFF = [10, 25, 40];
const mix = (x) => { x = Math.imul(x ^ (x >>> 16), 2246822507);
  x = Math.imul(x ^ (x >>> 13), 3266489909); return (x ^ (x >>> 16)) >>> 0; };
const rng = (t) => { let s = mix(t);
  return () => (s = (s * 1664525 + 1013904223) >>> 0) / 4294967296; };
const pick = (u, w) => { let t = 0;
  for (let i = 0; i < w.length; i++) { t += w[i]; if (u < t) return i; } return w.length - 1; };
const REGION = [["north", 0.28, 21], ["south", 0.22, 17], ["east", 0.18, 26],
  ["west", 0.14, 14], ["central", 0.18, 23]];
const SUBSCRIBER = [], DROPPED = [];
for (let i = 0; i < RAW; i++) {
  const r = rng(SEED + i), b = REGION[pick(r(), REGION.map((x) => x[1]))];
  const k = { subscriber_id: 10001 + i, region: b[0], base: b[2],
    household_size: pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05]),
    satisfaction: pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17]) + 1 };
  (r() < 0.046 ? DROPPED : SUBSCRIBER).push(k);
}
// The dataset produced by the previous lessons; the draws of dropped subscribers and of the
// deduplicated second record are preserved so that the usage values stay the same.
const SATISFACTION = new Map(SUBSCRIBER.map((a) => [a.subscriber_id, a.satisfaction]));
const keys = new Set(SUBSCRIBER.map((a) => a.subscriber_id));
export const LONG = [];
for (const k of [...SUBSCRIBER, ...DROPPED]) {
  const r = rng(SEED + 7000 + k.subscriber_id), n = pick(r(), [0.05, 0.12, 0.21, 0.62]);
  for (let d = 0; d < n; d++) {
    const t = r() < 0.038 ? 0 : Math.round(k.base * Math.exp((r() + r() + r() - 1.5) * 0.62)
      * (1 - d * 0.05) * 100) / 100;
    if (!keys.has(k.subscriber_id)) { r(); continue; }
    LONG.push({ subscriber_id: k.subscriber_id, region: k.region, period: PERIOD[d],
      usage_m3: t, satisfaction: SATISFACTION.get(k.subscriber_id),
      tier: t < TARIFF[0] ? 1 : t < TARIFF[1] ? 2 : t < TARIFF[2] ? 3 : 4 });
    r();
  }
}

export const FIELD = [
  ["region", "unordered", (o) => o.region],
  ["satisfaction", "ordered", (o) => o.satisfaction],
  ["tariff_tier", "ordered", (o) => o.tier],
];
// Ordered encoding gives each category an integer: a single column. Unordered encoding
// opens a separate 0/1 column for each category and claims no order.
export const codeMap = (d) => Object.fromEntries(d.map((a, j) => [a, j + 1]));
export const indicator = (o, f, d) => d.map((a) => (f(o) === a ? 1 : 0));
// In an unordered field the order does not come from the field itself; an ordering is chosen.
const NAMES = [...new Set(LONG.map((o) => o.region))];
const freq = (a) => LONG.filter((o) => o.region === a).length;
export const ORDERING = [
  ["S1 register order", ["west", "east", "south", "north", "central"]],
  ["S2 first seen in the dataset", [...NAMES]],
  ["S3 by row frequency, descending", [...NAMES].sort((a, b) => freq(b) - freq(a))],
];
export const mean = (a) => a.reduce((x, y) => x + y, 0) / a.length;
export const gen = (w) => (h) => h.map((v, j) => (j ? String(v).padStart(w[j])
  : String(v).padEnd(w[0]))).join("");

if (import.meta.url === `file://${process.argv[1]}`) {
  const T = LONG.reduce((a, o) => a + o.usage_m3, 0);
  console.log(`model (seed ${SEED}): in ${LONG.length} rows, ${T.toFixed(1)} m3, ` +
    `${FIELD.length} categorical fields — same as the previous lesson`);
  const y = gen([18, 10, 12, 14, 11, 8, 8]);
  console.log("\n" + y(["field", "scale", "categories", "ordered code", "indicator", "in",
    "out"]));
  let s1 = 0, s2 = 0;
  for (const [name, scale, f] of FIELD) {
    const d = [...new Set(LONG.map(f))];
    s1 += 1; s2 += d.length;
    console.log(y([name, scale, d.length, 1, d.length, LONG.length, LONG.length]));
  }
  console.log(`\nencoding produces no rows: ${LONG.length} -> ${LONG.length}. Ordered encoding adds ` +
    `${s1} columns, unordered encoding adds ${s2} columns; neither drops any row`);
  const o1 = LONG[0], d1 = [...new Set(LONG.map((o) => o.region))];
  console.log(`example row ${o1.subscriber_id} ${o1.region}: indicator columns ` +
    `[${indicator(o1, (o) => o.region, d1).join(" ")}] (column order ${d1.join(" ")})`);

  console.log(`\nthree defensible orderings for region and the codes they give:`);
  const y2 = gen([32, 44]);
  for (const [name, d] of ORDERING) {
    const k = codeMap(d);
    console.log(y2([name, d.map((a) => `${a}=${k[a]}`).join(" ")]));
  }
  console.log(`none of the three was read from the data; there is no order among region names ` +
    `and which ordering was chosen is not visible at all in the encoded column`);
}

model (seed 20260218): in 3199 rows, 63060.6 m3, 3 categorical fields — same as the previous lesson

field                  scale  categories  ordered code  indicator      in     out
region             unordered           5             1          5    3199    3199
satisfaction         ordered           5             1          5    3199    3199
tariff_tier          ordered           4             1          4    3199    3199

encoding produces no rows: 3199 -> 3199. Ordered encoding adds 3 columns, unordered encoding adds 14 columns; neither drops any row
example row 10001 central: indicator columns [1 0 0 0 0] (column order central north south west east)

three defensible orderings for region and the codes they give:
S1 register order                    west=1 east=2 south=3 north=4 central=5
S2 first seen in the dataset         central=1 north=2 south=3 west=4 east=5
S3 by row frequency, descending      north=1 south=2 central=3 east=4 west=5
none of the three was read from the data; there is no order among region names and which ordering was chosen is not visible at all in the encoded column

Encoding is the second step in this course that leaves the row count unchanged: the in and out columns are 3,199 in all three fields. What changes is the column count. Ordered encoding produces a single column per field and comes to 3 columns for three fields; unordered encoding opens a column for each category and comes to 14 columns for the same three fields. That is the cost of unordered encoding and it grows with the number of categories; its gain is that it puts no category before or after another.

The example row makes this concrete: a row in the central region sets only one of the five indicators to 1, and the remaining four are 0. In these five numbers no region is greater than another. In ordered encoding, on the other hand, the same information comes down to a single number, and that number settles somewhere.

The lower block opens the core question of the lesson: where will the code of the region field come from. The field itself has no order, so an ordering is chosen, and all three choices are defensible: following the fixed order of the region register, keeping the order in which they are first seen in the dataset, and arranging them in descending order of row frequency. The three give three different codes. west is 1 in register order and 5 in frequency order. Someone looking at the encoded column sees only a number; which ordering was chosen leaves no trace in that column.

There is no such choice in the satisfaction and tariff_tier fields. The order among the tiers comes from the field itself, and the person encoding cannot change it, only choose by which numbers it will be represented. The difference is here, and the next section measures it with row counts.

The Same Threshold, Three Different Sets of Rows

// threshold.mjs — the same threshold selects three different sets of rows under three
// orderings. In an ordered field the same experiment gives a single set; the difference
// comes not from the encoding but from the scale.
import { LONG, ORDERING, codeMap, mean, gen } from "./encode.mjs";

const LIMIT = 3; // the upper half of five categories; codes 3, 4, and 5 are selected
const y = gen([32, 25, 9, 13]);
console.log(`threshold: "rows whose region code is ${LIMIT} and above"`);
console.log("\n" + y(["ordering", "selected regions", "rows", "mean m3"]));
const sets = [];
for (const [name, d] of ORDERING) {
  const k = codeMap(d), picked = LONG.filter((o) => k[o.region] >= LIMIT);
  sets.push(new Set(picked.map((o) => `${o.subscriber_id}|${o.period}`)));
  console.log(y([name, d.filter((a) => k[a] >= LIMIT).join(" "), picked.length,
    mean(picked.map((o) => o.usage_m3)).toFixed(3)]));
}
const inHowMany = LONG.map((o) => sets.filter((s) => s.has(`${o.subscriber_id}|${o.period}`)).length);
const count = (n) => inHowMany.filter((x) => x === n).length;
console.log(`\nof ${LONG.length} rows, ${count(3)} were selected in all three decisions, ` +
  `${count(2)} in two, ${count(1)} in only one, ${count(0)} in none`);
const n = sets.map((s) => s.size);
const m = sets.map((s) => mean(LONG.filter((o) => s.has(`${o.subscriber_id}|${o.period}`))
  .map((o) => o.usage_m3)));
console.log(`same data, same threshold, same calculation: between ${Math.min(...n)} and ${Math.max(...n)} rows ` +
  `and between ${Math.min(...m).toFixed(3)} and ${Math.max(...m).toFixed(3)} m3`);

// In an ordered field the threshold is defined by category: "good and above". Every
// increasing encoding puts the same category on the same side, because the order is in the field itself.
const GRADE = ["very bad", "bad", "fair", "good", "very good"];
const CODE = [["K1 equal intervals", [1, 2, 3, 4, 5]], ["K2 widening intervals", [2, 5, 9, 14, 20]]];
const y2 = gen([26, 22, 15, 9, 13]);
console.log(`\nthreshold: "rows whose satisfaction is ${GRADE[3]} and above"`);
console.log("\n" + y2(["encoding", "codes", "threshold code", "rows", "mean m3"]));
for (const [name, K] of CODE) {
  const e = K[GRADE.indexOf("good")], picked = LONG.filter((o) => K[o.satisfaction - 1] >= e);
  console.log(y2([name, K.join(" "), e, picked.length, mean(picked.map((o) => o.usage_m3)).toFixed(3)]));
}
console.log(`both encodings selected the same rows: because the order is in the field and not ` +
  `in the code, the threshold is independent of the encoding`);

threshold: "rows whose region code is 3 and above"

ordering                                 selected regions     rows      mean m3
S1 register order                     south north central     2166       19.741
S2 first seen in the dataset              south west east     1678       18.480
S3 by row frequency, descending         central east west     1633       20.661

of 3199 rows, 0 were selected in all three decisions, 2278 in two, 921 in only one, 0 in none
same data, same threshold, same calculation: between 1633 and 2166 rows and between 18.480 and 20.661 m3

threshold: "rows whose satisfaction is good and above"

encoding                                   codes threshold code     rows      mean m3
K1 equal intervals                     1 2 3 4 5              4     1615       19.584
K2 widening intervals                2 5 9 14 20             14     1615       19.584
both encodings selected the same rows: because the order is in the field and not in the code, the threshold is independent of the encoding

The upper table is the finding of the lesson. The threshold is a single sentence and looks reasonable: “rows whose region code is 3 and above”, that is, the upper half of five categories. The same sentence, on the same data, selects three different sets of regions under three defensible orderings. The numbers of selected rows are 2,166, 1,678, and 1,633; the mean usage ranges from 18.480 m³ to 20.661 m³. All three numbers were calculated correctly, and none of them raises an error.

The line below gives the measure of the matter: none of the 3,199 rows is selected in all three decisions. 2,278 rows are selected in two decisions, and 921 rows in only one decision. That is, the expression “the regions whose code is 3 and above” has no fixed meaning; each ordering selects a different three, and the three selections do not intersect. This is the exact definition of the wrong result that encoding an unordered field in order produces: the result is not a wrongly calculated number but a number that does not belong to the data. The source of the number is how the region names were lined up.

The second table repeats the same experiment on an ordered field, and the difference is clear. The threshold is again a single sentence: “rows whose satisfaction is good and above”. The distances between the two encodings are entirely different, since one counts the tiers as equally spaced and the other widens the gaps among the upper tiers, but both select the same 1,615 rows and give the same 19.584 m³. The reason is simple: the order is in the field itself, the job of the code is to carry it without breaking it, and every increasing encoding does so.

Put side by side, the two tables give a decision rule. If a field is ordered, ordered encoding is legitimate, and the choice of code does not move thresholds based on categories. If a field is unordered, ordered encoding makes the decision unreadable from the code: the threshold has been applied not to the data but to the encoding order. Unordered encoding closes this trap, because in indicator columns there is no such thing as “3 and above”; which categories were selected has to be written by name, and at that moment the choice becomes visible.

The row accounting of the lesson closes this way: 3,199 rows went in, 3,199 rows came out, and encoding dropped not a single row. In return, a single threshold that came after the encoding selected sets of rows from the same dataset that vary between 1,633 and 2,166, and whether 921 rows were counted or not depended on a single ordering decision. Encoding itself is lossless; the loss appears at the first step where the false order carried by the code is used.

Summary

• Encoding does not change the row count: 3,199 rows went in, 3,199 rows came out. Ordered encoding adds 3 columns for three fields, and unordered encoding adds 14 columns for the same three fields.
• In an unordered field the code comes not from the data but from a chosen ordering: the register, first-seen, and frequency orderings give the west region different codes between 1 and 5.
• The same threshold selects 2,166, 1,678, and 1,633 rows under three orderings; the mean varies between 18.480 and 20.661 m³ and no calculation raises an error.
• None of the 3,199 rows is selected in all three decisions; whether 921 rows are counted depends on a single ordering decision.
• In an ordered field the choice of code does not move the threshold: two different encodings give the same 1,615 rows and the same 19.584 m³, because the order is in the field itself and not in the code.

Next Step

This topic reshaped the dataset according to the question: tables were merged, the shape was changed, groups were summarized, new fields were derived, and categories were encoded. At every step the rows that went in, came out, and disappeared were counted, but all of these counts were made one by one with small, hand-written pieces of code, and none of them stands together in one place. Someone preparing the same dataset a second time cannot find which threshold was chosen with which justification or which ordering was used. The next lesson takes up documenting the preparation steps with a script: it counts how many steps can be reproduced and sums up the row accounting across the seventeen lessons of the course end to end.