Model Selection Criteria

All five families were measured in the same order, and what emerged was not a ranking but a distribution. One question remains: which family for the work at hand. This lesson answers that question not with preference but with criteria.

A criterion’s job is to translate the information at hand — which queries an application asks and how often — into a family name. This has two conditions. First, every criterion must tie to a number: “it depends” carries no decision, but “below the third degree, the two models touch the same number” does. Second, the criteria must run together over a mix: the cheapest family for a single query and the cheapest family for the whole workload need not be the same. The lesson first ties the criteria to numbers, then scores five workloads across five families and computes the threshold at which the winner changes hands as the mix shifts.

Five Criteria and the Numbers They Tie To

Access pattern is the strongest criterion, but its strength depends on the pattern. In the graph lesson, the two models touched the same number at the first and second degree (22 and 218); the divergence was 1.18 times at the third degree, 1.56 at the fourth, 65.2 at the fifth. The criterion follows from this: relationship traversal only starts pointing to a family name past the third degree — before that it carries no decision.

Query surface width is how many separate fields carry a filter. If there is one field — and it is the key — the key contract is enough. The moment a second field arrives, the difference the key-value lesson measured opens up: the 359 members over the penalty threshold cost 1 round trip and 6,980 bytes with the store’s own index, 20,000 round trips and 11.6 MB with a key scan.

Write/read ratio divides what is gained on the read side by what is paid on the write side. The key-value lesson measured a 74-byte addition raising write amplification to a factor of 400.6 in the whole-value design, against a factor of 1 for separate keys; on the read side, the whole value costs 1 round trip, the separate key costs 2 in an ordered key space and 66 in a hashed one. The break-even point falls out of division (a calculation): the whole value stays ahead up to one write per 399.6 reads in an ordered key space, one write per 6.1 reads in a hashed one.

The consistency requirement determines where the index gets kept. The secondary index the application keeps itself brings the round trips down from 20,000 to 360, but reads the 243,132-byte list whole on every query — 34.8 times the 6,980 bytes the store’s own index reads — and leaves a window of inconsistency with the record.

Schema change frequency asks whether the check happens at write time or read time. The document lesson measured a query answered from the index silently missing 7,373 records where three schema versions sit side by side; a defensive read gives the correct answer but raises entries scanned from 18,456 to 50,000, a factor of 2.71 — a cost paid continuously if the schema changes often.

From Access Pattern to Family

Every row of the table is an access pattern; the numbers were read from the five lessons’ runs.

Access pattern	Family that cheapens it	Measured difference	Guarantee given up
Point read by a single key	key-value	1 round trip, 1 record; relational side 2 round trips and 9 records, or with a single join 953 bytes instead of 732	Condition on a field, partial update, and join drop out of the contract
Reading a nested structure whole	document	1 round trip, 437 bytes; relational side 4 round trips and 552 bytes, or with a single join 60 rows and 7,840 bytes	Write-time schema checking: 7,373 records silently missed in a three-version collection
Range slice from a wide row	wide-column	a month’s 333 loans in a single request from a 4,000-column row; 263,867 columns scanned with a partition key that does not match the pattern	The partition key serves one pattern; serving a second raises entries to 527,734, write amplification 2.0
Relationship traversal (depth ≥ 3)	graph	217,544 edges at the fifth degree; the join produces 14,190,839 interim rows — 65.2 times	A single declarative statement: the same job splits into five separate statements on the relational side
Time range scan	time series	one shelf’s one day is a single segment and 1,440 entries; 17,280 entries under row layout	The unit of access is the segment, not the measurement: a one-hour window resolves 1,440 entries for 60 measurements
Full scan over a field subset	wide-column	the circulation summary reads 51 pages under family layout, 1,872 under row layout — 36.7 times	All fields of a single record is 5 pages instead of 1; layout cuts both ways
Filtering by a secondary field	document (the store’s own index)	359 members over the threshold, 1 round trip and 6,980 bytes; a key scan costs 20,000 round trips and 11.6 MB	An index assumes a schema; under version drift, a union of two indexes costs 2 round trips, a defensive scan 50,000 entries

The table reads in one direction only: left to right. It does not start as if a family name is already in hand and search for the pattern that fits it; what is in hand is an access pattern, and the table converts it into a family. The fourth column is part of the result too — the cost of what is gained on each row is written on the same row.

The Same Workload Set, Five Families

The table decides for individual patterns one at a time. A real application, though, runs not a single pattern but a mix. The setup below scores the library domain’s five workloads across five families.

NS30: the catalog carries 50,000 books and 20,000 members, a member’s loan history is 4,000 columns in a wide row and a month’s slice is 333 loans, twelve shelf sensors write once every 60 seconds. NS31: the scoring unit is a “unit touched,” and a page, a record, a column, an edge, and a segment entry are counted equally; if a family does not carry a pattern’s mechanics, that workload is a scan of the underlying collection in that family. NS32: mix shares sum to 1, and the cost is a weighted average per request.

Eighteen of the twenty-five cells were read from the five lessons’ runs (measurement), seven were filled by the NS31 rule (assumption): document and graph for point read; key-value, graph, and time series for nested read; document and graph for range slice. The weighted totals and thresholds are of the calculation class.

// selection/scoring.mjs — five library-domain workloads scored across five model
// families. Cells were read from the five lessons' runs; unmeasured cells were
// filled by the NS31 rule. The unit is "unit touched" (record, page, column, edge,
// segment entry). The total is a weighted average, the threshold is solved by
// binary search; the input is fixed, the output is independent of the run.
const FAMILY = ["key-value", "document", "wide-column", "graph", "time-series"];
const WORKLOADS = [                                 // [name, cost across the five families]
  ["point read",            [1, 1, 4, 1, 1440]],    // 1 round trip / 4 pages / unit of access is the segment
  ["nested read",           [1, 1, 4, 22, 1440]],   // document 1 round trip; graph at the first degree
  ["range slice",           [263867, 4000, 333, 4000, 263867]],   // partition key matches: 333
  ["traversal (degree 5)",  [14190839, 14190839, 14190839, 217544, 14190839]],
  ["time range",            [17280, 17280, 17280, 17280, 1440]],  // segment elimination
];
const MIX = [                                       // shares sum to 1 within a mix
  ["catalog-weighted",    [0.70, 0.29, 0.01, 0, 0]],
  ["history-weighted",    [0.30, 0.10, 0.58, 0, 0.02]],
  ["monitoring-weighted", [0.15, 0.05, 0, 0, 0.80]],
];

const total = (k) => FAMILY.map((_, a) => WORKLOADS.reduce((s, [, n], i) => s + k[i] * n[a], 0));
const cheapest = (t) => FAMILY.filter((_, a) => t[a] === Math.min(...t)).join("/");
function threshold(base, target) {                  // target workload's share is p; the rest keeps the base's proportion
  const weights = (p) => { const mix = base.map((w) => w * (1 - p)); mix[target] += p; return mix; };
  const initial = cheapest(total(weights(0)));
  let a = 0, b = 1;
  for (let i = 0; i < 200; i += 1) {
    const p = (a + b) / 2;
    if (cheapest(total(weights(p))) === initial) a = p; else b = p;
  }
  return { share: (a + b) / 2, before: initial, after: cheapest(total(weights(Math.min(1, b * 1.01)))) };
}
const row = (name, h, extra) => name.padEnd(22) + h.map((v) => v.padStart(13)).join("") + "  " + extra;

console.log("matrix — unit touched per workload" +
  `\n${"workload".padEnd(22)}${FAMILY.map((a) => a.padStart(13)).join("")}  ${"spread".padEnd(12)}cheapest`);
for (const [name, n] of WORKLOADS)
  console.log(row(name, n.map(String),
    `${(Math.max(...n) / Math.min(...n)).toFixed(1) + "x"}`.padEnd(12) + cheapest(n)));

console.log(`\nmix — weighted total per request` +
  `\n${"mix".padEnd(22)}${FAMILY.map((a) => a.padStart(13)).join("")}  winner`);
for (const [name, k] of MIX) {
  const t = total(k);
  console.log(row(name, t.map((v) => v.toFixed(2)), cheapest(t)));
}

console.log("\nthreshold — when the winner changes as a single workload is added to the catalog mix");
for (const [name, i] of [["traversal (degree 5)", 3], ["time range", 4]]) {
  const e = threshold(MIX[0][1], i);
  console.log(name.padEnd(22) + `share ${(e.share * 100).toFixed(6).padStart(9)} %` +
    `  (one in every ${String(Math.round(1 / e.share)).padStart(6)} requests)` +
    `  ${e.before} -> ${e.after}`);
}

matrix — unit touched per workload
workload                  key-value     document  wide-column        graph  time-series  spread      cheapest
point read                        1            1            4            1         1440  1440.0x     key-value/document/graph
nested read                       1            1            4           22         1440  1440.0x     key-value/document
range slice                  263867         4000          333         4000       263867  792.4x      wide-column
traversal (degree 5)       14190839     14190839     14190839       217544     14190839  65.2x       graph
time range                    17280        17280        17280        17280         1440  12.0x       time-series

mix — weighted total per request
mix                       key-value     document  wide-column        graph  time-series  winner
catalog-weighted            2639.66        40.99         7.29        47.08      4064.27  wide-column
history-weighted          153388.86      2666.00       540.34      2668.10    153647.66  wide-column
monitoring-weighted        13824.20     13824.20     13824.80     13825.25      1440.00  time-series

threshold — when the winner changes as a single workload is added to the catalog mix
traversal (degree 5)  share  0.000285 %  (one in every 351177 requests)  wide-column -> graph
time range            share 20.389929 %  (one in every      5 requests)  wide-column -> time-series

The first reading of the matrix rows gives the strength of the criterion. Key-value and document are tied on point read and nested read: both touch 1 unit. These two workloads carry no information for choosing between them — the decision is made on the write side, not the read side, with the 399.6 and 6.1 numbers above. Graph is also at 1 unit in the same row; the family’s distinguishing power is not in point read but in the fourth row.

The Mix That Changes the Winner

The mix table shows how the single-pattern decision breaks down. In the catalog-weighted mix, the winner is wide-column (7.29 units); document follows at 40.99. In the history-weighted mix, wide-column wins again but widens the gap to 540.34 against 2,666.00. In the monitoring-weighted mix, the ranking reverses: time series, at 1,440.00 units, stays nine times below wide-column’s 13,824.80. Three mixes, two winners.

The threshold rows carry the real decision. Adding a fifth-degree chain question to the catalog-weighted mix flips the winner from wide-column to graph once that question’s share exceeds 0.000285% — one in every 351,177 requests. Adding a time-range query to the same mix puts the threshold at 20.4%, one in five requests. Five orders of magnitude separate the two thresholds.

The reason is visible in the matrix: the traversal row’s most expensive cell is 14,190,839, the time-range row’s is 17,280. However small a workload’s share of the mix, if it costs millions of units in the wrong family it can determine the total alone. The criterion takes this shape: not an access pattern’s share, but the product of its share and its cost in the wrong family. A rare but very expensive query is more decisive than a frequent, cheap one.

What the Model Does Not Count

The scoring model cannot be used without its limits being written down. Three matter.

First, the unit: NS31 counts a page, a record, and a segment entry equally, though these are different orders of magnitude. The model therefore carries an order-of-magnitude difference, not the ranking itself — the 7.29-against-40.99 gap in the catalog mix falls inside the model’s margin of error, the 1,440-against-13,824 gap in the monitoring mix does not.

Second, writing: the model counts only reading. Wide-column wins two mixes because its partition key happens to be chosen exactly for that mix’s access pattern. The wide-column lesson measured that serving a second pattern means writing the same data twice, entries rising from 263,867 to 527,734, write amplification 2.0 — a cost the model does not count; the gain is repaid there.

Third, the matrix’s seven assumption cells are derived from a rule, not measured. The cells that change the result are of the measurement class (the whole traversal row, the whole time-range row, wide-column’s 333 in the range slice); the next step is always the same: once the decision narrows to one family, the measurement is repeated in that family’s own setup.

Summary

• A criterion carries no decision until it is tied to a number: traversal only points to a family name past the third degree (1.18 → 1.56 → 65.2 times), the whole value stays ahead in an ordered key space up to a ratio of one write per 399.6 reads, and schema versions sitting side by side make the correct answer 2.71 times more expensive.
• Every one of the seven access patterns points to a family, and the cost of what is gained is written on the same row: point read to key-value (no condition on a field), nested read to document (no write-time checking), range slice and full scan to wide-column (a single partition key, a single pattern), traversal to graph (no single-statement query), time range to time series (the unit of access is the segment).
• Scoring the five workloads leaves key-value and document tied on point read and nested read (both 1 unit); these two carry no information for choosing between them, and the decision is made on the write side.
• The cheapest family for a single pattern and for the whole mix are not the same: wide-column wins the catalog- and history-weighted mixes (7.29 and 540.34), time series wins the monitoring-weighted mix (1,440.00).
• The winner is decided not by share alone but by share times cost in the wrong family: the winner changes when traversal is asked once in every 351,177 requests, but a time-range query needs only one in five — five orders of magnitude apart.
• The scoring model counts units equally and measures only reading, so it carries an order-of-magnitude difference, not the ranking itself; wide-column’s gain is repaid by the 2.0 write amplification it does not count.

Next Step

The criteria point to a family. When the decision arrives here — say, keeping the catalog and loan data in the document model — it might seem like the question is settled. It is only starting.

Choosing a family is not choosing a schema. The same library domain can be built in the document model in ways that differ enormously: loan records can be embedded inside the member document or kept by reference in a separate collection, a book’s copies can be an array or their own documents, a date can be text or a time type. This lesson’s scoring matrix wrote the “document” column as a single number; in reality, that column is a range that shifts by multiples depending on the schema chosen, and its width has not been measured yet. The next topic goes inside the document model and starts from the bottom: which types a document’s fields carry, how those types are represented on disk, and what the choice of type does to the bytes stored.