Data Fabric and Data Mesh

In the previous lesson six subsets were taken out, and the partition key, the refresh order and the derivation rule of each were determined by a single party. The region unit only read the set written for it. What would happen if the decision did not stay there: if the region itself chose the layout key, set up the write schedule itself, and published its own set like a product.

This question looks like an architectural preference, yet it changes two numbers, and those two numbers can be measured. The first is the number of parties that make the layout decision: in how many separate places the question “by which key will these rows be filed” is answered. The second is the number of layouts an end-to-end query touches: how many separate partitioning decisions a single question that covers all regions passes over. The lesson measures these two numbers on three arrangements, then sweeps the per-file constant on which every cost of this course rests, to show which result depends on that constant.

• SA51. The source, the pipeline and the storage layout are a fiction and are defined within the lesson. No warehouse, lake, catalog product, query engine or cloud storage service is called. Seed 20260218.
• SA52. A data fabric is an arrangement in which the rows are written in more than one place but the key decision stays in a single standard. A data mesh is an arrangement in which the unit that produces the rows owns both the writing and the key choice; each unit publishes its own data product.
• SA53. The number of addresses the query knows is written separately as a third number. The fabric has a single access point, the mesh five. This number does not change the cost.
• SA54. There are two question mixes. Mix A is five local questions (each unit one period of its own region) plus one countrywide question (the same period, all regions). Mix B adds an unpruned question on top of A: all periods of a single subscriber.
• SA55. The procedure of ownership and governance was measured in the Governance, Risk and Compliance course of the Cybersecurity curriculum, and the number of the ownership gap in the Data Governance lesson of the Introduction to Data Engineering course; it is not repeated here. This lesson counts ownership only as which layout is in whose hands.
• SA56. The per-file constant is a choice and is swept in this lesson: 60, 120, 240, 480. Cost is not a share but an integer; if a file share is printed, the resolution step is 0.0091.

How Many Hands the Decision Is In

The three arrangements are built on the same 2,200 rows. The central setup has a single store, and its key is period. In the fabric the rows are written to five separate places, but a single party decides what the key will be, and all five use period. In the mesh each unit chooses its own key: two units choose period, one period+type, one does no partitioning at all, and one chooses only type. These choices are not made in bad faith; none of them wrote down which question it was made for.

# FICTIONAL. The meter readings of the measurement network, the pipeline and the storage layout
# are defined within the lesson; no warehouse, lake, catalog product or query engine is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value

REGION = ["east", "west", "north", "south", "central"]
TYPE = ["residential", "commercial"]
PERIODS = [f"2026-{a:02d}" for a in range(1, 13)]
SUBSCRIBERS = {}
for i in range(200):
    r = rng(SEED + 37 * i)
    SUBSCRIBERS[f"A{i + 1:03d}"] = {"region": REGION[int(r() * 5)], "type": TYPE[int(r() * 2)],
                                    "start": 10000 + int(r() * 60000)}


def generate_source():
    records = []
    for i, (sub, a) in enumerate(SUBSCRIBERS.items()):
        e = a["start"]
        for p in PERIODS:
            r = rng(SEED + 101 * i + 7 * PERIODS.index(p))
            e += 4 + int(r() * 46)
            records.append({"subscriber": sub, "period": p, "region": a["region"], "type": a["type"],
                            "index": e, "unit": "m3"})
    return records


def s1_screen(records):
    REQUIRED = ("subscriber", "period", "region", "index", "unit")
    keep = [k for k in records if all(a in k for a in REQUIRED)]
    return keep, len(records) - len(keep)


def s2_dedupe(records):
    seen, keep = set(), []
    for k in records:
        key = (k["subscriber"], k["period"])
        if key in seen:
            continue
        seen.add(key)
        keep.append(k)
    return keep, len(records) - len(keep)


def s3_consumption(records):
    table = {(k["subscriber"], k["period"]): k for k in records}
    out, dropped = [], 0
    for k in records:
        i = PERIODS.index(k["period"])
        if i == 0:
            continue
        prev = table.get((k["subscriber"], PERIODS[i - 1]))
        if prev is None:
            dropped += 1
            continue
        out.append(dict(k, m3=int(k["index"]) - int(prev["index"])))
    return out, dropped

COLUMNS = ("subscriber", "period", "region", "type", "index", "unit", "m3")
FILE_CONSTANT = 120        # fixed cost of touching one file , in cells


def run_pipeline(source):
    v, _ = s1_screen(source)
    v, _ = s2_dedupe(v)
    v, _ = s3_consumption(v)
    return v


def write(records, partitioning=()):
    """Distributes the records into files by the partition key."""
    layout = {}
    for k in records:
        key = tuple(k[a] for a in partitioning)
        layout.setdefault(key, []).append(k)
    return layout


def query(layout, partitioning, selected, filter_=None, columnar=True):
    """A field in the partition key prunes files , any other field makes the whole file be read."""
    filter_ = {} if filter_ is None else filter_
    pruning = {a: v for a, v in filter_.items() if a in partitioning}
    remaining = {a: v for a, v in filter_.items() if a not in partitioning}
    files, cells, matched = 0, 0, 0
    columns_read = len(selected) if columnar else len(COLUMNS)
    for key, record in layout.items():
        if pruning:
            match = dict(zip(partitioning, key))
            if any(match[a] != v for a, v in pruning.items()):
                continue
        files += 1
        cells += len(record) * columns_read
        matched += sum(1 for k in record if all(k[a] == v for a, v in remaining.items()))
    return {"files": files, "cells": cells, "matching_rows": matched,
            "cost": files * FILE_CONSTANT + cells}


ALL_ROWS = run_pipeline(generate_source())
SELECTED = ("subscriber", "m3")
P07 = {"period": "2026-07"}
SUB = {"subscriber": "A007"}
# Three arrangements. Who owns the key choice: the center , a single standard , the unit.
CENTRAL = {"": ("period",)}
FABRIC = {r: ("period",) for r in REGION}            # distributed write , single key decision
MESH = {"east": ("period",), "west": ("period", "type"), "north": (),
        "south": ("period",), "central": ("type",)}  # each unit chose its own key
COUNTS = {"central": (1, 1, 1), "fabric": (1, 5, 1), "mesh": (5, 5, 5)}


def build(keys):
    if list(keys) == [""]:
        return {"": write(ALL_ROWS, keys[""])}
    return {r: write([k for k in ALL_ROWS if k["region"] == r], keys[r]) for r in keys}


def measure(keys, layouts):
    """Five local questions (each unit the period of its own region) , one countrywide question ,
    one subscriber question. In the central setup all three go to the single layout."""
    if list(keys) == [""]:
        d, a = layouts[""], keys[""]
        local = sum(query(d, a, SELECTED, dict(P07, region=r))["cost"] for r in REGION)
        return local, query(d, a, SELECTED, P07)["cost"], query(d, a, SELECTED, SUB)["cost"]
    local = sum(query(layouts[r], keys[r], SELECTED, P07)["cost"] for r in REGION)
    sub = sum(query(layouts[r], keys[r], SELECTED, SUB)["cost"] for r in REGION)
    return local, local, sub


SETUPS = [("central", CENTRAL), ("fabric", FABRIC), ("mesh", MESH)]
print(f"rows {len(ALL_ROWS)} , file constant {FILE_CONSTANT} cells\n")
print("setup      deciding parties  layouts touched  addresses  files   local x5  countrywide  subscriber")
for name, keys in SETUPS:
    lay = build(keys)
    y, u, a = measure(keys, lay)
    t, dd, ad_r = COUNTS[name]
    print(f"{name:<10} {t:>16}  {dd:>15}  {ad_r:>9}  {sum(len(x) for x in lay.values()):>5}"
          f"   {y:>8}  {u:>11}  {a:>10}")

print("\nin the mesh, local question per unit:  key / files / cost")
lay = build(MESH)
for r in REGION:
    res = query(lay[r], MESH[r], SELECTED, P07)
    print(f"  {r:<7} {str(MESH[r]):<18} {len(lay[r]):>3} files , {res['files']:>2} touched"
          f" , cost {res['cost']:>5}")

rows 2200 , file constant 120 cells

setup      deciding parties  layouts touched  addresses  files   local x5  countrywide  subscriber
central                   1                1          1     11       2600          520        5720
fabric                    1                5          1     55       1000         1000       11000
mesh                      5                5          5     47       3140         3140       10040

in the mesh, local question per unit:  key / files / cost
  east    ('period',)         11 files ,  1 touched , cost   188
  west    ('period', 'type')  22 files ,  2 touched , cost   302
  north   ()                   1 files ,  1 touched , cost  1132
  south   ('period',)         11 files ,  1 touched , cost   200
  central ('type',)            2 files ,  2 touched , cost  1318

The first three columns carry the lesson’s two numbers. In the central setup the decision is in one hand and the end-to-end query touches a single layout: 1 and 1. In the fabric the rows are spread over five places but the key decision is still in one hand: 1 and 5. In the mesh both are five. The address column shows the third distinction; the fabric puts five stores behind a single access point, the mesh does not. The number of addresses has no contribution at all to the cost columns: the fabric pays 1,000 because that is where the rows are; that the address is single does not change the cells read. An access layer reduces not the cost but who has to know.

Two things are read together in the cost columns. On the local questions the fabric is three times cheaper than the central setup (1,000 against 2,600), because in the central layout asking for one period of one region makes it read all the rows of that period, and five regions read the same file five times. In the distributed write the five local questions land on disjoint sets, and their total comes out exactly equal to the cost of the countrywide question. On the countrywide question, on the other hand, the central setup pays 520 and the fabric 1,000: the same 400 cells are read, and the difference is that the file constant is paid five times instead of once. On the subscriber question a gap opens, 5,720 against 11,000, because there is no key there that prunes and all fifty-five files are touched.

The 3,140 of the mesh, on the other hand, is the number of unmeasured key choices, not of distribution. The sub-table separates it unit by unit: east 188, south 200 and west 302 pay little, while north 1,132 and central 1,318 pay a lot. The decision of two units alone costs 2,450, more than two thirds of the mesh’s local total. North does no partitioning, so it reads the whole of its single file; central chose type, so the period filter prunes nothing. The rule of the course shows up here for the second time: a layout whose query is not written down is counted as unmeasured, and five unmeasured decisions cost more than one measured decision. Distributing ownership does not create this gap; distributing ownership without a measure does.

When the File Constant Is Swept

Every cost of this course rests on one choice: the price of touching a file is 120 cells. The number is a fiction and could have been otherwise. The only way to learn how much all the comparisons above owe to that choice is to change the constant and repeat the same measurement.

# The definitions of the previous block carry on: build, measure, SETUPS, COUNTS, ALL_ROWS, FILE_CONSTANT.
ARRANGEMENTS = {name: (keys, build(keys)) for name, keys in SETUPS}
NAMES = ("central", "fabric", "mesh")


def mixes():
    """A: five local questions + one countrywide question. B: A , plus one subscriber question."""
    c = {}
    for name, (keys, layouts) in ARRANGEMENTS.items():
        local, country, sub = measure(keys, layouts)
        c[name] = (local + country, local + country + sub)
    return c


print("const  central-A   fabric-A     mesh-A  central-B   fabric-B     mesh-B")
for constant in (60, 120, 240, 480):
    FILE_CONSTANT = constant
    c = mixes()
    print(f"{constant:>5}  " + "  ".join(f"{c[a][0]:>9}" for a in NAMES)
          + "  " + "  ".join(f"{c[a][1]:>9}" for a in NAMES))

# The cost is linear in the constant: cost(constant) = cells + constant x touched files.
FILE_CONSTANT = 0
base = mixes()
FILE_CONSTANT = 1
slope = {a: tuple(mixes()[a][i] - base[a][i] for i in (0, 1)) for a in NAMES}
FILE_CONSTANT = 120
print("\nsetup          mix A                   mix B")
for a in NAMES:
    print(f"{a:<9}  {base[a][0]:>7} + const x {slope[a][0]:>3}"
          f"      {base[a][1]:>7} + const x {slope[a][1]:>3}")
threshold = (base["central"][0] - base["fabric"][0]) // (slope["fabric"][0] - slope["central"][0])
print(f"\nmix A, the constant at which the fabric and the central setup are equal: {threshold}")
print("independent of the constant: parties deciding the layout / layouts touched , "
      + " , ".join(f"{a} {COUNTS[a][0]}/{COUNTS[a][1]}" for a in NAMES))

const  central-A   fabric-A     mesh-A  central-B   fabric-B     mesh-B
   60       2760       1400       5440       7820       9100      12660
  120       3120       2000       6280       8840      13000      16320
  240       3840       3200       7960      10880      20800      23640
  480       5280       5600      11320      14960      36400      38280

setup          mix A                   mix B
central       2400 + const x   6         6800 + const x  17
fabric         800 + const x  10         5200 + const x  65
mesh          4600 + const x  14         9000 + const x  61

mix A, the constant at which the fabric and the central setup are equal: 400
independent of the constant: parties deciding the layout / layouts touched , central 1/1 , fabric 1/5 , mesh 5/5

The sweep gives results of three different kinds. The first are those independent of the constant: the number of parties deciding the layout and the number of layouts touched stay 1/1, 1/5 and 5/5 at all four values. The two main numbers of this lesson are the numbers of an organizational decision, not of a cost model, and the file constant does not touch them. In the same way the mesh being more expensive than the fabric holds at all four values; the price of unmeasured key choices is independent of the constant.

The second is the ranking, which depends on the constant, and it is the most important warning of this lesson. In mix A the fabric is cheaper than the central setup at 60, 120 and 240, and not at 480. The linear form gives the reason: the fabric pays 10 per file against 800 cells, the central setup 6 per file against 2,400 cells. The cell advantage of the fabric is fixed, its file disadvantage is multiplied by the constant; the two are equal at exactly 400. That 120 was chosen makes the sentence “distributed writing makes local questions cheaper” true; had 480 been chosen, the same sentence would be false. A comparison of layouts cannot be passed on without writing the constant it rests on.

The third is the result that depends on the question mix, and sweeping the constant does not remove it. Mix B adds a single unpruned question on top of A, and the ranking is reversed at all four values: it starts with central 7,820 and fabric 9,100, and at 480 the difference is 14,960 against 36,400. The slope column explains it: in B the file coefficient of the fabric is 65, that of the central setup 17. A single question for which there is no pruning key turns the file count of the distributed write directly into cost. Distributing breaks things too; where it breaks them is the questions that no key helps.

Summary

• Distributing ownership changes two numbers: the number of parties that make the layout decision and the number of layouts an end-to-end query touches. The central setup gives 1/1, the data fabric 1/5, the data mesh 5/5.
• What separates the fabric from the mesh is not the distribution of the writing but the distribution of the key decision; the number of addresses the query knows is a third number and does not change the cost at all.
• The local total of the mesh is 3,140, that of the fabric 1,000; the source of the difference is not distribution but unmeasured key choices, and the decision of two units alone costs 2,450.
• When the file constant is swept, the two main numbers and the mesh being more expensive than the fabric do not change at any of the four values; what changes is the ranking of the fabric and the central setup, and the point of equality is 400.
• The cost is linear in the constant (cells + constant x files), so a comparison cannot be passed on without writing the constant it rests on.
• When a single unpruned question is added, the distributed setups lose at all four constant values: in mix B the central setup pays 8,840, the fabric 13,000, the mesh 16,320.

Next Step

In this lesson five stores, forty-seven files and five separate key decisions were counted. Every cost measured rested on the assumption that it was already known which store the question would go to. In the mesh arrangement there were five addresses and the query knew all of them; yet the name of a region unit’s product, the columns it carries, its partition key and the periods it covers cannot be found unless they are written down somewhere. The next lesson turns this into a single number: how many entries are looked at to find a data asset, with and without a catalog. In the same lesson the catalog’s own flaw is measured, how many entries now point to a layout that no longer exists, and this topic closes after six lessons.