Key Performance Indicators

The previous lesson took the twelve metrics as given and measured only their place on the page. One of the metrics, the missing reading rate, fed two of the three decisions. From the moment a number starts producing decisions it gains a new property: the person who reports it can move the number without touching the thing it measures at all.

The measure of this lesson is gameability: in how many ways and by how many points a metric can be moved without moving the thing it measures at all. Metric design itself is not built here. Error budgets, service level indicators and the choice of operational metrics were built in the Observability and Reliability, System Administration, and DevOps Culture and Fundamentals courses; here a single metric is taken and only how loose its definition is gets counted.

• RP7. The dataset is inherited from the Data Collection and Preparation course and is fictional: a reading table of 3,199 rows, 1,329 subscribers, three periods, five regions. The seed is written into the code.
• RP8. The metric is also fictional: the reading coverage rate, the ratio of the subscribers with a nonzero reading in period 3 to the subscribers registered at the start of the period.
• RP9. Moving the number never touches the measurement network: no reading is added, deleted or changed. Only the definition or the presentation of the metric changes.
• RP10. Each of the six ways is defensible on its own; none of them fabricates data.
• RP11. The gaming threshold is 0.05 points. Its rationale is that a single subscriber in the denominator is worth 0.075 points; the threshold filters out every movement smaller than one subscriber.
• RP12. The real drop in the measurement network is also fictional and is generated with a separate seed.

Metric

# metric.py — MODEL: the fictional dataset from K03 is generated with the same seed; a single
# key performance indicator (KPI) is built on top of it.
import math

SEED, RAW, M32 = 20260218, 1400, 0xFFFFFFFF
REGION = [("north", 0.28, 21), ("south", 0.22, 17), ("east", 0.18, 26),
          ("west", 0.14, 14), ("central", 0.18, 23)]


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


SUBSCRIBERS, DISCARDED = [], []
for i in range(RAW):
    r = rng(SEED + i)
    b = REGION[pick(r(), [x[1] for x in REGION])]
    pick(r(), [0.06, 0.24, 0.30, 0.24, 0.11, 0.05])
    pick(r(), [0.08, 0.14, 0.27, 0.34, 0.17])
    (DISCARDED if r() < 0.046 else SUBSCRIBERS).append(
        {"subscriber_id": 10001 + i, "region": b[0], "base": b[2]})
VALID = {k["subscriber_id"] for k in SUBSCRIBERS}
READINGS = {}
for k in SUBSCRIBERS + DISCARDED:
    r = rng(SEED + 7000 + k["subscriber_id"])
    for d in range(pick(r(), [0.05, 0.12, 0.21, 0.62])):
        t = 0.0 if r() < 0.038 else math.floor(
            k["base"] * math.exp((r() + r() + r() - 1.5) * 0.62) * (1 - d * 0.05) * 100
            + 0.5) / 100
        if k["subscriber_id"] in VALID:
            READINGS.setdefault(k["subscriber_id"], {})[d + 1] = t
        r()

TARGET = 65.0


def coverage(period=3, zero_counts=False, narrow_denominator=False, wide=False, data=None):
    data = data if data is not None else READINGS
    denominator = [k for k in SUBSCRIBERS if not narrow_denominator or data.get(k["subscriber_id"])]
    numerator = 0
    for k in denominator:
        o = data.get(k["subscriber_id"], {})
        windows = (2, 3) if wide else (period,)
        if any(d in o and (zero_counts or o[d] > 0.0) for d in windows):
            numerator += 1
    return numerator / len(denominator) * 100, numerator, len(denominator)


baseline, numerator0, denominator0 = coverage()
print(f"fictional dataset (seed {SEED}): {len(SUBSCRIBERS)} subscribers, "
      f"{sum(len(v) for v in READINGS.values())} readings")
print(f"metric — reading coverage rate (period 3): {numerator0} / {denominator0} = {baseline:.3f}")

fictional dataset (seed 20260218): 1329 subscribers, 3199 readings
metric — reading coverage rate (period 3): 810 / 1329 = 60.948

What the metric wants to measure can be said in one sentence: how many of the subscribers the measurement network was able to reach within the period. When this sentence is turned into a number, at least four decisions are made and none of them is written in the sentence. Who is in the denominator, is zero a reading, when does the period close, with how many decimals is the number written.

Six Ways

Each way loosens one of these decisions. None of them fabricates data, none touches the measurement network, and each can be defended in a meeting with a single sentence.

# Six ways. None of them touches the measurement network: no reading is added, deleted or changed.
# Each way loosens one item of the definition or of the presentation, and each is defensible.
THRESHOLD_POINTS = 0.05
TIGHT = {"denominator", "zero", "window", "presentation", "aggregation", "target"}


def region_rate(data):
    rates = []
    for b, _, _ in REGION:
        group = [k for k in SUBSCRIBERS if k["region"] == b]
        rates.append(sum(1 for k in group if data.get(k["subscriber_id"], {}).get(3, 0.0) > 0.0)
                     / len(group) * 100)
    return sum(rates) / len(rates)


def ways(data, fixed=frozenset()):
    t = coverage(data=data)[0]
    w = [("denominator narrowing", "denominator", coverage(data=data, narrow_denominator=True)[0], "%"),
         ("counting zero as a reading", "zero",
          coverage(data=data, zero_counts=True)[0], "%"),
         ("window widening (period 2 or 3)", "window",
          coverage(data=data, wide=True)[0], "%"),
         ("rounding to a whole number", "presentation", float(round(t)), "%"),
         ("unweighted region average", "aggregation", region_rate(data), "%"),
         ("presenting by target normalization", "target", t / TARGET * 100, "% of target")]
    return [(name, item, t if item in fixed else v, unit) for name, item, v, unit in w]


print(f"baseline metric {baseline:.3f}, target {TARGET:.0f}. One subscriber in the denominator is "
      f"worth {100 / denominator0:.3f} points, so the gaming threshold is {THRESHOLD_POINTS} points.\n")
print(f"{'way':<38}{'reported value':>15}{'movement':>9}  unit")
for name, item, v, unit in ways(READINGS):
    print(f"{name:<38}{v:>15.3f}{v - baseline:>9.3f}  {unit}")
one = coverage(zero_counts=True, narrow_denominator=True, wide=True)
print(f"{'first three ways together':<38}{one[0]:>15.3f}{one[0] - baseline:>9.3f}  "
      f"%   ({one[1]} / {one[2]})")

baseline metric 60.948, target 65. One subscriber in the denominator is worth 0.075 points, so the gaming threshold is 0.05 points.

way                                    reported value movement  unit
denominator narrowing                          64.286    3.338  %
counting zero as a reading                     63.055    2.107  %
window widening (period 2 or 3)                81.791   20.843  %
rounding to a whole number                     61.000    0.052  %
unweighted region average                      60.985    0.037  %
presenting by target normalization             93.766   32.818  % of target
first three ways together                      87.381   26.433  %   (1101 / 1260)

Without a single reading changing in the measurement network, the same metric can be reported anywhere between 60.948% and 93.766%. The defensibility of the ways is real too. Removing the 69 subscribers with no readings at all from the denominator looks reasonable, because the meters of these subscribers could never be reached. A zero reading is a reading: the meter was read, no consumption came out. Saying that a subscriber with a reading in period 2 or 3 was “reached” can also be defended.

The two largest movements come from the two ways that look most innocent. Window widening carries 20.843 points, target normalization 32.818 points. The second changes the unit of the number: the 93.766 that is written is not a coverage rate but the percentage of the target that was realized. If the unit is not written under the same line, the reader reads it back as the coverage itself.

The ways are not mutually exclusive. When the first three are applied together, the number written into the report, staying in the same unit, becomes 87.381: 1,101 subscribers out of 1,260. Three defensible decisions made by three different people in three different meetings carry the metric 26.433 points up without anyone fabricating anything.

The hardest to notice of the ways is the first, and the reason lies in the concept of the previous lesson. If the report line says only “64.286%”, the reader has nothing with which to tell apart a ratio with a numerator of 810 and a denominator of 1,260 from a ratio with a numerator of 810 and a denominator of 1,329. Whether a ratio can be read back depends not on an anchor but on its numerator and denominator standing next to it; a ratio whose two parts are not written hides its own definition.

Redefinition

The redefinition is six items and each item binds one way. Then the same six ways are tried again.

# The redefinition is six items and each item binds one way: the denominator is the subscriber
# registered at the start of the period, zero is not counted as a reading, the window closes at
# period 3, the rate is written with three decimals, aggregation is subscriber-weighted, and
# target normalization is forbidden.
def try_ways(fixed, data=READINGS):
    t = coverage(data=data)[0]
    return [(name, abs(v - t) > THRESHOLD_POINTS, v) for name, item, v, unit in ways(data, fixed)]


loose, tight = try_ways(frozenset()), try_ways(TIGHT)
print(f"{'way':<38}{'loose definition':>18}{'tight definition':>18}")
for (name, la, lv), (_, ta, tv) in zip(loose, tight):
    print(f"{name:<38}{('moved ' + format(lv, '.3f')) if la else 'did not move':>18}"
          f"{('moved ' + format(tv, '.3f')) if ta else 'did not move':>18}")
unclosed = [a for a in loose if not a[1]]
print(f"\nways exceeding the threshold: {sum(1 for a in loose if a[1])} under the loose definition, "
      f"{sum(1 for a in tight if a[1])} under the tight definition ({len(loose)} ways tried)")
for name, _, v in unclosed:
    print(f"not closed, did not work on this dataset: {name} — {abs(v - baseline):.3f} points, "
          f"smaller than the {100 / denominator0:.3f} points one subscriber is worth")

way                                     loose definition  tight definition
denominator narrowing                       moved 64.286      did not move
counting zero as a reading                  moved 63.055      did not move
window widening (period 2 or 3)             moved 81.791      did not move
rounding to a whole number                  moved 61.000      did not move
unweighted region average                   did not move      did not move
presenting by target normalization          moved 93.766      did not move

ways exceeding the threshold: 5 under the loose definition, 0 under the tight definition (6 ways tried)
not closed, did not work on this dataset: unweighted region average — 0.037 points, smaller than the 0.075 points one subscriber is worth

The number of ways exceeding the threshold falls from 5 to 0. The real warning in the table is in the fifth row. The unweighted region average did not exceed the threshold even under the loose definition, because in this dataset the coverage rates of the five regions are close enough to each other. Counting this way as “closed” would mean accepting that what keeps the number in place is not the definition but the size distribution of the regions. If one of the regions shrank or its coverage diverged, the same way would exceed the threshold, and if the definition has no counterpart for it, nothing would stop it.

This also determines how the gameability number is to be read: a way that was tried and did not move is not a closed way. What is counted is how many ways the definition binds, not how many ways worked in this run.

Writing all six items into a definition lengthens the metric, and this is a cost. When the one-line expression “reading coverage rate” turns into a six-item definition, the reader of the report can no longer read the metric back at a glance. The way the cost is paid is to write the definition not into the report but into the place the report rests on: the report line carries only the numerator, the denominator and the definition identifier.

Real Drop

The tight definition may have a cost: if making the metric ungameable also makes it insensitive to the thing it measures, there is no gain. To measure this, a real loss is produced in the measurement network.

# Real drop: a portion of the period 3 readings is really lost in the measurement network.
# This is not moving the number; the thing that is measured itself changes.
DROP_DATA, lost = {}, 0
for k in SUBSCRIBERS:
    o = dict(READINGS.get(k["subscriber_id"], {}))
    if 3 in o and rng(SEED + 91000 + k["subscriber_id"])() < 0.14:
        del o[3]
        lost += 1
    DROP_DATA[k["subscriber_id"]] = o
new = coverage(data=DROP_DATA)[0]
print(f"period 3 readings lost in the measurement network: {lost}")
print(f"number given by the tight definition: {baseline:.3f} -> {new:.3f}  ({new - baseline:+.3f} points)\n")
print(f"{'loose definition ways, same drop':<38}{'reported value':>15}"
      f"{'before the drop':>17}  unit")
masking = 0
for name, item, v, unit in ways(DROP_DATA):
    masking += 1 if v >= baseline else 0
    print(f"{name:<38}{v:>15.3f}{('above' if v >= baseline else 'below'):>17}  {unit}")
print(f"\n{masking} of the six ways keep the number above its pre-drop value in the same real drop;\n"
      f"one of them does it by changing the unit")

period 3 readings lost in the measurement network: 106
number given by the tight definition: 60.948 -> 53.273  (-7.675 points)

loose definition ways, same drop       reported value  before the drop  unit
denominator narrowing                          56.190            below  %
counting zero as a reading                     55.079            below  %
window widening (period 2 or 3)                81.490            above  %
rounding to a whole number                     53.000            below  %
unweighted region average                      53.470            below  %
presenting by target normalization             81.959            above  % of target

2 of the six ways keep the number above its pre-drop value in the same real drop;
one of them does it by changing the unit

The tight definition passes the loss through as it is: when 106 readings are lost, the number drops from 60.948 to 53.273, that is by 7.675 points. Being ungameable does not mean being insensitive.

Two of the six ways of the loose definition keep the number above its pre-drop value in the same real drop. Window widening writes 81.490, because most owners of the readings lost in period 3 still have their period 2 reading. Target normalization writes 81.959. Both reports come from the same dataset, both are computed correctly, and one says coverage fell while the other says it rose.

The gameability of the metric finds its full meaning here. Moving the number does not just make a good number look better; it makes bad news invisible. A real loss of seven points can pass without being reported at all in a metric with a loose definition.

There is one more distinction to note. Window widening covering up the drop is not a sign of bad faith: a subscriber whose period 2 reading stands has really been reached, and the person who chooses that way does not have to be doing it knowingly. When the definition is not written, the way can be chosen without anyone preferring it; because it is not written with which window the metric was computed in the previous period, the use of a different window this period goes unnoticed. Gameability is not a personality trait but a property of the definition.

Summary

• Gameability is a countable quantity: in how many ways and by how many points a metric can be moved without touching the thing it measures at all.
• When the coverage rate, which stands at 60.948%, is tried in six ways, five exceed the threshold of 0.05 points and the number written into the report can stand anywhere between 60.948 and 93.766.
• The two largest movements are window widening (20.843 points) and target normalization (32.818 points); the second changes the unit of the number, and if the unit is not written the reader reads it back as coverage.
• The six-item redefinition lowers the ways exceeding the threshold from 5 to 0; a way that does not exceed the threshold is not counted as closed, because its movement of 0.037 points comes not from the definition but from the size distribution of the regions.
• The ways are not mutually exclusive: when the first three are applied together the number, staying in the same unit, becomes 87.381 (1,101 subscribers out of 1,260) and is carried 26.433 points up.
• When 106 readings are really lost in the measurement network, the tight definition drops by 7.675 points; two of the six ways of the loose definition keep the number above its pre-drop value in the same drop.

Next Step

This lesson fixed the definition of the metric, but fixing the definition is not fixing the number. The tight definition says “the window closes at period 3”; when a reading that belongs to the closed window enters the system after the report was produced, the same definition applied to the same dataset gives a different number. The report stands on a source, and that source is connected to the report either by a live query, or by a periodic extract, or by a pre-aggregated model. The next lesson runs the three connection schemes with the same metric, counts the difference among them and shows which questions the report data model closes forever.