Human Evaluation

The previous lesson measured two automated evaluators against the human label: the criteria-based approach gave 0.8250 agreement, the model-as-judge 0.8833, and between the judge’s own two runs the decision changed on sixteen records. Both sides of the comparison rested on one column, the record’s true class, that is, the human label.

This lesson measures that column. The human label is produced by reading a record and deciding, and that decision is volatile too: two raters looking at the same record can give different labels. The number measured is inter-rater agreement, with three questions: how much does it depend on the scale and the instruction, in which class does disagreement gather, and does raising agreement make the label more correct.

• EM18. The record set, the classes and the defense are built as in the previous two lessons and are fictional. An attack is a feature profile, not a text; the records’ text is harmless support requests unrelated to the class.
• EM19. All three raters are fictional; no real person, institution or country name appears. They are called d1, d2 and d3.
• EM20. All three see the same evidence: the record’s flag list. They differ in the weight given to each flag, their thresholds and their volatility, which depends on the seed and the record number.
• EM21. Three setups are compared: a binary scale, a three-valued scale (undecided in between) and an instruction with examples. The instruction’s text is not written, its effect is built: the examples pull the weights toward a shared anchor and halve the volatility.
• EM22. Agreement is the average, over the three pairs, of the share of records on which a pair gives the same label; unanimity is the share on which all three do. A record’s majority label is harmful if at least two of three votes are harmful; an undecided vote counts toward neither side.
• EM23. Agreement is read over two hundred and forty records, so its step is 1/240 = 0.0042. The second run uses seed 20260219; a difference smaller than this step cannot be measured.

Three Raters, Three Setups

The first block builds the record set and the measure as in the previous lesson, then defines the three raters and runs the three setups. The raters see only the flag list.

# FICTIONAL. Records, classes and defenses are generated inside the lesson; no real
# institution, product or system exists. An attack is not a TEXT, it is a FEATURE PROFILE.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Fictional, entirely harmless support texts. They have NO relation to the class.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]

RECORDS = []
for i in range(240):
    r = rng(SEED + 6151 * i)
    c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
    source = SOURCES[pick(r(), SOURCE_SHARE[c])]
    feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
    RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat,
                    "text": TEXT[int(r() * len(TEXT))],
                    "length": 40 + int(r() * 260)})


def harmful(k):
    return k["class"] != "clean"


def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


def measure(records, decision, truth=harmful):
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), truth(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}


# ---- FICTIONAL raters: no real person, institution or country exists.
# All three see the same flag list; their weights, thresholds and volatility differ.
RATERS = {
    "d1": {"threshold": 0.46, "volatility": 0.14,
           "weights": [0.46, 0.20, 0.44, 0.36, 0.34, 0.30, 0.08]},
    "d2": {"threshold": 0.58, "volatility": 0.22,
           "weights": [0.40, 0.10, 0.50, 0.44, 0.22, 0.14, 0.04]},
    "d3": {"threshold": 0.34, "volatility": 0.30,
           "weights": [0.30, 0.34, 0.28, 0.24, 0.40, 0.38, 0.22]},
}
SHARED_WEIGHTS = [0.42, 0.22, 0.44, 0.38, 0.34, 0.28, 0.10]   # the anchor of the instruction with examples
SHARED_THRESHOLD = 0.46


def evidence(k, d, with_examples=False, seed=SEED):
    """The harmfulness evidence the rater reads from the record's flag list."""
    a = RATERS[d]
    wt, threshold, vol = a["weights"], a["threshold"], a["volatility"]
    if with_examples:                    # the examples pull the weights toward the shared anchor
        wt = [0.35 * x + 0.65 * y for x, y in zip(wt, SHARED_WEIGHTS)]
        threshold, vol = 0.35 * threshold + 0.65 * SHARED_THRESHOLD, vol / 2
    p = sum(w for o, w in zip(FEATURES, wt) if k["features"][o])
    r = rng(seed ^ trace_id(d + "-" + str(k["no"])))
    return p + (r() - 0.5) * vol, threshold


def label(k, d, scale="binary", with_examples=False, seed=SEED, band=0.10):
    p, threshold = evidence(k, d, with_examples, seed)
    if scale == "three":                 # the third value: undecided
        if p >= threshold + band:
            return "harmful"
        return "clean" if p < threshold - band else "undecided"
    return "harmful" if p >= threshold else "clean"


def setup(records, scale="binary", with_examples=False, seed=SEED):
    return {d: [label(k, d, scale, with_examples, seed) for k in records]
            for d in RATERS}


def pairwise_agreement(t):
    """Inter-rater agreement: the average, over three pairs, of the share of records on which the pair matches."""
    names, n = sorted(t), len(next(iter(t.values())))
    pairs = [(names[0], names[1]), (names[0], names[2]), (names[1], names[2])]
    return sum(sum(1 for i in range(n) if t[a][i] == t[b][i]) / n for a, b in pairs) / 3


def disagreeing(t):
    """The positions of the records on which the three raters do not give the same label."""
    names, n = sorted(t), len(next(iter(t.values())))
    return [i for i in range(n) if len({t[a][i] for a in names}) > 1]


def majority(t, i):
    """The human label of the record: harmful if at least two of the three votes are harmful."""
    return sum(1 for a in t if t[a][i] == "harmful") >= 2


SETUPS = [("binary scale", "binary", False),
          ("three-valued scale", "three", False),
          ("instruction with examples", "binary", True)]
N = len(RECORDS)

print(f"{'setup':<26} {'agreement':>9} {'undecided':>9} {'harmful label':>15}"
      f" {'vs actual class':>15}")
for name, scale, with_examples in SETUPS:
    t = setup(RECORDS, scale, with_examples)
    undecided = sum(1 for a in t for e in t[a] if e == "undecided") / (3 * N)
    z = sum(1 for i in range(N) if majority(t, i))
    match = sum(1 for i, k in enumerate(RECORDS) if majority(t, i) == harmful(k)) / N
    print(f"{name:<26} {pairwise_agreement(t):>9.4f} {undecided:>9.4f} {z:>15} {match:>15.4f}")
print(f"number of records that are actually harmful {sum(1 for k in RECORDS if harmful(k))}"
      f" , agreement step 1/{N} = {1 / N:.4f}")

print("\ntwo runs , volatility of the raters (seed 20260218 / 20260219)")
print(f"{'setup':<26} {'run 1':>8} {'run 2':>9} {'diff':>8} {'step':>6}")
for name, scale, with_examples in SETUPS:
    a = pairwise_agreement(setup(RECORDS, scale, with_examples, SEED))
    b = pairwise_agreement(setup(RECORDS, scale, with_examples, SEED + 1))
    print(f"{name:<26} {a:>8.4f} {b:>9.4f} {b - a:>+8.4f} {abs(b - a) * N:>6.0f}")

setup                      agreement undecided   harmful label vs actual class
binary scale                  0.8417    0.0000              63          0.8417
three-valued scale            0.7583    0.1486              49          0.8000
instruction with examples     0.9306    0.0000              63          0.8417
number of records that are actually harmful 93 , agreement step 1/240 = 0.0042

two runs , volatility of the raters (seed 20260218 / 20260219)
setup                         run 1     run 2     diff   step
binary scale                 0.8417    0.8333  -0.0083      2
three-valued scale           0.7583    0.7708  +0.0125      3
instruction with examples    0.9306    0.9361  +0.0056      1

Agreement Rose, the Label Did Not Improve

On the binary scale agreement is 0.8417. On the three-valued scale it is 0.7583, twenty steps down; with the instruction with examples 0.9306, twenty-one steps up. The two-run table says these movements are real: the raters’ own volatility shifts agreement by at most three steps, 0.0125.

That the third value lowers agreement looks backwards at first glance. But the three-valued scale has two boundaries and each can be crossed separately: 0.1486 of the votes come out as undecided, and agreement is lost whenever one rater says undecided and another harmful. The cost shows in the label column too: the majority label calls forty-nine records harmful instead of sixty-three, against ninety-three that are actually harmful.

The last column carries the real finding of the lesson. The instruction with examples raised agreement by twenty-one steps, but the overlap of the majority label with the actual class stayed at 0.8417, as on the binary scale; the number of records it calls harmful is the same too, sixty-three. The instruction brought the raters closer to one another, not closer to the truth. Agreement measures consistency, and three people saying the same wrong thing also give high agreement.

Where Disagreement Gathers

The second block dumps disagreement in three cross-sections: per class, by flag count, and with unanimous records separated from the ones that split. The last table measures the same rule-based filter against two label columns.

TABLE = {name: setup(RECORDS, scale, with_examples) for name, scale, with_examples in SETUPS}

print(f"{'class':<20} {'records':>7} {'binary':>12} {'three-valued':>14} {'examples':>10}"
      f" {'step':>7}")
for s in CLASSES:
    idx = [i for i, k in enumerate(RECORDS) if k["class"] == s]
    p = [len(set(disagreeing(TABLE[name])) & set(idx)) / len(idx) for name, _, _ in SETUPS]
    print(f"{s:<20} {len(idx):>7} {p[0]:>12.4f} {p[1]:>14.4f} {p[2]:>10.4f}"
          f" {1 / len(idx):>7.4f}")

print("\ndisagreement by flag count , binary scale")
dis = set(disagreeing(TABLE["binary scale"]))
for name, n in (("no flags", 0), ("exactly one flag", 1)):
    idx = {i for i, k in enumerate(RECORDS)
           if sum(1 for o in FEATURES if k["features"][o]) == n}
    print(f"  {name:<16} {len(idx):>4} records , disagreeing share {len(dis & idx) / len(idx):.4f}")

print("\nlabel soundness: how much the majority label overlaps the actual class")
for name, _, _ in SETUPS:
    t, a = TABLE[name], set(disagreeing(TABLE[name]))
    unanimous = [i for i in range(N) if i not in a]
    unanimity_acc = sum(1 for i in unanimous if majority(t, i) == harmful(RECORDS[i])) / len(unanimous)
    split_acc = sum(1 for i in a if majority(t, i) == harmful(RECORDS[i])) / len(a)
    print(f"  {name:<26} unanimity {len(unanimous):>3} records / {unanimity_acc:.4f} ,"
          f" disagreeing {len(a):>3} records / {split_acc:.4f}")

print("\nsame defense , two different label columns")
print(f"{'label column':<22} {'tp':>4} {'fp':>4} {'fn':>4} {'precision':>10}"
      f" {'recall':>11} {'false block rate':>17}")
t = TABLE["binary scale"]
for name, truth in (("actual class", harmful),
                    ("majority label", lambda k: majority(t, k["no"] - 1))):
    o = measure(RECORDS, rule_filter, truth)
    print(f"{name:<22} {o['tp']:>4} {o['fp']:>4} {o['fn']:>4} {o['precision']:>10.4f}"
          f" {o['recall']:>11.4f} {o['false_block_rate']:>17.4f}")
print(f"share of the corrupted_input class that the majority label calls harmful "
      f"{sum(1 for i, k in enumerate(RECORDS) if k['class'] == 'corrupted_input' and majority(t, i)) / 19:.4f}")

class                records       binary   three-valued   examples    step
clean                    147       0.1020         0.1633     0.0476  0.0068
indirect_instruction      24       0.2500         0.4167     0.0000  0.0417
hidden_field_request      17       0.4118         0.5294     0.2353  0.0588
privilege_escalation      10       0.2000         0.5000     0.1000  0.1000
source_spoofing           23       0.5652         0.6522     0.3043  0.0435
corrupted_input           19       0.7368         0.8947     0.3158  0.0526

disagreement by flag count , binary scale
  no flags          118 records , disagreeing share 0.0000
  exactly one flag   53 records , disagreeing share 0.5094

label soundness: how much the majority label overlaps the actual class
  binary scale               unanimity 183 records / 0.9126 , disagreeing  57 records / 0.6140
  three-valued scale         unanimity 160 records / 0.9500 , disagreeing  80 records / 0.5000
  instruction with examples  unanimity 215 records / 0.8744 , disagreeing  25 records / 0.5600

same defense , two different label columns
label column             tp   fp   fn  precision      recall  false block rate
actual class             74   23   19     0.7629      0.7957            0.1565
majority label           63   34    0     0.6495      1.0000            0.1921
share of the corrupted_input class that the majority label calls harmful 0.2105

Disagreement is uneven. In clean records it is 0.1020, in the corrupted input class 0.7368, more than twelve times the class step. The most striking row of the previous lesson was in this class, the judge’s 1.0000 against the criteria-based 0.2632; that comparison rested on the part of the set where agreement is lowest.

The flag-count cross-section gives the reason. In the one hundred and eighteen records with no flags, disagreement is 0.0000: with no evidence, all three say clean. In the fifty-three records carrying exactly one flag it is 0.5094, nearly a coin toss. These fifty-three records are the borderline example set of the first lesson of this topic, where the precision of the rule-based filter came out at 0.4444 and its false block rate at 0.7143; those two numbers were computed against a column on which the three raters agree on only half.

The third cross-section gives the second cost of raising agreement. With the instruction with examples, unanimity rises from one hundred and eighty-three records to two hundred and fifteen, but its overlap with the actual class falls from 0.9126 to 0.8744: the instruction moved records that used to split into unanimity, some to the wrong side. Unanimity reads as a stamp of soundness, and the instruction lowered the stamp’s reliability. On the records that split, the majority label is already at 0.6140, and exactly 0.5000 on the three-valued scale.

The Same Defense, Two Different Label Columns

The last table ties the lesson to the rule of the course. The rule-based filter did not change; only the column it is measured against did. Against the actual class the recall is 0.7957. Against the majority label the same filter gives 1.0000, because no harmful record escapes: the filter already catches all sixty-three records the majority calls harmful.

This does not mean the filter is good. The last row gives the reason: the majority label calls only 0.2105 of the corrupted input class harmful. The raters weight that class’s flags low, and the rule-based filter does not read those two flags at all. Because their blind spots are the same, the label column cannot measure what the filter misses. The recall difference of 0.2043 is not a measurement error; it comes from the column. Precision goes the other way, from 0.7629 to 0.6495, because among the records the majority calls clean the false alarms rise from twenty-three to thirty-four.

Summary

• Three fictional raters on one flag list gave three agreements in three setups: 0.8417 on the binary scale, 0.7583 on the three-valued scale, 0.9306 with the instruction with examples. The step is 0.0042 and the two-run band at most three steps, so both movements are real.
• The third value lowered agreement and narrowed the label column: 0.1486 of the votes came out undecided, and the majority label called forty-nine records harmful instead of sixty-three.
• The instruction with examples raised agreement by twenty-one steps, but the overlap of the majority label with the actual class stayed at 0.8417 and the accuracy of unanimity fell from 0.9126 to 0.8744. Agreement measures consistency, not accuracy.
• Disagreement gathers in the harmful classes and in thin evidence: 0.7368 in corrupted input, 0.5094 in the fifty-three records with exactly one flag, 0.0000 in the one hundred and eighteen without flags.
• Measured against the majority label, the same rule-based filter reaches 1.0000 recall instead of 0.7957, because the label column shares its blind spot: it calls only 0.2105 of corrupted input harmful.

Next Step

For three lessons the system being measured never changed; the set, the defense, the judge and the raters were fixed, and only the way of measuring moved. In production the opposite happens: the set stays and the system changes. The instruction is rewritten, a threshold nudged, a rule added, the model behind it drifts without notice. The next lesson sets a fixed set against a changing system and asks one question: which of these changes really moved the number, and which moved it by less than the run band, which makes it unmeasurable.