Content Moderation

All the numbers of the previous lesson were read from a single decision: block or let through. The rule-based filter blocked 74 of the 93 harmful records, blocked 23 clean records, and its average recall came out 0.7957. The last column of the same table, however, showed that 14 of the 19 missed records came from a single class. The average number did not carry that concentration.

When a defense does not stop at blocking but also tells the record which kind of harmful it is, two things change at once. First, the filter can gain a rule that looks at the class it was missing. Second, and more important, the measure stops being a single pair of numbers: each class gets its own precision and recall. This lesson builds that distinction and counts what the average hides.

• DF8. The record set, the classes, the features and the content moderation step are fictional; no real model endpoint is called. The seed is 20260218, the second run uses 20260219.
• DF9. This is not an attack lesson. All of the code that runs sits on the defense side: classification, filter and count. No working attack, payload or evasion routine is written.
• DF10. An attack is not a text but a feature profile. The text field of the records equals one of six ordinary support requests and has no relation to the class; the moderation step does not read that text either, it reads only the structural features.
• DF11. The baseline is the undefended setup: no filter or moderation step, harmful records blocked 0, false positives 0, class labels produced 0.
• DF12. The content moderation step returns a single class name; the priority order is a fiction, and if a record has more than one feature only the class at the front is written.
• DF13. The general routine of model evaluation and subgroup analysis was measured in the Supervised Learning and Model Evaluation, Interpretability and Ethics courses and is not repeated here; this lesson adds only the reading per defense class.
• DF14. The step of a per-class rate is the size of that class and is larger than the general step of the set: 0.1000 in the smallest class, 0.0417 in the largest harmful class. Over all harmful records the step of recall is 0.0108, the step of the false block rate is 0.0068.

A Defense That Produces a Class Name

The first block sets up the course’s shared definition: the record set, the classes, the rule-based filter, the model-based filter and the measure. Which class the six support sentences fall into is random; the class information lives only in the PROFILE dictionary.

# FICTIONAL. The record set, classes, features and defense layers are defined inside
# the lesson; no real model endpoint is called.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE
CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
# Feature probability per class (fictional). Clean records carry small shares too:
# in the real world, well-meant text can carry these features as well.
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Fictional, entirely harmless support texts. They have NO relation to the class;
# the class information lives only in the feature profile.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]

RECORDS = []
for i in range(240):
    r = rng(SEED + 6151 * i)
    c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
    source = SOURCES[pick(r(), SOURCE_SHARE[c])]
    feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
    RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat,
                    "text": TEXT[int(r() * len(TEXT))],
                    "length": 40 + int(r() * 260)})


def harmful(k):
    return k["class"] != "clean"


# ---- DEFENSE 1: rule filter (structural, explainable)
def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


# ---- DEFENSE 2: model-based filter (score + threshold)
WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44,
          "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38,
          "unusual_stamp_distribution": 0.30, "excessive_length": 0.12}
SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12}


def score(k):
    p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]]
    r = rng(SEED ^ trace_id(k["no"]))
    return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6)


def model_filter(k, threshold=0.40):
    return score(k) >= threshold


# ---- MEASURE: what was blocked and what was blocked by mistake are written together
def measure(records, decision):
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}

Precision and Recall per Class

The content moderation step differs from the rule-based filter in two points: it returns a class name, and it reads one more feature, the stamp distribution. The priority order puts each class’s own distinguishing feature at the front. The second block first prints the binary reading — which number comes out if saying “not clean” counts as blocking — and then opens the same decision class by class.

def content_moderation(k):
    """Defense: assigns a class name to the record. No attack text is read;
    only structural features are evaluated in priority order."""
    o = k["features"]
    if o["hidden_field_name"]:
        return "hidden_field_request"
    if o["side_effect_tool_name"]:
        return "privilege_escalation"
    if o["source_stamp_mismatch"]:
        return "source_spoofing"
    if o["unusual_stamp_distribution"]:
        return "corrupted_input"
    if o["instruction_mode"] or o["role_change_request"]:
        return "indirect_instruction"
    return "clean"


binary = measure(RECORDS, lambda k: content_moderation(k) != "clean")
print("binary reading, if saying 'not clean' counts as blocking:")
print(f"  tp {binary['tp']}  fp {binary['fp']}  fn {binary['fn']}"
      f"  precision {binary['precision']}  recall {binary['recall']}"
      f"  false block rate {binary['false_block_rate']}")
print()
print(f"{'class':<20}{'n':>4}{'assigned':>9}{'correct':>8}{'wrong class':>13}"
      f"{'clean':>6}{'precision':>10}{'recall':>12}{'step':>8}")
for s in CLASSES[1:]:
    g = [k for k in RECORDS if k["class"] == s]
    a = [k for k in RECORDS if content_moderation(k) == s]
    d = sum(1 for k in a if k["class"] == s)
    ws = sum(1 for k in a if harmful(k) and k["class"] != s)
    cl = sum(1 for k in a if not harmful(k))
    print(f"{s:<20}{len(g):>4}{len(a):>9}{d:>8}{ws:>13}{cl:>6}"
          f"{d / len(a):>10.4f}{d / len(g):>12.4f}{1 / len(g):>8.4f}")
passed = [k for k in RECORDS if content_moderation(k) == "clean"]
pc = sum(1 for k in passed if not harmful(k))
print(f"{'clean':<20}{147:>4}{len(passed):>9}{pc:>8}{len(passed) - pc:>13}{'-':>6}"
      f"{pc / len(passed):>10.4f}{pc / 147:>12.4f}{1 / 147:>8.4f}")
print()
print("classes of the harmful records passed as clean:",
      {s: sum(1 for k in passed if k["class"] == s) for s in CLASSES[1:]
       if sum(1 for k in passed if k["class"] == s)})

binary reading, if saying 'not clean' counts as blocking:
  tp 87  fp 32  fn 6  precision 0.7311  recall 0.9355  false block rate 0.2177

class                  n assigned correct  wrong class clean precision      recall    step
indirect_instruction  24       21       8            4     9    0.3810      0.3333  0.0417
hidden_field_request  17       32      14            9     9    0.4375      0.8235  0.0588
privilege_escalation  10       24       7           14     3    0.2917      0.7000  0.1000
source_spoofing       23       22      14            4     4    0.6364      0.6087  0.0435
corrupted_input       19       20      12            1     7    0.6000      0.6316  0.0526
clean                147      121     115            6     -    0.9504      0.7823  0.0068

classes of the harmful records passed as clean: {'hidden_field_request': 1, 'privilege_escalation': 1, 'source_spoofing': 1, 'corrupted_input': 3}

What the Average Hides

The first rows show the best recall so far: 0.9355. Where the undefended baseline blocked 0 harmful records, the moderation step flags 87 of the 93 harmful records and misses only 6. This is where a defense would be reported with a single line, and this line is wrong — more precisely, what it says is true but it does not answer the question asked. What is expected of a content moderation step is not that the record is blocked but that it is told what it is; blocking was already the job of the rule-based filter.

The class table opens the same decision by name and no single rate remains. The recall of the class label spreads between 0.3333 and 0.8235, and its precision between 0.2917 and 0.6364. Two classes stand out.

The recall of the hidden_field_request class is the highest at 0.8235, but its precision is 0.4375: the moderation step puts this label on 32 records and only 14 of them really belong to that class. Because the label stands at the front of the priority order, it pulls in every record that carries a hidden field name — 9 of them from another harmful class, 9 of them entirely clean.

privilege_escalation is sharper: the label is put on 24 records, 7 are correct, 14 are from another harmful class. This row carries a warning — the decision to block is right for these 14 records, because all of them really are harmful; only the decision given to the name is wrong. The class precision of a defense can be 0.2917 and the same defense can look good in the binary reading. Blocking and naming are two separate jobs and are measured with two separate numbers.

The last row writes the cost: the moderation step counts 115 of the 147 clean records as clean and puts a harmful class name on the other 32. The false block rate is 0.2177, nine resolution steps higher than the 0.1565 of the rule-based filter. Half of the 6 missed harmful records again belong to the corrupted input class.

Recall per Class, Three Defenses

The same class split applies to the two filters of the previous lesson. The third block prints four columns: the rule-based filter, the model-based filter, the second run of the same model filter, and the binary decision of the moderation step. What is measured here is not the class label but whether the record is flagged.

def score_second(k):
    """Same model, second run: only the noise seed changes."""
    p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]]
    r = rng(20260219 ^ trace_id(k["no"]))
    return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6)


print(f"{'class':<20}{'n':>4}{'rule':>10}{'model 0.50':>12}"
      f"{'run 2':>10}{'moderation':>12}{'step':>8}")
harmful_records = [k for k in RECORDS if harmful(k)]
for s in CLASSES[1:]:
    g = [k for k in RECORDS if k["class"] == s]
    rates = [sum(1 for k in g if f(k)) / len(g)
             for f in (rule_filter, lambda k: model_filter(k, 0.50),
                       lambda k: score_second(k) >= 0.50,
                       lambda k: content_moderation(k) != "clean")]
    print(f"{s:<20}{len(g):>4}{rates[0]:>10.4f}{rates[1]:>12.4f}"
          f"{rates[2]:>10.4f}{rates[3]:>12.4f}{1 / len(g):>8.4f}")
rates = [sum(1 for k in harmful_records if f(k)) / len(harmful_records)
         for f in (rule_filter, lambda k: model_filter(k, 0.50),
                   lambda k: score_second(k) >= 0.50,
                   lambda k: content_moderation(k) != "clean")]
print(f"{'average':<20}{len(harmful_records):>4}{rates[0]:>10.4f}{rates[1]:>12.4f}"
      f"{rates[2]:>10.4f}{rates[3]:>12.4f}{1 / len(harmful_records):>8.4f}")
print()
print("false block rate:",
      {"rule": measure(RECORDS, rule_filter)["false_block_rate"],
       "model 0.50": measure(RECORDS, lambda k: model_filter(k, 0.50))["false_block_rate"],
       "moderation": binary["false_block_rate"]})

class                  n      rule  model 0.50     run 2  moderation    step
indirect_instruction  24    1.0000      0.9583    0.9583      1.0000  0.0417
hidden_field_request  17    0.9412      0.7059    0.7059      0.9412  0.0588
privilege_escalation  10    0.9000      0.8000    0.8000      0.9000  0.1000
source_spoofing       23    0.8696      0.8261    0.8261      0.9565  0.0435
corrupted_input       19    0.2632      0.2105    0.3684      0.8421  0.0526
average               93    0.7957      0.7097    0.7419      0.9355  0.0108

false block rate: {'rule': 0.1565, 'model 0.50': 0.0544, 'moderation': 0.2177}

The Class Below the Baseline

The average of 0.7957 of the rule-based filter lies between 0.8696 and 1.0000 in four classes and is 0.2632 in the fifth. The average is four good numbers carrying one bad number; this row does not tell the reader which class is unprotected. The moderation step raises that class to 0.8421, because it reads the stamp distribution that the rule set does not look at — but the cost is in the same row: the false block rate rises from 0.1565 to 0.2177, and the blocked clean records from 23 to 32.

The corrupted input row has to be read together with the second run. The recall of the model-based filter in this class moves between 0.2105 and 0.3684 across the two runs; the difference is 0.1579, that is, three steps of the resolution of that class. In the other four classes the difference is exactly zero. The weakest class is also the most unstable class, and the number reported from a single run for that class is not a number but a band.

Now the harshest application of the course’s rule. In the indirect_instruction class the moderation step puts this label on 21 records; 8 are correct, 4 are from another harmful class, and 9 are entirely clean. If the class label is tied to an action — a warning specific to that class, a rewrite, a human review — the action fires correctly 8 times and wrongly 13 times in this set, and in 9 of the wrong firings there is a real subscriber on the other side. On the undefended baseline this action never fires: neither rightly nor wrongly. If the cost of a wrong firing is not smaller than the gain of a right firing, in this class the defense is below the baseline. The difference between 8 correct labels and 9 labels put on clean records is a single record and could change sign in another draw; what is readable is that the ratio is close to one.

The fact that the same class reads 1.0000 in the binary column completes this table. The moderation step blocks all of the indirect instruction class and puts the right name on a third of it. The choice of which to write, if a single rate were reported, depends on what the defense was built for — and that is why both are written.

Summary

• The content moderation step that produces a class name blocks 87 harmful records and blocks 32 clean records in the binary reading: precision 0.7311, recall 0.9355, false block rate 0.2177.
• The class label of the same moderation step is much weaker: class precision spreads between 0.2917 and 0.6364, class recall between 0.3333 and 0.8235. Blocking and naming are two separate jobs.
• The average of 0.7957 of the rule-based filter is above 0.8696 in four classes and 0.2632 in the corrupted input class; the moderation step raises that class to 0.8421, and the false block rate rises from 0.1565 to 0.2177.
• The corrupted input class moves between 0.2105 and 0.3684 across two runs, that is, three steps of the class resolution; in the other four classes the difference is zero.
• In the indirect instruction class the label is put on 21 records, 8 are correct and 9 land on entirely clean records; if the class label is tied to an action, the action fires wrongly far more often than rightly and the defense falls below the baseline in that class.

Next Step

In two lessons the defense decided on the whole of the record: let through, block, or put a class name. In all three the record was an indivisible whole. Yet the part of a record that should go to the model can be separated from the part that should not: the subscriber identity, the contact information, the address and the access string can be removed and the rest sent. This is a defense that blocks no record, so it has no false positive — its measure has to be read from somewhere else. The next lesson counts how many records masking cleans and how many tasks lose the field they require.