Data Exfiltration

The previous two lessons tied the measure of the defense to a single question: was the record blocked or not. Some harm has nothing to do with blocking. When a record passes, what is lost is sometimes not an action but a piece of information — the system instruction given to the model, a subscriber’s identity, an address, or an access string. This loss cannot be undone: a blocked operation can be retried, a disclosed field cannot be taken back.

In this lesson data exfiltration is only counted. No exfiltration procedure, query form, or disclosure example is written; what is written is four numbers: how many records carry a hidden field name, from how many of them masking removes the field, how many the filter catches, and how many records are left that none of them catches. That last number is the real finding of the lesson.

• AS17. The record set, classes, secret field classes, and defense layers are fictional; no real personal data or access string is used. Seed 20260218.
• AS18. This is not an attack lesson. All code that runs is on the defense side: counting, masking, filter. No working exfiltration procedure or query form is written.
• AS19. An attack is not a text but an attribute profile; the text of the records is entirely harmless support requests and has no relation to the class.
• AS20. Personal data and the masking procedure were measured in the Application Security and DevSecOps and the Machine Learning and Data Science curricula; they are not repeated. Here the count covers only the data going to the model.
• AS21. Masking is not a block: it does not stop the record, it only removes the field. Measuring it as a decision is an approximation, which the lesson writes separately.
• AS22. Which field class a record requests is derived from the record number; this is not a detection but a distribution built to split the count.
• AS23. The exfiltration surface is the union of two conditions: records that carry the hidden field name flag and records in the hidden field request class. They overlap; neither contains the other.
• AS24. The smallest step of recall is 1/93 = 0.0108, and the step of the false block rate is 1/147 = 0.0068. A one-record residue sits exactly on this band.

Exfiltration Is a Loss, Not an Action

Exfiltration has two targets, both in the system’s own context. The first is system instruction disclosure: the instruction given to the model carried into the output. The harm is not that the text is secret but that the rules the defense rests on become visible. The second is sensitive data disclosure: a subscriber record, address, or access string that entered the context showing up in the response. The record set collects both in one structural flag, hidden_field_name, and splits it into four field classes: identity, contact, location, access.

# FICTIONAL. The record set, classes, secret field classes and defense layers are defined
# inside the lesson; no real personal data is used. No attack TEXT is written.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- FICTIONAL records: no attack TEXT is written, an attack is a FEATURE PROFILE
CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
# Feature probability per class (fictional). Clean records carry small shares too:
# in the real world, well-meant text can carry these features as well.
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Fictional, entirely harmless support texts. They have NO relation to the class;
# the class information lives only in the feature profile.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]

RECORDS = []
for i in range(240):
    r = rng(SEED + 6151 * i)
    c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
    source = SOURCES[pick(r(), SOURCE_SHARE[c])]
    feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
    RECORDS.append({"no": i + 1, "class": c, "source": source, "features": feat,
                    "text": TEXT[int(r() * len(TEXT))],
                    "length": 40 + int(r() * 260)})


def harmful(k):
    return k["class"] != "clean"


# ---- DEFENSE 1: rule filter (structural, explainable)
def rule_filter(k, rules=("instruction_mode", "hidden_field_name", "side_effect_tool_name",
                          "source_stamp_mismatch")):
    return any(k["features"][o] for o in rules)


# ---- DEFENSE 2: model-based filter (score + threshold)
WEIGHT = {"instruction_mode": 0.42, "role_change_request": 0.26, "hidden_field_name": 0.44,
          "side_effect_tool_name": 0.40, "source_stamp_mismatch": 0.38,
          "unusual_stamp_distribution": 0.30, "excessive_length": 0.12}
SOURCE_BONUS = {"user": 0.00, "retrieved_document": 0.18, "tool_output": 0.12}


def score(k):
    p = sum(WEIGHT[o] for o in FEATURES if k["features"][o]) + SOURCE_BONUS[k["source"]]
    r = rng(SEED ^ trace_id(k["no"]))
    return min(max(p + (r() - 0.5) * 0.20, 0.0), 1.6)


def model_filter(k, threshold=0.40):
    return score(k) >= threshold


# ---- DEFENSE 4: personal data masking (count only)
SECRET_FIELD = {"subscriber_id": "identity", "contact_info": "contact", "address": "location",
                "access_string": "access"}


def redact(k):
    """Fictional: in records that carry a hidden field name, the field is removed."""
    return {"redacted": k["features"]["hidden_field_name"]}


# ---- MEASURE: what was blocked and what was blocked by mistake are written together
def measure(records, decision):
    tp = fp = tn = fn = 0
    for k in records:
        flagged, actual = decision(k), harmful(k)
        if flagged and actual:
            tp += 1
        elif flagged and not actual:
            fp += 1
        elif not flagged and actual:
            fn += 1
        else:
            tn += 1
    precision = tp / (tp + fp) if tp + fp else 0.0
    recall = tp / (tp + fn) if tp + fn else 0.0
    return {"tp": tp, "fp": fp, "fn": fn, "tn": tn,
            "precision": round(precision, 4), "recall": round(recall, 4),
            "false_block_rate": round(fp / (fp + tn), 4) if fp + tn else 0.0}

How Many Records Request It

The second block counts the surface: how many records carry a hidden field name, which field class each requests, and how many of them are actually harmful.

FIELDS = list(SECRET_FIELD)


def field_class(k):
    """Which field class a record with a hidden field name requests (fictional, bound to the record)."""
    return FIELDS[trace_id(k["no"]) % len(FIELDS)]


carriers = [k for k in RECORDS if k["features"]["hidden_field_name"]]
print("records with a hidden field name:", len(carriers),
      " harmful:", sum(1 for k in carriers if harmful(k)),
      " clean:", sum(1 for k in carriers if not harmful(k)))
print("\nfield class      requests  harmful  clean  category")
for a in FIELDS:
    g = [k for k in carriers if field_class(k) == a]
    print(f"{a:<16} {len(g):>8} {sum(1 for k in g if harmful(k)):>8}"
          f" {sum(1 for k in g if not harmful(k)):>6}  {SECRET_FIELD[a]}")

print("\nclass distribution of the flagged records")
for c in CLASSES:
    n = sum(1 for k in carriers if k["class"] == c)
    if n:
        print(f"  {c:<20} {n:>3}")

print("\nif masking is measured on its own as a decision")
m = measure(RECORDS, lambda k: redact(k)["redacted"])
print(f"  removed fields {m['tp'] + m['fp']}  of which harmful {m['tp']}"
      f"  clean {m['fp']}  harmful untouched {m['fn']}")
print(f"  precision {m['precision']}  recall {m['recall']}"
      f"  false block rate {m['false_block_rate']}")

records with a hidden field name: 32  harmful: 23  clean: 9

field class      requests  harmful  clean  category
subscriber_id           9        6      3  identity
contact_info           10        6      4  contact
address                 5        4      1  location
access_string           8        7      1  access

class distribution of the flagged records
  clean                  9
  indirect_instruction   4
  hidden_field_request  14
  privilege_escalation   2
  source_spoofing        2
  corrupted_input        1

if masking is measured on its own as a decision
  removed fields 32  of which harmful 23  clean 9  harmful untouched 70
  precision 0.7188  recall 0.2473  false block rate 0.0612

Thirty-two records carry a hidden field name, and 9 are clean. Those nine are not an attack: they are ordinary requests from people asking for their own address or verifying their own subscriber identity. A hidden field name is not an accusation but a count: the flag does not show intent, it only says which field the record touches.

The flag is not evenly spread across classes either. Fourteen come from the hidden field request class, but nine are clean and the other nine come from four different harmful classes. Exfiltration is not the behavior of a single class but a result that can be a by-product of many; a defense aimed at one class never sees part of the surface.

The field class table does not equalize cost. Eight requests go to the access_string class and seven are harmful. Disclosing a contact field is upsetting, disclosing an access string is permanent: the string stays valid until renewed, and renewal depends on the disclosure being noticed. A count’s size is not the harm’s size.

The last section measures masking as a decision and shows why that is misleading. Precision 0.7188, recall 0.2473, false block rate 0.0612. Recall being this low does not make masking a poor defense: it never tries to catch 70 of the 93 harmful records, because its job is to remove the field, not to block. A defense put on the wrong measure misleads more than one not measured.

What the Layer Adds and the Residue

The third block builds the exfiltration surface and applies the layers in order: the filter blocks first, masking removes the field on the remainder, and what is left is the residue. The setup runs twice: with the full rule filter, and with a narrow filter whose hidden field rule was removed.

NARROW = ("instruction_mode", "side_effect_tool_name", "source_stamp_mismatch")
SURFACE = [k for k in RECORDS
           if k["features"]["hidden_field_name"] or k["class"] == "hidden_field_request"]
HARMFUL = [k for k in SURFACE if harmful(k)]
print("exfiltration surface:", len(SURFACE), " of which harmful:", len(HARMFUL))
print(f"\n{'setup':<30} {'blocked':>8} {'masked':>11} {'residue':>8}"
      f" {'overall recall':>18}")
for name, rules in (("full rule filter", None), ("hidden field rule removed", NARROW)):
    decision = rule_filter if rules is None else (lambda k, rl=rules: rule_filter(k, rl))
    stopped = [k for k in HARMFUL if decision(k)]
    extra = [k for k in HARMFUL if not decision(k) and redact(k)["redacted"]]
    print(f"{name:<30} {len(stopped):>8} {len(extra):>11}"
          f" {len(HARMFUL) - len(stopped) - len(extra):>8} {measure(RECORDS, decision)['recall']:>18.4f}")

residue = [k for k in HARMFUL if not rule_filter(k) and not redact(k)["redacted"]]
for k in residue:
    print("\nresidue record  no", k["no"], " class", k["class"], " source", k["source"])
    print("  flagged features:", [o for o in FEATURES if k["features"][o]] or "none")
    print("  score", round(score(k), 4), " flagged at the lowest threshold (0.30):",
          model_filter(k, 0.30))
    print("  text:", k["text"])


def build_set(seed):
    """Same definition, another seed: the second run for the band of the residue."""
    ks = []
    for i in range(240):
        r = rng(seed + 6151 * i)
        c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
        source = SOURCES[pick(r(), SOURCE_SHARE[c])]
        feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
        ks.append({"no": i + 1, "class": c, "source": source, "features": feat,
                   "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)})
    return ks


print(f"\n{'run':<8} {'flagged':>8} {'surface':>7} {'harmful':>8} {'narrow filter':>13}"
      f" {'masking':>10} {'residue':>8}")
for name, ks in (("first", RECORDS), ("second", build_set(20260219))):
    y = [k for k in ks if k["features"]["hidden_field_name"] or k["class"] == "hidden_field_request"]
    z = [k for k in y if harmful(k)]
    d = [k for k in z if rule_filter(k, NARROW)]
    e = [k for k in z if not rule_filter(k, NARROW) and redact(k)["redacted"]]
    print(f"{name:<8} {sum(1 for k in ks if k['features']['hidden_field_name']):>8} {len(y):>7}"
          f" {len(z):>8} {len(d):>13} {len(e):>10} {len(z) - len(d) - len(e):>8}")

exfiltration surface: 35  of which harmful: 26

setup                           blocked      masked  residue     overall recall
full rule filter                     25           0        1             0.7957
hidden field rule removed            16           9        1             0.6989

residue record  no 16  class hidden_field_request  source user
  flagged features: none
  score 0.0088  flagged at the lowest threshold (0.30): False
  text: mahallede kesinti ne zaman bitecek

run       flagged surface  harmful narrow filter    masking  residue
first          32      35       26            16          9        1
second         25      26       16            10          6        0

The first row is an example of the second claim of the course, written by name: adding masking on top of the full rule filter adds zero. The filter already blocks 25 of the 26 harmful records, because hidden_field_name is one of its four rules. No record is left for masking to touch, yet it still removes the field from 9 clean records, and their responses come back incomplete. The added layer imposes a cost and catches no new harmful record; a layer’s value comes not from itself but from what the layer beneath it leaves.

The second row confirms this. With the hidden field rule removed, the filter blocks 16 of the 26, overall recall falls from 0.7957 to 0.6989, and 9 records appear for masking to touch. The same defense on the same set protects 0 or 9 records, depending on the layer beneath it. The count of one layer is meaningless alone; it can be read only together with the rest of the stack.

The third number is the same in both setups: the residue is 1. Record 16 is from the hidden field request class, comes from the user, and carries no flag. Its score is 0.0088, so it is not flagged even at the lowest threshold; the rule filter finds no flag to look at, masking no field to remove. Its text does not help either: an ordinary outage question. A harmful record whose profile cannot be told apart from clean ones is residue no layer can catch.

The last table writes that the size of the residue cannot be measured. One record is exactly one step on the recall axis (0.0108), exactly on the resolution of the set, not above it. On the set produced with the second seed, the same setup finds the residue 0. The two runs say: the residue is not zero, but how many it is cannot be said with this set. Knowing it is not zero is enough, because calling a system safe depends not on a zero residue but on writing down what it is.

Summary

• The loss in exfiltration cannot be undone: a blocked operation can be retried, a disclosed field cannot be taken back; so the measure is a count, not a block.
• 32 records carry a hidden field name and 9 are clean; the flag is a count, not an accusation, and with fourteen records from the hidden field request class it spreads into four harmful classes.
• Across field classes, count and harm are not proportional: the smallest group, the access string (8 requests, 7 harmful), carries the most permanent result.
• Measured as a decision, masking has recall 0.2473, but this is a wrong measure, not a fault of masking; its job is to remove the field, not to block.
• On top of the full rule filter, masking adds 0, at the cost of the field removed from 9 clean records; with the hidden field rule removed, the same layer protects 9 records.
• The residue is 1 record: it carries no flag, scores 0.0088, and has a profile indistinguishable from clean records; in the second run it is 0, so its size cannot be measured with this set.

Next Step

In three lessons the surface was read from one place: the text entering the model. The two forms of injection were separated by source, the filter threshold was swept, records requesting a hidden field were counted, and the residue none catches was written down. All these measures concerned what the model reads. A system has more than reading: the tools it can call, their side effects and the authority behind them. The training data has not been asked about at all. The harm of such a record comes not from carrying a piece of information but from triggering an action, and the defense changes too: instead of filtering, asking. The next lesson counts the records that carry a side-effect tool name and measures the precision of the human approval gate.