Model Inversion and Inference Attacks

Everything counted in the two previous lessons was on the input side: the flag the record carried, the channel it came from, the approval gate sending it to a human, and whether it was taken into the corpus. Both the rule filter and the source gate made their decision before the model produced a response. The information a system leaks, however, does not have to pass through the input. The model can put an unasked field in its response; it can reveal that a subscriber is among the records through an expression that carries no field name; and neither produces a flag on the input side.

This lesson treats two classes. Model inversion is recovering a field of the training data through the response. Membership inference is reading from the response whether a particular record is in the set. Both are treated here as a class and a count; no inference procedure, query sequence, or recovery method is written. The personal data and masking procedure was measured in the Model Evaluation, Interpretability and Ethics course and is not repeated; the only thing added here is the count on the output side.

• AS57. The records, the responses, the secret field class, and the output filter are fictional; no real model is called and no real personal data is used. Seed 20260218.
• AS58. An attack is not a text but an attribute profile. The text of the records is entirely harmless support requests and has no relation to the class.
• AS59. The text of a response is not produced; the only things produced and counted are the field names the response carries. A field being present in a response does not mean the value of that field is correct, and this lesson makes no such claim.
• AS60. The baseline is the unfiltered output: no field is removed, removed fields are 0, fields deleted from clean responses are 0.
• AS61. meter_history is deliberately kept outside the secret field class. It is an indirect identifier: alone it carries no identity but it narrows down the record. A filter based on a name list sees it only when it is written into the list.
• AS62. The membership signal carries no field name. It is a yes-or-no that shows the record is in the set, and for this reason a name list cannot remove it.
• AS63. Resolution: in 240 responses the step of a response-level share is 0.0042. Field counts are integers and no ratio is printed. The second run takes the same records with seed 20260219.
• AS64. The cost on the task of masking on the input side belongs to the defense topic and is not measured here; the cost in this lesson is only the fields deleted from the output.

Leakage Is Counted in the Output

The first block builds the set and adds the output side on top. A response is not a text but a list of fields: which names are present in the response, plus a signal that shows the record is in the set. A field entering the response does not depend on its having been requested — there is a low share in unrequested records as well, and the model inversion class stands exactly there.

# FICTIONAL. Records, classes and defenses are defined inside the lesson; no real
# system is called. No attack TEXT is written: an attack is a FEATURE PROFILE.
SEED, M32 = 20260218, 0xFFFFFFFF


def rng(t):
    x = ((t ^ (t >> 16)) * 2246822507) & M32
    x = ((x ^ (x >> 13)) * 3266489909) & M32
    s = [(x ^ (x >> 16)) & M32]

    def next_value():
        s[0] = (s[0] * 1664525 + 1013904223) & M32
        return s[0] / 4294967296
    return next_value


def pick(u, w):
    t = 0.0
    for i, x in enumerate(w):
        t += x
        if u < t:
            return i
    return len(w) - 1


CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation",
           "source_spoofing", "corrupted_input"]
SOURCES = ["user", "retrieved_document", "tool_output"]
# Features are structural; none of them is a working payload.
FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name",
            "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"]
PROFILE = {
    "clean":                [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05],
    "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12],
    "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08],
    "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07],
    "source_spoofing":      [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06],
    "corrupted_input":      [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62],
}
SOURCE_SHARE = {
    "clean":                [0.72, 0.20, 0.08],
    "indirect_instruction": [0.05, 0.70, 0.25],
    "hidden_field_request": [0.55, 0.30, 0.15],
    "privilege_escalation": [0.40, 0.35, 0.25],
    "source_spoofing":      [0.10, 0.60, 0.30],
    "corrupted_input":      [0.65, 0.20, 0.15],
}
# Entirely harmless support texts; they have NO relation to the class.
TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi",
        "sayac endeksi yanlis girilmis olabilir",
        "mahallede kesinti ne zaman bitecek",
        "tarife basamagimi ogrenmek istiyorum",
        "ariza kaydim icin ekip ne zaman gelir",
        "adres degisikligi nasil yapilir"]


def build_set(seed=SEED):
    ks = []
    for i in range(240):
        r = rng(seed + 6151 * i)
        c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])]
        source = SOURCES[pick(r(), SOURCE_SHARE[c])]
        feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])}
        ks.append({"no": i + 1, "class": c, "source": source, "features": feat,
                   "text": TEXT[int(r() * len(TEXT))],
                   "length": 40 + int(r() * 260)})
    return ks


RECORDS = build_set()


def harmful(k):
    return k["class"] != "clean"


def trace_id(m):
    h = 2166136261
    for c in str(m):
        h = ((h ^ ord(c)) * 16777619) & M32
    return h


# ---- OUTPUT SIDE. The response TEXT is not produced; the FIELDS IT CARRIES are counted.
SECRET_FIELD = {"subscriber_id": "identity", "contact_info": "contact", "address": "location",
                "access_string": "access"}
INDIRECT = "meter_history"       # NOT in the secret field class: an indirect identifier


def respond(k, seed=SEED):
    """FICTIONAL response. A field may appear whether or not it was requested; the membership
    signal carries no field name, it only shows that the record is in the set."""
    r = rng(seed ^ trace_id("yanit" + str(k["no"])))
    requested = k["features"]["hidden_field_name"]
    fields = [a for a in SECRET_FIELD if r() < (0.46 if requested else 0.06)]
    if r() < (0.30 if requested else 0.18):
        fields.append(INDIRECT)
    return {"no": k["no"], "requested": requested, "fields": fields, "membership": r() < 0.22}


def output_filter(resp, names):
    """Defense: a field whose name appears in the list is removed from the response."""
    return dict(resp, fields=[a for a in resp["fields"] if a not in names])


def secret_count(resp):
    return sum(1 for a in resp["fields"] if a in SECRET_FIELD)

What the Output Filter Counts

The second block first counts what the responses carry, then runs the output filter with two name lists. The last column is the critical one: the number of responses carrying any trace after the filter, that is, responses whose field or membership signal remains.

RESPONSES = [respond(k) for k in RECORDS]
asked = [r for r in RESPONSES if r["requested"]]
print("responses", len(RESPONSES),
      " carrying a field of a secret field class", sum(1 for r in RESPONSES if secret_count(r)),
      " total fields", sum(secret_count(r) for r in RESPONSES))
print("  requested", sum(1 for r in asked if secret_count(r)), "/", len(asked),
      "  unasked", sum(1 for r in RESPONSES if not r["requested"] and secret_count(r)),
      "/", len(RESPONSES) - len(asked))
print("  indirect identifier", sum(1 for r in RESPONSES if INDIRECT in r["fields"]),
      " membership signal", sum(1 for r in RESPONSES if r["membership"]))

print("\noutput filter          secret field carrying fields membership responses with trace")
for name, names in (("unfiltered baseline", ()), ("narrow list (4 names)", tuple(SECRET_FIELD)),
                    ("wide list (5)", tuple(SECRET_FIELD) + (INDIRECT,))):
    F = [output_filter(r, names) for r in RESPONSES]
    print(f"{name:<22} {sum(secret_count(r) for r in F):>12} {sum(1 for r in F if r['fields']):>15}"
          f" {sum(1 for r in F if r['membership']):>10}"
          f" {sum(1 for r in F if r['fields'] or r['membership']):>20}")

clean = {k["no"] for k in RECORDS if not harmful(k)}
print("\nof the 106 fields the narrow list removed, from clean records",
      sum(secret_count(r) for r in RESPONSES if r["no"] in clean),
      " clean responses losing a field", sum(1 for r in RESPONSES if r["no"] in clean and secret_count(r)))

print("\nsame records, two runs")
for t in (20260218, 20260219):
    R = [respond(k, t) for k in RECORDS]
    F = [output_filter(r, tuple(SECRET_FIELD)) for r in R]
    print(f"  {t}: carrying fields {sum(1 for r in R if secret_count(r))}"
          f"  fields {sum(secret_count(r) for r in R)}"
          f"  indirect {sum(1 for r in R if INDIRECT in r['fields'])}"
          f"  membership {sum(1 for r in R if r['membership'])}"
          f"  carrying a trace after the filter {sum(1 for r in F if r['fields'] or r['membership'])}")

responses 240  carrying a field of a secret field class 71  total fields 106
  requested 31 / 32   unasked 40 / 208
  indirect identifier 47  membership signal 57

output filter          secret field carrying fields membership responses with trace
unfiltered baseline             106             102         57                  137
narrow list (4 names)             0              47         57                   94
wide list (5)                     0               0         57                   57

of the 106 fields the narrow list removed, from clean records 58  clean responses losing a field 41

same records, two runs
  20260218: carrying fields 71  fields 106  indirect 47  membership 57  carrying a trace after the filter 94
  20260219: carrying fields 70  fields 104  indirect 55  membership 54  carrying a trace after the filter 94

Recall as Long as the List

The first line gives the baseline: 71 of 240 responses carry at least one field of the secret field class, and 106 fields in total come out. In the unfiltered setup none of these is removed, and the fields deleted from clean responses are 0. By the rule of the course both numbers are zero at the baseline, and what the defense adds is written on top of this zero.

The second line carries the class distinction of this lesson. Of the 32 records with a hidden field name, 31 have a response that carries a field as expected. But 40 of the responses in which fields appear come from the 208 records that never asked for a field. So the fields appearing unasked are more than those appearing on request. This is the count of the model inversion class, and no defense on the input side can see it: there is no hidden field name in the request, no instruction mode, no flag. This class is counted only on the output side, because it exists only there.

The third table shows what the filter based on a name list does. The four-name list removes all of the 106 fields: the remaining secret field is 0. A setup that looks at this column says “leakage is zero” about itself, and what it says is true. The three columns beside it say something else. Responses carrying a field fall from 102 to 47, not to zero, because the remaining 47 responses carry meter_history and that name is not in the list. Responses carrying a membership signal stay at 57, not moving at all. And the last column gives the total: responses carrying a trace fall from 137 to 94. The filter removed the trace completely from only 43 of the 240 responses, yet the secret field column writes one hundred percent. The recall of a name list is as long as the list, and read from its own list that recall is always complete.

This line has a structural side, and if it is missed it is read as a measurement. The filter takes the names from a list, and the secret field column counts from the same list; that the column comes out zero after the filter has run is not a finding but true by definition. For a zero to say anything, the counting side must be wider than the filtering side. The columns of 47 and 57 in the table exist for exactly this reason: the indirect identifier and the membership signal are not in the filter’s list but are in the scope of the count, and the residue becomes visible only in the gap between these two scopes. When the list of the filter and the definition of the measure are the same, a setup cannot see its own limit, and the only thing it sees is its own list.

Adding a fifth name to the list corrects part of this: responses carrying a field fall to 0, and responses carrying a trace fall from 94 to 57. The remaining 57 are the membership signal, and they do not come out however long the list grows, because there is no name to remove. This is the relation of the membership inference class to masking: masking deletes a field, it cannot delete a presence. A response carrying the difference between “no record found for this subscriber” and “a record was found for this subscriber” makes the composition of the set open without containing any secret field. The defense of this class is not a name list and does not show in this table.

The cost line writes the second half of the rule. 58 of the 106 removed fields come from the responses of clean records, and 41 clean responses lose a field. None of these records falls in an attack class: a subscriber asking for their address cannot receive it, a record that wants to verify contact information comes back empty. The filter does not look at where the removed field came from, because if it could it would already be a classifier. When the 106 removed fields and the 58 deleted clean fields are not written together, this defense counts as unmeasured, and the sentence “leakage is zero” alone is half of this measurement.

The last two lines give the band of measuring an uncertain component. The same records, second run: the responses carrying a field move from 71 to 70, total fields from 106 to 104, indirect identifier from 47 to 55, membership signal from 57 to 54. None of them is fixed. The number of responses carrying a trace after the filter, on the other hand, is 94 in both runs. The only number that comes out the same in two runs is the size of the surface the filter cannot close: the components move, but the total stays in place, because the number of paths left outside the list does not depend on the list itself.

Summary

• In the unfiltered baseline 71 of 240 responses carry a field of the secret field class and 106 fields in total come out; removed fields are 0, fields deleted from clean responses are 0.
• 31 of the responses in which fields appear come from the 32 records that asked for the field, and 40 from the 208 records that never asked; the model inversion class produces no flag on the input side.
• The four-name output filter removes all 106 fields and the secret field column writes 0; yet responses carrying a trace fall only from 137 to 94. The recall of a name list is as long as the list.
• The fifth name also removes the indirect identifier and responses carrying a trace fall to 57; the remaining 57 are membership signals, and since they carry no field name they cannot be removed with a name list.
• 58 of the 106 removed fields come from the responses of clean records and 41 clean responses lose a field; without writing this number the filter counts as unmeasured.
• The field counts move across two runs; responses carrying a trace after the filter come out 94 in both runs.

Next Step

In six lessons the attack surface was divided into five classes and the share the defense catches was counted separately in each: records carrying the instruction mode, the hidden field request, the side-effect tool name, records whose source stamp does not match, and fields on the output side. All of them had a common assumption, and that assumption was never written: that the input is a meaningful text. The rule filter looks for an instruction mode, the output filter looks for a field name, the source gate reads a stamp — all three expect to find a readable structure in front of them. The deliberate corruption of the input, that is, moving the structure itself, is a separate surface that does not meet this expectation, and it already stands in the set as its own class. The next lesson looks at that class and compares the share the defense catches there with the shares of the other classes.