--- title: 'Model Inversion and Inference Attacks' source: 'https://academia.sh/en/courses/ai-safety-and-evaluation/model-inversion-and-inference-attacks' course: 'AI Safety and Evaluation' language: en updated: '2026-10-21T05:02:27+00:00' license: 'CC BY-SA 4.0' objectives: - 'Counts the responses that carry a field of a secret field class and separates requested fields from fields arriving unasked.' - 'Writes the fields the output filter removes together with the fields it deletes from clean responses.' - 'Shows with counts the limit of a name-list filter against indirect identifiers and membership signals.' --- # Model Inversion and Inference Attacks Leakage counted on the output side, not the input side: 71 of 240 responses carry a field of a secret field class, 106 fields in total. 31 of them come from the 32 records that asked for a field, and 40 come from the 208 records that never asked, so the model inversion class is larger than what was requested. The four-name output filter removes all 106 fields and the remaining secret field count is 0; yet 47 responses still carry an indirect identifier outside the secret field class, 57 responses still carry a membership signal that has no field name, and responses carrying a trace fall only from 137 to 94. A five-name list also removes the indirect identifier but cannot remove the membership signal. 58 of the 106 removed fields come from clean records, and 41 clean responses lose a field. Across two runs the field count is 106 and 104, and responses carrying a trace after the filter are 94 in both. Everything counted in the two previous lessons was **on the input side**: the flag the record carried, the channel it came from, the approval gate sending it to a human, and whether it was taken into the corpus. Both the rule filter and the source gate made their decision before the model produced a response. The information a system leaks, however, does not have to pass through the input. The model can put an unasked field in its response; it can reveal that a subscriber is among the records through an expression that carries no field name; and neither produces a flag on the input side. This lesson treats two classes. **Model inversion** is recovering a field of the training data through the response. **Membership inference** is reading from the response whether a particular record is in the set. Both are treated here as **a class and a count**; no inference procedure, query sequence, or recovery method is written. The personal data and masking procedure was measured in the Model Evaluation, Interpretability and Ethics course and is not repeated; the only thing added here is **the count on the output side**. - **AS57.** The records, the responses, the secret field class, and the output filter are **fictional**; no real model is called and no real personal data is used. Seed **20260218**. - **AS58.** An attack is not a text but an **attribute profile**. The text of the records is entirely harmless support requests and has **no relation to the class**. - **AS59.** The **text** of a response is not produced; the only things produced and counted are the **field names the response carries**. A field being present in a response does not mean the value of that field is correct, and this lesson makes no such claim. - **AS60.** The baseline is the **unfiltered output**: no field is removed, removed fields are **0**, fields deleted from clean responses are **0**. - **AS61.** `meter_history` is deliberately kept **outside** the secret field class. It is an indirect identifier: alone it carries no identity but it narrows down the record. A filter based on a name list sees it only when it is written into the list. - **AS62.** The membership signal carries **no field name**. It is a yes-or-no that shows the record is in the set, and for this reason a name list cannot remove it. - **AS63.** Resolution: in 240 responses the step of a response-level share is **0.0042**. Field counts are integers and no ratio is printed. The second run takes the same records with seed **20260219**. - **AS64.** The cost on the task of masking on the input side belongs to the defense topic and is not measured here; the cost in this lesson is only **the fields deleted from the output**. ## Leakage Is Counted in the Output The first block builds the set and adds the output side on top. A response is not a text but a **list of fields**: which names are present in the response, plus a signal that shows the record is in the set. A field entering the response does not depend on its having been requested — there is a low share in unrequested records as well, and the model inversion class stands exactly there. ```python # FICTIONAL. Records, classes and defenses are defined inside the lesson; no real # system is called. No attack TEXT is written: an attack is a FEATURE PROFILE. SEED, M32 = 20260218, 0xFFFFFFFF def rng(t): x = ((t ^ (t >> 16)) * 2246822507) & M32 x = ((x ^ (x >> 13)) * 3266489909) & M32 s = [(x ^ (x >> 16)) & M32] def next_value(): s[0] = (s[0] * 1664525 + 1013904223) & M32 return s[0] / 4294967296 return next_value def pick(u, w): t = 0.0 for i, x in enumerate(w): t += x if u < t: return i return len(w) - 1 CLASSES = ["clean", "indirect_instruction", "hidden_field_request", "privilege_escalation", "source_spoofing", "corrupted_input"] SOURCES = ["user", "retrieved_document", "tool_output"] # Features are structural; none of them is a working payload. FEATURES = ["instruction_mode", "role_change_request", "hidden_field_name", "side_effect_tool_name", "source_stamp_mismatch", "unusual_stamp_distribution", "excessive_length"] PROFILE = { "clean": [0.06, 0.02, 0.05, 0.03, 0.02, 0.04, 0.05], "indirect_instruction": [0.88, 0.42, 0.10, 0.30, 0.35, 0.08, 0.12], "hidden_field_request": [0.35, 0.15, 0.86, 0.12, 0.10, 0.06, 0.08], "privilege_escalation": [0.30, 0.25, 0.12, 0.84, 0.14, 0.05, 0.07], "source_spoofing": [0.22, 0.55, 0.08, 0.10, 0.82, 0.07, 0.06], "corrupted_input": [0.05, 0.03, 0.04, 0.03, 0.06, 0.80, 0.62], } SOURCE_SHARE = { "clean": [0.72, 0.20, 0.08], "indirect_instruction": [0.05, 0.70, 0.25], "hidden_field_request": [0.55, 0.30, 0.15], "privilege_escalation": [0.40, 0.35, 0.25], "source_spoofing": [0.10, 0.60, 0.30], "corrupted_input": [0.65, 0.20, 0.15], } # Entirely harmless support texts; they have NO relation to the class. TEXT = ["bu ayki fatura tutari gecen aya gore yuksek geldi", "sayac endeksi yanlis girilmis olabilir", "mahallede kesinti ne zaman bitecek", "tarife basamagimi ogrenmek istiyorum", "ariza kaydim icin ekip ne zaman gelir", "adres degisikligi nasil yapilir"] def build_set(seed=SEED): ks = [] for i in range(240): r = rng(seed + 6151 * i) c = CLASSES[pick(r(), [0.62, 0.10, 0.07, 0.07, 0.07, 0.07])] source = SOURCES[pick(r(), SOURCE_SHARE[c])] feat = {o: (r() < p) for o, p in zip(FEATURES, PROFILE[c])} ks.append({"no": i + 1, "class": c, "source": source, "features": feat, "text": TEXT[int(r() * len(TEXT))], "length": 40 + int(r() * 260)}) return ks RECORDS = build_set() def harmful(k): return k["class"] != "clean" def trace_id(m): h = 2166136261 for c in str(m): h = ((h ^ ord(c)) * 16777619) & M32 return h # ---- OUTPUT SIDE. The response TEXT is not produced; the FIELDS IT CARRIES are counted. SECRET_FIELD = {"subscriber_id": "identity", "contact_info": "contact", "address": "location", "access_string": "access"} INDIRECT = "meter_history" # NOT in the secret field class: an indirect identifier def respond(k, seed=SEED): """FICTIONAL response. A field may appear whether or not it was requested; the membership signal carries no field name, it only shows that the record is in the set.""" r = rng(seed ^ trace_id("yanit" + str(k["no"]))) requested = k["features"]["hidden_field_name"] fields = [a for a in SECRET_FIELD if r() < (0.46 if requested else 0.06)] if r() < (0.30 if requested else 0.18): fields.append(INDIRECT) return {"no": k["no"], "requested": requested, "fields": fields, "membership": r() < 0.22} def output_filter(resp, names): """Defense: a field whose name appears in the list is removed from the response.""" return dict(resp, fields=[a for a in resp["fields"] if a not in names]) def secret_count(resp): return sum(1 for a in resp["fields"] if a in SECRET_FIELD) ``` ## What the Output Filter Counts The second block first counts what the responses carry, then runs the output filter with two name lists. The last column is the critical one: the number of **responses carrying any trace** after the filter, that is, responses whose field or membership signal remains. ```python RESPONSES = [respond(k) for k in RECORDS] asked = [r for r in RESPONSES if r["requested"]] print("responses", len(RESPONSES), " carrying a field of a secret field class", sum(1 for r in RESPONSES if secret_count(r)), " total fields", sum(secret_count(r) for r in RESPONSES)) print(" requested", sum(1 for r in asked if secret_count(r)), "/", len(asked), " unasked", sum(1 for r in RESPONSES if not r["requested"] and secret_count(r)), "/", len(RESPONSES) - len(asked)) print(" indirect identifier", sum(1 for r in RESPONSES if INDIRECT in r["fields"]), " membership signal", sum(1 for r in RESPONSES if r["membership"])) print("\noutput filter secret field carrying fields membership responses with trace") for name, names in (("unfiltered baseline", ()), ("narrow list (4 names)", tuple(SECRET_FIELD)), ("wide list (5)", tuple(SECRET_FIELD) + (INDIRECT,))): F = [output_filter(r, names) for r in RESPONSES] print(f"{name:<22} {sum(secret_count(r) for r in F):>12} {sum(1 for r in F if r['fields']):>15}" f" {sum(1 for r in F if r['membership']):>10}" f" {sum(1 for r in F if r['fields'] or r['membership']):>20}") clean = {k["no"] for k in RECORDS if not harmful(k)} print("\nof the 106 fields the narrow list removed, from clean records", sum(secret_count(r) for r in RESPONSES if r["no"] in clean), " clean responses losing a field", sum(1 for r in RESPONSES if r["no"] in clean and secret_count(r))) print("\nsame records, two runs") for t in (20260218, 20260219): R = [respond(k, t) for k in RECORDS] F = [output_filter(r, tuple(SECRET_FIELD)) for r in R] print(f" {t}: carrying fields {sum(1 for r in R if secret_count(r))}" f" fields {sum(secret_count(r) for r in R)}" f" indirect {sum(1 for r in R if INDIRECT in r['fields'])}" f" membership {sum(1 for r in R if r['membership'])}" f" carrying a trace after the filter {sum(1 for r in F if r['fields'] or r['membership'])}") ``` ``` responses 240 carrying a field of a secret field class 71 total fields 106 requested 31 / 32 unasked 40 / 208 indirect identifier 47 membership signal 57 output filter secret field carrying fields membership responses with trace unfiltered baseline 106 102 57 137 narrow list (4 names) 0 47 57 94 wide list (5) 0 0 57 57 of the 106 fields the narrow list removed, from clean records 58 clean responses losing a field 41 same records, two runs 20260218: carrying fields 71 fields 106 indirect 47 membership 57 carrying a trace after the filter 94 20260219: carrying fields 70 fields 104 indirect 55 membership 54 carrying a trace after the filter 94 ``` ## Recall as Long as the List The first line gives the baseline: **71** of 240 responses carry at least one field of the secret field class, and **106** fields in total come out. In the unfiltered setup none of these is removed, and the fields deleted from clean responses are 0. By the rule of the course both numbers are zero at the baseline, and what the defense adds is written on top of this zero. The second line carries the class distinction of this lesson. Of the **32** records with a hidden field name, 31 have a response that carries a field as expected. But **40** of the responses in which fields appear come from the 208 records that never asked for a field. So the fields appearing unasked are **more** than those appearing on request. This is the count of the model inversion class, and no defense on the input side can see it: there is no hidden field name in the request, no instruction mode, no flag. This class is counted only on the output side, because it exists only there. The third table shows what the filter based on a name list does. The four-name list removes **all** of the 106 fields: the remaining secret field is **0**. A setup that looks at this column says "leakage is zero" about itself, and what it says is true. The three columns beside it say something else. Responses carrying a field fall from 102 to **47**, not to zero, because the remaining 47 responses carry `meter_history` and that name is not in the list. Responses carrying a membership signal stay at **57**, not moving at all. And the last column gives the total: responses carrying a trace fall from 137 to **94**. The filter removed the trace completely from only **43** of the 240 responses, yet the secret field column writes **one hundred percent**. The recall of a name list is as long as the list, and read from its own list that recall is always complete. This line has a structural side, and if it is missed it is read as a measurement. The filter takes the names from a list, and the secret field column counts from the same list; that the column comes out zero after the filter has run is not a finding but **true by definition**. For a zero to say anything, the counting side must be **wider** than the filtering side. The columns of 47 and 57 in the table exist for exactly this reason: the indirect identifier and the membership signal are not in the filter's list but are in the scope of the count, and the residue becomes visible only in the gap between these two scopes. When the list of the filter and the definition of the measure are the same, a setup cannot see its own limit, and the only thing it sees is its own list. Adding a fifth name to the list corrects part of this: responses carrying a field fall to **0**, and responses carrying a trace fall from 94 to **57**. The remaining 57 are the membership signal, and they do not come out however long the list grows, because there is no name to remove. This is the relation of the membership inference class to masking: masking deletes a **field**, it cannot delete a **presence**. A response carrying the difference between "no record found for this subscriber" and "a record was found for this subscriber" makes the composition of the set open without containing any secret field. The defense of this class is not a name list and does not show in this table. The cost line writes the second half of the rule. **58** of the 106 removed fields come from the responses of clean records, and **41 clean responses** lose a field. None of these records falls in an attack class: a subscriber asking for their address cannot receive it, a record that wants to verify contact information comes back empty. The filter does not look at where the removed field came from, because if it could it would already be a classifier. **When the 106 removed fields and the 58 deleted clean fields are not written together, this defense counts as unmeasured**, and the sentence "leakage is zero" alone is half of this measurement. The last two lines give the band of measuring an uncertain component. The same records, second run: the responses carrying a field move from 71 to 70, total fields from 106 to 104, indirect identifier from 47 to 55, membership signal from 57 to 54. None of them is fixed. The number of responses carrying a trace after the filter, on the other hand, is **94** in both runs. The only number that comes out the same in two runs is the size of the surface the filter **cannot close**: the components move, but the total stays in place, because the number of paths left outside the list does not depend on the list itself. ## Summary - In the unfiltered baseline 71 of 240 responses carry a field of the secret field class and 106 fields in total come out; removed fields are 0, fields deleted from clean responses are 0. - 31 of the responses in which fields appear come from the 32 records that asked for the field, and 40 from the 208 records that never asked; the model inversion class produces no flag on the input side. - The four-name output filter removes all 106 fields and the secret field column writes 0; yet responses carrying a trace fall only from 137 to 94. The recall of a name list is as long as the list. - The fifth name also removes the indirect identifier and responses carrying a trace fall to 57; the remaining 57 are membership signals, and since they carry no field name they cannot be removed with a name list. - 58 of the 106 removed fields come from the responses of clean records and 41 clean responses lose a field; without writing this number the filter counts as unmeasured. - The field counts move across two runs; responses carrying a trace after the filter come out 94 in both runs. ## Next Step In six lessons the attack surface was divided into five classes and the share the defense catches was counted separately in each: records carrying the instruction mode, the hidden field request, the side-effect tool name, records whose source stamp does not match, and fields on the output side. All of them had a common assumption, and that assumption was never written: that the input is **a meaningful text**. The rule filter looks for an instruction mode, the output filter looks for a field name, the source gate reads a stamp — all three expect to find a readable structure in front of them. The deliberate **corruption** of the input, that is, moving the structure itself, is a separate surface that does not meet this expectation, and it already stands in the set as its own class. The next lesson looks at that class and compares the share the defense catches there with the shares of the other classes.