Contextual integrity, from Nissenbaum to LLMs

Everything else in the paper depends on one choice of definition. If privacy meant "never output sensitive facts," the right assistant would simply refuse to use its memory, and the benchmark would be trivial. Because the paper adopts contextual integrity, an assistant can fail in two directions: by sending a fact where it does not belong, or by holding back a fact the task needs. That two-sided view is why the paper always reports a leak score next to a usefulness score, and why the headline spread across eight models runs from 14.82% to 69.14% Violation@5 (Table 1) without any model getting both scores right.

First, which "context"?

In LLM writing, "context" usually means the context window: the tokens the model can see. This paper uses the word in its older, social sense, and says so in a footnote to the abstract:

Throughout this paper, "context" refers to the social context for information sharing (e.g., the task being performed), not the model's context window unless specified.— CIMemories, abstract, footnote a

The distinction matters because the paper uses both meanings at once. The user's memories sit in the context window as a text prefix (covered in Part 02), while the social context is the task and recipient: "apply for a bank loan" to a loan officer, "explain current symptoms" to an emergency room intake desk. When this map says context, it means the second.

Privacy as appropriate flow

Helen Nissenbaum's 2004 article Privacy as Contextual Integrity (§1 cites it alongside Barth et al. 2006) starts from an observation about ordinary life. People share medical details with doctors, salaries with mortgage lenders and relationship troubles with friends, and none of that feels like a privacy loss. The same details moving to an employer, a landlord or a stranger do feel like one. What changed was not the information but where it went.

CI therefore makes the unit of analysis an information flow, not a piece of data. Every social setting (medicine, banking, friendship, employment) carries expectations about what information belongs in it and how that information may travel onward. The 2004 article names these as norms of appropriateness and norms of distribution. A flow respects contextual integrity when it matches those norms; it violates CI when it breaks them, even if the information was never secret.

Later formal work, including the Barth, Datta, Mitchell and Nissenbaum paper the authors cite, describes a flow by a small set of parameters. The common version has five:

Change any one parameter and the same fact can go from fine to a violation. This five-part framing is background from the CI literature, not something CIMemories restates; the paper's own summary is one line: CI "defines privacy violations as inappropriate information flows against societal norms" (§1).

Why "public vs private" does not work

The intuitive alternative is a sensitivity list: mark health, finances and legal trouble as private, everything else as shareable. The paper's running example in Figure 1 shows why that fails. A synthetic user, Douglas Perry, has memories that include his annual income and a well-managed case of mild hypertension.

Neither fact has a fixed status. A rule that hides health data breaks the doctor's appointment; a rule that allows financial data sends his income to his physician. Any static label is wrong for at least one recipient. Contextual integrity handles this because the label belongs to the pair (fact, context), not the fact alone. CIMemories builds its dataset that way: every attribute gets a separate CI label for each task, either necessary or inappropriate, and pairs the labellers disagree on are set aside as ambiguous (how those labels are produced is Part 03).

Mapped onto the five parameters, a CIMemories flow looks like this: the subject is the user, the sender is the assistant writing on the user's behalf, the information type is a remembered attribute within one of nine information domains, the recipient comes from the task, and the norm is the per-pair label. The transmission principle is folded into the task description, which says why the message is being sent.

How earlier benchmarks used CI

§2 places CIMemories in a short line of work that turned CI into tests for language models. Each one asks a slightly different question:

Read as a sequence, these move from "can a model tell which flows are acceptable?" toward "does a model behave acceptably when doing real work?" ConfAide's 39% figure was an early signal that the second question has a worse answer than the first. The paper also notes, in §4.1.2, that ConfAide and PrivacyLens found LLMs often agree with, or are more conservative than, humans when labelling information as private, which is the observation CIMemories relies on when it uses a model to produce its labels.

The gap: one secret is too easy

The paper's critique of this line of work is about scale and structure, not correctness. Prior benchmarks, it says in §1, typically test

simple scenarios with minimal information (e.g., a single secret to protect and one piece of information to reveal)— CIMemories, §1

A scenario with one secret tells you whether the model can spot the dangerous fact when it is the obvious odd one out. It does not tell you what happens when the model is holding a dossier and must pick a handful of relevant details from more than a hundred, nearly all of which are fine somewhere. §2 makes the same point about persistent memory: the density of private information keeps rising as a memory grows, and the same attribute can be appropriate for one task and inappropriate for another.

CIMemories answers with two kinds of composition (Figure 1). Memory composition lets the authors choose how many attributes are in memory and which are necessary or inappropriate for a given task. Multi-task composition runs each user through many tasks, so a leak can be counted the first time any task exposes a fact. The resulting set has 10 profiles, each with an average of 147 attributes and 45 contexts according to §1 (Table 2 gives 146.7 ± 2.5 and 45.7 ± 2.9). Both compositions get their own page, Part 05.

Two smaller differences follow from the CI framing. First, CIMemories scores sharing as well as withholding: a model that omits a necessary fact loses completeness, so silence is not a safe answer. Second, labels are tied to recipients drawn from everyday interactions (doctors, employers, landlords, attorneys) rather than to a list of sensitive categories, which is the CI view put into practice.

Check yourself

1. Why would a sensitivity list (health = private, finance = private) fail on the Douglas Perry example?
   Answer

   Because the same fact changes status with the recipient. In Figure 1 income is necessary for the financial advisor and inappropriate for the doctor, while hypertension is the reverse. Any fixed label is wrong for one of the two tasks; CI labels the (attribute, task) pair instead.

2. When this paper says "context," what does it mean, and what does it not mean?
   Answer

   The social context of the sharing, meaning the task and its recipient. It does not mean the model's context window, unless the paper says so, as stated in the abstract's footnote.

3. What is the paper's main complaint about benchmarks like ConfAide and CI-Bench?
   Answer

   Not that they are wrong but that they are small in structure: minimal information, typically one secret to protect and one fact to reveal. They do not model a memory that accumulates over time, where many attributes are each appropriate for some recipients and not others (§1, §2).

Further reading, verified

• CIMemories §2, Related Work, the paper's own survey of CI benchmarks and memory-augmented LLMs; with Figure 1 for the Douglas Perry example.
• Nissenbaum (2004), "Privacy as Contextual Integrity", Washington Law Review 79:119. The source of the theory; cited in §1.
• Mireshghallah et al., "Can LLMs Keep a Secret?" (ConfAide), ICLR 2024; the 39% GPT-4 figure.
• Shao et al., PrivacyLens and Cheng et al., CI-Bench, both cited in §2.
• Fan et al., GoldCoin; Shvartzshnaider et al., LLM-CI (its arXiv page now carries the later title "Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric"); Zharmagambetov et al., AgentDAM. All cited in §2.
• Agentic privacy field map, a sibling explainer of "Open and Emergent Problems in Agentic Privacy and Security: A Contextual Angle" (field map extra, not cited by the paper).