Read the paper in 10 minutes

The paper has one claim and three supporting experiments. The claim: with persistent memory, privacy stops being about one secret and becomes about routing many facts to many recipients, and current models route badly. The experiments test whether the obvious fixes (a bigger model, a reasoning chain, a careful prompt) change that, and whether more memory makes it worse. Read this page top to bottom and you have the argument; follow any section's link for the detail.

§1 Introduction: the question

Assistants now keep notes about their users across sessions. One way to use those notes is retrieval; the simpler way that long context windows made popular is to paste every remembered statement in front of the conversation, a needle-in-a-haystack setup. Either way the model now writes emails, replies and forms on the user's behalf while holding a persistent memory full of income figures, diagnoses and court cases. The paper asks whether the model uses that pile appropriately for whoever is receiving the message.

The yardstick is contextual integrity: a disclosure is a violation when it breaks the norms of the setting it happens in, not because the fact is secret in itself. A footnote in the paper pins down that "context" here means the social setting of the task, not the model's context window. The introduction names the two design moves that set this benchmark apart, memory composition and multi-task composition, and previews the headline numbers (§1, Figure 1).

Figure 1's running example makes the problem concrete. A synthetic user's memory holds both an annual income of 72k and well-managed mild hypertension. A message to a financial advisor should mention the income and not the blood pressure; a message to a primary care doctor should do the reverse. No fixed "private" flag on either fact gets both cases right. Deep dives: contextual integrity and why memory makes it hard.

§2 Related work: what was missing

Contextual integrity is not new to LLM evaluation. ConfAide found GPT-4 revealing private information 39% of the time; PrivacyLens turned privacy seeds into agent trajectories; CI-Bench built 44,000 synthetic dialogues across eight domains; GoldCoin tied the theory to laws such as HIPAA; LLM-CI used factorial vignettes; AgentDAM, contemporaneous, measured data minimization in web agents (§2). The gap the authors point to is scale of the haystack: those tests usually hand the model one secret and one thing to share. Memory-augmented systems, from retrieval-augmented generation to Mem0-style memory stores, accumulate dense personal data where the same attribute is fine in one flow and wrong in the next, and no earlier CI benchmark modelled that. Deep dive: p01.

§3 Framework: two scores, always together

The formal setup is small. Each user has a set of attributes with values; a memory generator turns them into memory statements; a task names a purpose and a recipient; the model writes a message from memory plus task. A REVEAL function reads the message and reports whether each attribute's exact value appears. A label oracle says, per task, which attributes should be withheld and which are needed (§3).

y ~ M(M_s · t)          message = model(memory prefix, then task)

The two metrics are deliberately asymmetric. Violation@n is a worst case per attribute: across every task where the attribute should stay out, and across n samples of each, did it ever appear? Average that over attributes that are private for at least one task (Definition 3.1). Completeness is an average case per task: what fraction of the needed attributes made it into the message (Definition 3.2).

The worst-case framing has a consequence worth internalising: if one sample leaks an attribute with probability p, n samples catch it with probability 1 − (1 − p)ⁿ, so rare leaks become visible as use grows. The authors also note why both scores are needed: a model that says nothing never violates, and a model that says everything is always complete, and both are useless. Deep dive: measuring it.

§4 The benchmark: synthetic people, real tasks

Each profile starts as a fake adult identity aged 21–70 from the Faker library. Three life events (drawn from lists of challenging ones such as bankruptcy or a pending legal case, and positive ones such as a promotion) and nine information domains seed GPT-OSS-120B, which writes seven attributes per domain per event, at most 189 per person (§4.1.1, Figure 11).

The contexts are 49 hand-picked recipient–task pairs: a loan officer for a personal loan, an ER intake desk, a landlord negotiating a lease, an immigration attorney (§4.1.2, Figure 12). The hard part is labelling every attribute–context pair. Humans disagree about privacy, so the authors have GPT-5 answer as each of the three Westin personas (fundamentalist, pragmatic, unconcerned), ten samples each, with an abstain option, then mix the answers using Westin's population priors. A pair gets a label only when the mixed distribution has zero entropy: everyone agrees it is necessary or everyone agrees it is inappropriate. Everything else is ambiguous and left out of the scores, which focuses the benchmark on clear-cut violations.

The last two rows are the shape of the problem: in a typical message, roughly seven remembered facts are wanted and over eighty must stay out (Table 2). Deep dive: building the benchmark.

§5.1 Setup: how the runs were done

Memories go in as a prefix using an OpenAI-style memory template, followed by a plain "write a message to this recipient for this purpose" directive (Figure 6). Each model produces n = 5 samples per task at its default sampling settings. DeepSeek-R1 acts as the judge, with a prompt that counts an attribute only if its entire value is stated explicitly (Figure 7). Open models ran on vLLM v0.10.1 across 8 H200 GPUs, and the 10-profile size was chosen to keep cost near $100 per model (§5.1). The strict judge matters when reading the results: partial or implied leaks are not counted. Deep dive: p04.

§5.2.1 RQ1: does any model get it right?

No. Every model in Table 1 leaks a substantial share of its at-risk attributes, and completeness sits between roughly 44% and 59%.

The two ends tell the story. GPT-4o leaks least but completes least; Qwen-3 32B is near the top on completeness and worst on leakage. Lower leakage mostly comes from saying less, not from choosing better. Figure 3's per-domain grid for GPT-5 shows why: when a task calls for a domain, the model pulls in the whole domain. Writing to a financial aid office, it shares 81.7% of the needed financial facts and also leaks unneeded financial facts at a 14.3% Violation@5 (Figure 3). The paper calls this a granularity failure and suggests helpfulness-maximising post-training as one possible cause. Table 3's excerpts make it tangible: a divorce case number sent to HR, a withheld $3,500 bonus mentioned to an ER intake desk (Table 3). Deep dive: frontier models, measured.

Figure 2: leaks add up with use

Figure 2 is the compositional result in one plot. For GPT-5, Violation@1 over a single task is 0.1%. Run the same user through 40 tasks and it is 9.6%. Sample each of those prompts five times and it is 25.1%, so about a quarter of the user's at-risk attributes surface somewhere they should not (Figure 2). Because the leaked attributes differ between identical prompts, the authors describe the behaviour as arbitrary and unstable. A single-task, single-sample evaluation would have reported almost no problem. Deep dive: the two compositions.

§5.2.2 RQ2: do the usual fixes work?

Three levers, three answers, all reported as trends in Figure 4 without printed values (§5.2.2).

The prompting result is the sharpest. More cautious wording (the Medium and High prompts in Figure 8) slides GPT-5 down a trade-off curve; the permissive one (labelled "Light" in Fig 4c, "Low" in Fig 8) slides it up. None moves it off the curve.

conservative language simply reduces overall verbosity of the model.— CIMemories, §5.2.2

That is the abstract's overgeneralization point: the model learns "share less" rather than "share this, not that", the same shape as the classic privacy–utility trade-off. Only test-time reasoning looked like a lever worth pulling. Deep dive: scale, reasoning and prompts.

§5.2.3 RQ3: does more memory hurt?

Yes. On a 5-profile subset with GPT-5, the authors hold the needed attributes fixed and add inappropriate ones to memory starting from zero. Violations climb steadily while completeness stays flat (Figure 5, §5.2.3). The extra facts buy nothing for the task and only add risk, so an assistant that remembers more over months of use should be expected to leak more, not less. Deep dive: p05.

§6 Discussion: examples, mitigations, limits

The discussion is short (§6). On mitigations, the authors judge bigger models and more elaborate prompts not viable, and see test-time reasoning as more plausible. They also suggest post-training rewards that penalise contextual integrity violations, and system-level, domain-specific guardrails at inference time. On limits, they name the synthetic profiles and the restriction to single-turn, no-tool interactions. A careful reader adds more: only 10 profiles, labels from one model playing three personas, a judge that only counts exact explicit values, and one memory format (full prefix, not retrieval). The map collects those, clearly marked as the map's reading, in limitations and caveats; practical lessons are in for builders.

§7 Conclusion

The conclusion restates the contribution: a contextual-integrity benchmark for memory-augmented assistants, two metrics that must be read together, and a synthetic pipeline that can scale. The finding it leaves behind is that violations are large, trading them down costs completeness, they accumulate with use, and the standard fixes of scale and prompting do not remove them (§7).

Appendices A and B: the prompts and seeds

The appendices are short but load-bearing. Appendix A holds the memory-and-task template (Fig 6), the judge prompt (Fig 7), the three privacy prompts (Fig 8), the profile-generation prompt (Fig 9) and the persona labelling prompts (Fig 10). Appendix B lists the event and domain seeds (Fig 11) and the 49 context seeds (Fig 12) (Appendix A, Appendix B). Reading the judge prompt next to the results is the quickest way to see how conservative the violation counts are. Code, raw profiles and the label generator are in the facebookresearch/CIMemories repo; the dataset is on Hugging Face (CC BY-NC 4.0).

What to remember

• Privacy is per flow, not per fact. The same attribute is necessary for one recipient and a violation for another, so a static sensitivity flag cannot be the control (Fig 1).
• Every model leaks, and less leakage mostly means less help. Violation@5 runs 14.82%–69.14% with completeness around half (Table 1).
• Exposure compounds. GPT-5 goes 0.1% → 9.6% → 25.1% as tasks and samples grow; single-shot evals understate the risk (Fig 2).
• Models pick the right domain and over-share inside it. The granularity failure (Fig 3).
• Scale saturates, prompts turn the volume down, reasoning helps some, and more memory means more leaks (Fig 4, Fig 5).

Check yourself

1. Why can GPT-4o have the lowest Violation@5 in Table 1 and still not be the model to copy?
   Answer

   Its completeness is also the lowest (43.95%). Leaking little because it says little is the degenerate "reveal nothing" direction that §3 warns about; the two metrics have to be read together.

2. Why does Violation@n rise when the same prompt is sampled more times?
   Answer

   It counts an attribute as violated if any sample, in any private task, reveals it. A per-sample leak probability p becomes 1 − (1 − p)ⁿ, which is why GPT-5 goes from 9.6% at one sample to 25.1% at five (Fig 2).

3. What did the privacy prompts change, and what did they fail to change?
   Answer

   They moved GPT-5 along the violation–completeness trade-off (fewer leaks, less needed detail, or the reverse) but did not improve the trade-off itself; the paper reads this as the model becoming less verbose overall (§5.2.2, Fig 4c).

Further reading, verified

• CIMemories, arXiv abstract page (v1 is the version mapped here; PDF)
• facebookresearch/CIMemories code and data (its README attributes the 0.1% → 9.6% → 25.1% curve to GPT-4o; the paper says GPT-5, and the paper is followed here)
• ConfAide, "Can LLMs Keep a Secret?" (Mireshghallah et al.), the earlier CI benchmark cited in §2
• CI-Bench (Cheng et al.), cited in §2
• Agentic privacy field map, a sibling map of "Open and Emergent Problems in Agentic Privacy and Security: A Contextual Angle" (field map extra)
• Glossary for every term linked above