MIRESHGHALLAH, MANGAOKAR ET AL. · ARXIV:2511.14937 · FAIR AT META10 profiles · ~147 attributes each · 8 models · 11 pages here

A worried plush raccoon in a MEMORY t-shirt writes a letter at a desk while an overflowing notebook of memories spills sticky notes beside it, some sliding toward the letter. — A memory that only grows: every remembered fact sits beside every letter the assistant writes.

CIMemories, mapped

An assistant that remembers you is useful because it can fill in the details you would otherwise retype. It is risky for the same reason: each detail is right for some recipients and wrong for others. CIMemories is a benchmark that measures whether a model with a long memory of you shares the right things with the right recipient. These pages walk through the paper section by section.

The one-paragraph answer. The authors give eight models a memory of about 147 synthetic facts per user and ask each to write messages to dozens of recipients, such as a loan officer, an ER nurse, a landlord or HR. A judge model checks which remembered facts ended up in each message. Every model leaks: across five samples per task, the share of a user's should-be-private attributes that show up somewhere they should not ranges from 14.8% (GPT-4o) to 69.1% (Qwen-3 32B) (Table 1). The models that leak least also complete the least. Leaks add up with use: GPT-5 goes from 0.1% after one task to 9.6% after 40, and to 25.1% when each prompt is sampled five times (Figure 2). Telling the model to be careful just makes it say less overall, so leaks and useful detail fall together.

Short on time

Terms

The numbers

Building one?

Tick a page when you have read it; this browser remembers.

01 · THE IDEA

Privacy as appropriate flow

The paper's footing: privacy means information going to the right place, not information kept secret. A remembered fact is not public or private on its own; whether sharing it is a violation depends on where it goes and why.

Contextual integrity, from Nissenbaum to LLMs

§1 · §2Nissenbaum 2004 · Barth et al. 2006ConfAide · PrivacyLens · CI-Bench

The theory in plain terms: an information flow has a sender, a recipient, a subject, a type of information and a norm that governs it. Also how earlier LLM benchmarks used the theory, and why "one secret to keep" was too easy a test.

Why persistent memory makes it hard

§1 · §2 · §3 problem setting · Fig. 6memories prefixed to the prompt

Memory turns every conversation into a dossier the model writes from. The same fact (income, a diagnosis, a court case) is essential for one recipient and out of place for the next, and the pile keeps growing.

02 · THE BENCHMARK

How CIMemories is built and scored

Synthetic people, real-world tasks, and labels drawn from three privacy personas. Two scores are always reported together, because either one alone can be maxed out by a useless model.

Building the benchmark: profiles, contexts, labels

§4.1 · Table 2 · Figs. 9–12146.7 attributes · 45.7 contexts per profile

Faker identities, three life events and nine domains per person, GPT-OSS-120B writing the memory statements, 49 seed tasks, and GPT-5 labelling every attribute–task pair as each of Westin's three privacy personas. A pair keeps a label only when every persona agrees.

Measuring it: reveal, Violation@n, Completeness

§3 · Defs. 3.1–3.2 · Fig. 7 · §5.1DeepSeek-R1 as the judge

How a reveal is detected, why violation is a worst case per attribute (did it leak ever, across tasks and samples) while completeness is an average per task, and why the 1 − (1 − p)ⁿ arithmetic matters.

The two compositions: memory and tasks

Fig. 1 · Fig. 2 · Fig. 5 · §5.2.30.1% → 9.6% → 25.1%

What makes the benchmark "compositional": the authors can choose which attributes are in memory and how many tasks a user runs. The more tasks and samples, the more leaks accumulate. Adding inappropriate memories raises violations while completeness stays flat.

03 · THE FINDINGS

What frontier models actually do

Every model leaks. Scaling helps at first, then levels off. Reasoning helps somewhat. Privacy prompts turn the volume down on everything at once instead of making the model more selective.

Frontier models, measured

§5.2.1 · Table 1 · Fig. 3 · Table 314.8%–69.1% Violation@5

All eight models side by side, the violation–completeness tension, the "granularity failure" (the model picks the right domain but shares every detail in it), and the concrete leaks: a divorce case number sent to HR, a withheld bonus sent to an ER intake desk.

Scale, reasoning, and why privacy prompts overgeneralize

§5.2.2 · Fig. 4a–c · Fig. 8Qwen-3 1.7B–32B · 30B Instruct vs Reasoning

Three fixes and how far each gets. A bigger model saturates. A reasoning chain cuts violations at little cost to completeness. Three levels of "be careful" wording move GPT-5 along a fixed trade-off line instead of moving the line.

04 · WHAT IT MEANS

Limits, and lessons for builders

What the benchmark does not test, and what follows for anyone building an assistant that remembers.

Limitations, caveats and mitigations

§6 · §7synthetic profiles · single-turn · no tools

The paper's stated limits plus the ones a careful reader adds: 10 profiles, model-generated labels, a strict judge that only counts fully explicit reveals, and one memory format. Then the mitigations the authors point to.

For builders: shipping a memory-enabled assistant

takeaways grounded in §5–§6

What to measure, why a single-sample eval underestimates leakage, why "be private" in the system prompt is not a control, and where recipient-aware filtering could sit in the pipeline.

Reference and sources

Read the paper in 10 minutesThe whole argument on one page, section by section, with every headline number cited GlossaryContextual integrity, Violation@n, completeness, REVEAL, Westin personas, granularity failure and the rest arXiv:2511.14937Mireshghallah, Mangaokar, Kokhlikyan, Zharmagambetov, Zaheer, Mahloujifar, Chaudhuri, CIMemories (v1, 18 Nov 2025). This map is written against v1. The paper, HTMLarXiv's HTML rendering; section, figure and table links on these pages jump into it facebookresearch/CIMemoriesCode: label generation, eval runner, metrics, the raw synthetic profiles (CC BY-NC 4.0) facebook/CIMemories on Hugging FaceThe dataset release ICLR 2026 versionPublished as "CIMemories: A Compositional Benchmark for Contextual Integrity in LLMs"; this map was not checked against the camera-ready