CIMEMORIES // FIELD MAP
← field map
PART 09 · WHAT IT MEANStakeaways grounded in §3 · §5 · §6 · Tables 1–3 · Figs 2–5

For builders: shipping a memory-enabled assistant

Seven engineering lessons, each pinned to a result in the paper
PaperCIMemories, §6 — Discussion (mitigations, limitations) ↗
TL;DR — If an assistant writes messages to other people from a long memory of its user, CIMemories says it will put the wrong facts in front of the wrong recipients, and that a single-shot eval will hide most of it. Track leakage and task completeness together, sample many tasks and many draws, and do not count a "be careful" system prompt as a privacy control. Filter at the level of individual remembered facts per recipient, keep memory scoped, and use the released benchmark as a regression test. These are the map's takeaways; each one points at the result that supports it.
GPT-5 leakage, 1 task
0.1%
Figure 2
after 40 tasks
9.6%
Figure 2
40 tasks × 5 samples
25.1%
Figure 2
to-share vs not, per context
6.7 / 83.7
Table 2 (means)

The rest of the map explains what the paper measured. This page asks what someone shipping a persistent memory feature should do differently because of it. The paper itself offers only a short list of mitigations in §6; everything else here is an engineering reading of its results, labelled as such. The setting it applies to is specific: a model that holds many memory statements about a user in its prompt and drafts a message for a third-party recipient. That is the setting of email drafting, auto-replies and form filling, and it is where the benchmark is strongest.

1. Ship two numbers, never one

The paper defines two metrics on purpose (§3, Def. 3.2). Violation@n asks whether each should-be-withheld attribute ever leaks; Completeness asks whether the facts a task needs actually make it into the message. Either one alone is trivially gamed: an assistant that never mentions anything personal scores a perfect violation rate and is useless, and one that pastes the whole memory scores perfect completeness and is a liability.

Table 1 shows that this is not a hypothetical. The model with the lowest leakage is also the one that completes least, and the reverse holds at the other end:

Model (Table 1)Violation@5 ↓Completeness ↑
GPT-4o14.82%43.95%
GPT-525.08%56.61%
Claude-4 Sonnet44.44%59.07%
Qwen-3 32B69.14%57.63%

Four of the eight rows, copied exactly; the full table is on Part 06. The practical consequence: when someone reports that a model change "reduced leakage", the first question is what happened to completeness. A dashboard that shows one without the other will reward whichever change makes the assistant say less. Read the pair as a point on a privacy–utility trade-off, and look for changes that move the point down without moving it left.

2. Single-shot evals understate exposure

The most useful single figure for an eval team is Figure 2. With one task and one sample, GPT-5 leaks 0.1% of a user's should-be-private attributes. Spread across 40 tasks it reaches 9.6%, and with five samples per task it reaches 25.1%, about a quarter of the attributes. The model did not get worse between those points; the measurement got closer to how a product is used.

Two things compound. Over many tasks, every attribute meets many recipients for whom it is inappropriate, and Violation@n takes the worst case over all of them (Def. 3.1). Over many samples, sampling noise does the rest: the abstract describes models leaking different attributes on identical prompts. The paper states the arithmetic: if one draw reveals an attribute with probability p, n draws catch it with probability 1−(1−p)n.

illustrative, not from the paper:
p = 0.05 per draw (a leak you'd call "rare")
n = 5   draws  -> 1 - 0.95^5  = 0.226   (about 1 in 4.4)
n = 20  draws  -> 1 - 0.95^20 = 0.642

A production assistant draws far more than five times per user. So the eval design follows: build a task suite that spans many recipients per user, sample each task several times at the temperature you ship with, and report the max-over-draws number, not the mean. A per-response leak rate measured once, on a handful of prompts, is the 0.1% end of Figure 2. More on the mechanics in Part 05.

3. A "be private" prompt is a dial, not a control

The obvious first fix is a system-prompt line telling the model to share only what is needed. The authors tried three levels of that on GPT-5 (§5.2.2, Fig. 4c, prompts in Fig. 8). Stricter wording did lower violations, but completeness fell with it, and the permissive wording moved both up. Every prompt landed on the same trade-off curve rather than below it.

conservative language simply reduces overall verbosity of the model.— CIMemories, §5.2.2

That is the overgeneralization the abstract describes as sharing everything or nothing: the instruction changes how much the model says, not whether it can tell which facts belong to which recipient. For a builder this means a privacy prompt is a tuning knob for where on the curve you sit. It is worth setting deliberately, but it cannot be the thing you point to when asked how leakage is prevented. §6 reaches the same verdict and lists prompting, along with model size, as not viable on its own. Part 07 has the exact prompt wording.

4. Filter facts, not topics

Figure 3 breaks GPT-5's results down by information domain. For a message to a financial aid office it shares most of the financial facts the task needs (81.7% completeness in finance) and also leaks financial facts the office should not get (14.3% violation in finance). The paper calls this a granularity failure: the model picks the right drawer, then empties it. The authors suggest helpfulness-oriented post-training may encourage this, since dumping everything is an easy way to look helpful.

Table 3 shows what that looks like in output: a medication dosage change appears in a note to a financial aid office, and a divorce case number appears in a benefits update to HR. Each is "on topic" for the recipient in a loose sense and still clearly out of place.

So a guard that works at the topic level ("finance is allowed for the loan officer") would pass exactly these leaks. Whatever filtering a system does has to operate on individual remembered facts, conditioned on the recipient and the purpose. One place it could sit, as the map's sketch rather than anything the paper built:

1 · scopeIdentify the recipient and purpose of the draft before any memory is loaded.
2 · selectPass only the facts judged necessary for that pair; hold the rest back from the prompt.
3 · draftGenerate the message from the reduced memory.
4 · checkRun a reveal judge on the draft and flag any withheld fact that still appears.

Step 4 is the benchmark's own REVEAL judge (Fig. 7) turned into a runtime check. Step 2 is the hard part, because it needs the same contextual judgment the models lack, but it can be a separate, narrower decision ("does a landlord need this one fact?") rather than something left implicit in a long generation. This is what §6 means by system-level, domain-specific inference-time guardrails.

5. More memory, more leakage: scope it

Figure 5 holds a user's necessary facts fixed and adds inappropriate ones. Violations climb steadily and completeness stays flat (§5.2.3). Extra memory bought no extra usefulness for those tasks and only added risk. The paper's reading is that growing personalization is itself in tension with contextual integrity.

Table 2 shows how lopsided the ratio already is: per context, an average of 6.7 attributes should be shared against 83.7 that should not. Almost everything in memory is irrelevant to any one message. That argues for ordinary data minimization applied to memory: expire or summarize facts whose usefulness has passed, keep sensitive event details (a court case number, a dosage date) out of the general pool unless a task needs them, and prefer loading a scoped subset over prefixing the full memory. The benchmark only tests the full-prefix design (§1), so whether retrieval-based memory fares better is untested here, not established.

6. Reasoning helped; size and post-training are the longer levers

Of the knobs the authors turned (Fig. 4), only one improved the trade-off rather than sliding along it. Comparing the Instruct and Reasoning variants of Qwen-3 30B with everything else fixed, reasoning cut violations with negligible loss of completeness (Fig. 4b). Making the Qwen-3 model larger helped at first and then saturated (Fig. 4a). §6 accordingly calls test-time reasoning the more plausible path, and adds two more: post-training whose rewards penalize contextual-integrity violations, and the system-level guardrails above.

For a team that cannot retrain a model, that reads as: if you have a choice of model mode for drafting messages from memory, test the reasoning variant on both metrics. Treat a larger model as an improvement that will run out. For a team that does post-train, the benchmark's labels are a ready source of reward signal for exactly the behavior Figure 3 shows going wrong.

7. Use the open benchmark as a regression test

The code is at facebookresearch/CIMemories and the data at facebook/CIMemories on Hugging Face (CC BY-NC 4.0, so check the non-commercial terms before building on it). The repo generates the labels, runs a model against the profiles with a DeepSeek-R1-0528 judge, and prints violation and completeness per profile. §5.1 puts a run at about $100 per model on 10 profiles, which is cheap enough to run on every candidate model or memory-format change.

Two cautions when reading your own numbers, both covered in Part 08. The judge only counts a fact as revealed if its entire value appears explicitly, so partial or implied leaks are not counted. And labels were kept only where all three Westin personas agreed, so ambiguous pairs are dropped. Both choices make the benchmark a floor on clear-cut leaks, not a full audit. A passing score means the obvious failures are under control, nothing more. The benchmark is also single-turn with no tools; agents that browse or call APIs need their own tests, such as AgentDAM, which the paper cites.

The leak rate a user experiences is set by how many tasks and draws the assistant runs, not by the rate on one prompt, and a privacy prompt only trades usefulness for silence. Measure both metrics at realistic volume and put the control in the pipeline, at the level of individual facts per recipient.

For the wider agent setting (tools, multi-step actions, other parties' data), see the sibling map on open problems in agentic privacy and security.

Check yourself

  1. A team reports that adding "only share what is strictly necessary" to the system prompt cut leakage by a third. What should you ask before accepting it as a fix?
    Answer

    What happened to completeness. In Fig. 4c, stricter prompts on GPT-5 lowered violations and completeness together; §5.2.2 attributes the drop to the model saying less overall. A real fix moves violations down without pulling completeness down with it.

  2. Your offline eval runs each of 10 prompts once and sees almost no leaks. Why does Figure 2 suggest this is misleading?
    Answer

    GPT-5 shows 0.1% at one task and one sample, but 9.6% across 40 tasks and 25.1% with five samples per task. Violation@n is a worst case over tasks and draws, and users generate many of both, so exposure grows as 1−(1−p)n.

  3. Why would a guardrail that allows "finance information" for a financial aid office still leak?
    Answer

    Fig. 3 shows the granularity failure: GPT-5 shared 81.7% of the needed financial facts to that office but also leaked unneeded financial facts (14.3% violation). Topic-level permission passes those; filtering has to be per fact and per recipient.

Further reading, verified

Next: Read the paper in 10 minutes · Back to the map.