Frontier models, measured

Parts 03–05 built the instrument: synthetic people with about 147 remembered facts each, dozens of recipients, a label on every fact–recipient pair, and two scores. This page is where the instrument is pointed at real models. It answers the paper's first research question, "do frontier LLMs respect the contextual integrity of user memories?", and the answer is no. The more useful lesson is in the shape of the failure. Models do not leak at random. They lean toward oversharing, and when they are tuned to share less they lose useful detail along with the leaks.

Table 1, as printed

Each model gets the same 10 profiles, each profile's memories prefixed to every task prompt, five samples per task (§5.1). The two columns are the two metrics defined in Part 04. Violation@5 (lower is better) is the share of a user's sometimes-private attributes that show up at least once where they should not, across every task and all five samples. Completeness (higher is better) is the average share of a task's necessary attributes that the message actually contains. Values are copied exactly, in the paper's row order and spelling:

Source: Table 1, "across 10 CIMemories user profiles". "Mistal" is the paper's typo for Mistral-7B Instruct v0.3 (Jiang et al. 2023). The prose in §5.2.1 rounds these to 14%, 43%, 57% and 69%; the introduction gives 14.8%, 43.9%, 57.6% and 69.1%. All three describe the same four cells.

Two things about the scale before reading the rows. First, Violation@5 is a worst case per attribute. One leak in one task in one of five samples counts the attribute as violated, so a quarter means a quarter of a user's sensitive facts escaped at least once over a full session of tasks, not that a quarter of messages leak. GPT-5's 25.08% matches the right edge of Figure 2 (25.1% at five trials), which shows how the number builds up task by task (Part 05). Second, the judge only counts a reveal when the full value is stated explicitly (Fig 7), so paraphrased or hinted leaks do not count. Both scores are best read as conservative.

The trade-off, read off the table

The authors' summary is that completeness "appears to be at odds with violations for most models." The two endpoints make the case. GPT-4o has by far the lowest violation rate, 14.82%, and also the lowest completeness, 43.95%. Qwen-3 32B is close to the top on completeness at 57.63% and leaks the most at 69.14%. That is the privacy–utility trade-off in its familiar form: a model that says less leaks less and also helps less.

The table is not a single tidy line, though. Treat each model as a point (violation, completeness) and ask which ones no other model beats on both axes at once. This is the map's reading of Table 1, not an analysis the paper runs.

Three models sit on the frontier. Moving from GPT-4o to GPT-5 buys about 12.7 points of completeness for about 10.3 points of extra violation. Moving on to Claude-4 Sonnet buys only about 2.5 more points of completeness for about 19.4 more points of violation (the differences are computed here from Table 1). So the trade-off is real at the frontier, and below it some models are simply worse on both counts. Mistral-7B is the clearest case: high leakage and low completeness together. The paper's word "most" leaves room for this.

Reasoning and non-reasoning models fall on both sides of these lines. o3, a reasoning model, is beaten on both axes by GPT-5. The controlled comparison of reasoning against no reasoning, with everything else held fixed, is in Part 07. Table 1 alone cannot separate reasoning from all the other differences between vendors.

Why completeness stalls near half

No model reaches 60% completeness. The range is 43.95% to 59.07%. The paper calls this "moderate completeness of around ∼50%" and connects it to Jiang et al. 2025 (PersonaMem, "Know Me, Respond to Me"), which found that models recall user facts and preferences from long histories only partly when they need them for a task (§5.2.1).

The setup explains why this matters. Per Table 2, a context has on average 6.7 ± 5.5 attributes to share and 83.7 ± 31.5 not to share, all in the same prefix of about 147 memory statements. The model's job is to find the handful that matter among close to a hundred that do not. Missing half of the useful ones while also letting through some of the harmful ones is not two separate faults. It is one weakness, poor selection, showing up on both scores. That is why the two scores have to be read together. A model with low completeness and low violation may not be cautious at all; it may simply use its memory less.

The granularity failure (Figure 3)

Figure 3 breaks GPT-5's two scores into a grid: rows are example tasks such as "Document Change In Financial Circumstances (Financial Aid Office)" or "Appeal Denied Claim (Health Insurance Coordinator)", and columns are the nine information domains, with completeness in one panel and Violation@5 in the other. The caption names the standard example. For the financial aid office, GPT-5 shares 81.7% of the necessary finance attributes and also has a 14.3% Violation@5 on finance attributes labelled inappropriate for that office. (The caption calls the completeness number "coverage"; it is the same metric.)

The authors call this a granularity failure. The model's first decision, which part of the user's life is relevant to this recipient, is mostly right. The second decision, which facts within that part, is where it goes wrong. A financial aid office needs to know that income dropped. It does not need every figure in the user's finances. The model treats "finance is relevant here" as permission to share everything in finance. This is a useful way to think about the problem because the mistake is not a misreading of the recipient. It is the absence of a filter below the domain level.

models can identify the right information domain to complete the task, but fail to discern between necessary and unnecessary information within that domain.— CIMemories, §5.2.1

Figure 3 prints a number in every cell, but this map quotes only the two the caption states. The figure in the HTML version is an image, so the cell-to-column assignment of the other values could not be checked against a second source.

Why would a model overshare? The reward-hacking hypothesis

The paper offers one explanation and labels it "one possible reason". Post-training rewards helpfulness, and the easiest way to look maximally helpful in a message to a loan officer or a doctor is to include everything you know that might be relevant. Under this view, oversharing is a form of reward hacking: the model has learned that thoroughness scores well, and nothing in its training penalized sending the wrong recipient a correct fact.

The paper does not test this. No experiment varies the post-training objective. It fits the evidence on this page: oversharing concentrates in domains the model has already judged relevant, which is where a thoroughness heuristic would act. It also explains why one of the mitigations in §6 is post-training with rewards that penalize contextual-integrity violations. If the incentive caused the problem, changing the incentive is the direct fix. Part 08 covers the mitigations.

What a leak looks like (Table 3)

Table 3 shows five violations from GPT-5 and Qwen-3 32B. The caption covers both models and does not say which model wrote which row. Here they are in one line each:

Two patterns stand out. First, the leaks are precise: identifiers, dates and amounts, exactly what the strict judge is built to catch. Second, not all of them are within-domain granularity failures. The blood-pressure dose sent to a financial aid office and the weight loss sent to a loan officer cross domains completely. A plausible reading (the map's, not the paper's) is that the model is building a story such as "my circumstances changed" or "my life is on track" and recruits any memory that supports it, whatever its domain. The ER row fits the same reading: the stressors become a list of money problems. §6 describes this example as "disclosing exact paycheck details to the Emergency Room", which is a loose summary of the excerpt. What the excerpt shows is a withheld bonus, legal fees and lost overtime hours.

Check yourself

1. GPT-4o has the lowest Violation@5 in Table 1. Why is that not enough to call it the most privacy-respecting model?
   Answer

   It also has the lowest completeness, 43.95%. Section 3 warns that either metric alone admits a degenerate assistant: a model that shares nothing scores perfectly on violation. GPT-4o's low leak rate comes partly from saying less overall, so it sits at one end of the trade-off rather than beating it.

2. In Figure 3, what pattern in a single row would signal a granularity failure?
   Answer

   High completeness and non-trivial Violation@5 in the same domain for the same task. For example, GPT-5 writing to the financial aid office shares 81.7% of necessary finance attributes and leaks unnecessary finance attributes at 14.3%. The domain choice was right; the selection within it was not.

3. Which Table 3 rows do not fit the within-domain granularity story, and what does that suggest?
   Answer

   The medication dose sent to the financial aid office and the weight loss sent to the loan officer. Both are health facts sent to finance recipients. Leakage is therefore not limited to oversharing inside a relevant domain; facts from unrelated domains can also be pulled in when they fit the story the message is telling.

Further reading, verified

• CIMemories §5.2.1 — RQ1 results, with Table 1 and Figure 3.
• CIMemories Table 3 and §6 Discussion — the violation excerpts and the mitigations they motivate.
• CIMemories §5.1 — setup behind every number here: 10 profiles, n = 5, the DeepSeek-R1 judge.
• Jiang et al. 2025, "Know Me, Respond to Me" (PersonaMem) — cited for the roughly 50% task recall of user facts.
• facebookresearch/CIMemories, eval/metrics.py — how Violation@n and completeness are computed in the code release (field map extra).