Evaluations and benchmarks

Part II of the report looked at the pieces of an agent: the policy engine, the system guardrails, the model's reasoning, the user interface. Part III steps back to deployment, and its first question is measurement (§10). Every earlier open problem ends in some version of "and then we need to show it works." This section says the tools for showing that are weak in general for frontier agents (§10.1), surveys what CI-specific benchmarks exist (§10.2 and Appendix D), and then lays out what is missing: real-user studies (§10.3) and evaluation over long time horizons (§10.4). The report's summary of open problems frames the goal as measuring whether complex behavior is appropriate, continuously and across long tasks (§1.2).

Six general difficulties

Before getting to privacy, §10.1 lists six reasons benchmarking frontier agents is hard at all (pp. 60–61). None is new on its own. The point is that they pile up for agents.

The fifth item names the tension the whole section keeps returning to:

"benchmarks must be sufficiently controlled to yield reproducible measurements, yet sufficiently realistic to avoid cuing the model into a distinct “evaluation mode.”"— the report, §10.1, p. 61

Our reading: for privacy this tension is worse than for coding or math. A privacy test usually has to stage a sensitive situation, and staged sensitive situations are exactly what a test-aware model is likely to recognize.

What 16 CI benchmarks cover

Appendix D is the most concrete part of this section. The report first builds an analysis framework of ten features (§D.1, Table 1, pp. 106–108), then scores 16 benchmarks on all of them (§D.2, Table 2, pp. 108–116). The features read like a checklist for anyone building such a benchmark:

• Contextualization (none / simple / rich): does the scenario give purpose, roles or detailed personas, so the same data can mean different things in different scenarios?
• Transmission principles (none / simple / complex): only "is this necessary or appropriate," or conditions such as an NDA, doctor-patient confidentiality, a subpoena, a commercial exchange or reciprocity?
• Ambiguity (no / implicit / explicit): can a flow be fine in one context and wrong in another, and is that designed in or only found later?
• Norms: legal, societal, organizational, other, or not specified.
• Realism: synthetic, grounded in real data, real, other, or not specified.
• Annotation: none, synthetic, non-expert raters, expert raters.
• Explanation: do labels come with a justification that a model's reasoning can be checked against?
• Metrics: leakage and/or utility, scored deterministically, heuristically or by an LLM judge.
• Granularity: is there a taxonomy for fine-grained error analysis?
• Modality: text, image, video, audio, multi-turn, tool use.

Below are five of those columns for all 16 benchmarks, taken from Table 2 (pp. 115–116).

A few things stand out in the prose of §D.2. Only PrivaCI-Bench uses real data, drawn from court judgments, regulatory enforcement trackers and published privacy policies; the report suggests real data is rare because of the risk of handling sensitive human data. No benchmark covers video or audio. Image appears only in AgentDAM, MPCI-Bench and SPILLage. Where expert raters appear, they are often the benchmark's own authors, and the report says plainly that calling them independent experts is debatable. Benchmarks also differ in what they treat as a norm: ConfAIde and CIMemories draw societal norms from literature and surveys, PrivaCI-Bench from law, ConVerse from the task itself.

CIMemories, the memory benchmark in row 12, has its own explainer: the CIMemories field map. The table shows where it sits among the 16: simple context, simple transmission principles, no designed ambiguity, synthetic data, text only.

The three limitations (§10.2)

The report boils the appendix down to three gaps across current benchmarks (§10.2, p. 62):

1. Transmission principles and ambiguity are simplified. Apart from the Camber framework (the work behind PrivacyLens+ and ConfAIde+), the report says most benchmarks filter out ambiguous edge cases to keep the ground truth deterministic.
2. Personas and annotations are synthetic. User profiles come from frontier LLMs, or crowd raters judge fictional vignettes. Neither is a real person working with a real agent.
3. CI parameters are static, single-session snapshots. Even the multi-turn agent benchmarks the report names (PrivacyLens-Live, AgentDAM, PrivacyBench) stop at one task or at most ten conversational turns. None tracks persistent context, cumulative leakage or norms that shift over years.

Our reading: limitation 1 is phrased more strongly than Table 2 alone suggests. By our count the table marks 8 of the 16 as explicit ambiguity and 2 as implicit. The two fit together if "explicit" often means a designed trap with one right answer, while Camber is the work that makes underspecified context its central subject. The table also lists AirGapAgent as "No" ambiguity while the D.2 prose describes its ambiguity as implicit. Treat individual cells as the authors' coding, not settled fact.

OP 10.1 · Why red-teaming stopped finding attacks

The question. Automated red-teaming against leading models now finds few or no new attacks. Is that because the models really are robust, because the test environments are too simple, because the models know they are being tested, or because the attack methods have fallen behind? (§10.1, p. 61)

Background. The report recalls that strong adaptive attacks have always been how empirical defenses get tested. In the adversarial-examples era they regularly broke proposed defenses. Between 2024 and 2026, it says, setting up strong automated attacks inside agent environments became standard practice. Then:

"This standard practice started to be marginalized in mid-2026: frontier labs like Anthropic deprecated automated red-teaming, citing limited benefits in discovering new ways to break the agent"— the report, §10.1, pp. 61–62, citing Anthropic (2026c)

The citation, Anthropic (2026c), is the "Introducing Claude Sonnet 5" announcement (June 2026) in the reference list. The report then offers two readings: the apparent claim that new models understand Contextual Security well enough to withstand the strongest automated attacks, and the alternative that rising test-awareness explains the silence. It says which is correct is an open question. Our check: on the announcement page as we fetched it, we did not find an explicit statement deprecating automated red-teaming, so read this as the report's characterization of that source.

Why it matters for agents. If the null result means robustness, pre-deployment forecasts are fine. If it means the tests are blind, every safety claim that rests on them is weaker than it looks. The report says the answer will matter a great deal for forecasting a model's Contextual Security before it ships.

What a good answer needs. A way to tell these four explanations apart, for example by varying environment realism and evaluation cues while holding the attack fixed. That is our illustration, not a method the report proposes.

OP 10.2 · Large-scale studies of real users

The gap. No comprehensive study yet measures how people perceive agents doing real tasks on their behalf (§10.3, p. 62). Real traces are rare, environments change fast, and perhaps only a subset of daily use runs outside robust sandboxes and without monitored tool use. So current studies rely on fictional scenarios, privacy seeds and common-sense judgments.

The questions OP 10.2 lists (pp. 62–63):

• Focus. Concentrate on settings with conflicting norms, misunderstandings and ambiguity.
• Ground truth. How to handle disagreement between users, cultural differences, users unsure of their own preferences, the privacy paradox, and users who do not understand the risks. How to combine observed behavior with self-reports.
• Study design. Capture human-agent vs agent-agent interaction, solo vs multi-person use, and different time scales, while balancing ecological validity, control and ethics.
• Simulation. Can synthetic users and attackers stand in for real ones without losing real uncertainty?
• Necessity. What do user studies add over shadow and A/B testing?

What the report points to. HCI has answered many of these in non-agent settings through interviews and controlled experiments; the report cites Martin & Nissenbaum (2017). The open direction is how those findings transfer, and how to build human-centered privacy research, with real and simulated users, that scales and replicates fast enough to keep up.

OP 10.3 · Evaluating long-term memory

§10.4 opens the long-horizon theme. CI's strength is that norms, actors and contexts change over time, yet benchmarks test frozen snapshots.

"As frontier agents transition into lifelong assistants, this dynamism is no longer a theoretical edge case but an aspect that demands adequate evaluation."— the report, §10.4, p. 63

The question. When an agent loads history from years of sessions, does it avoid surfacing "stale" information that conflicts with the user's current preferences or current norms? (The report points to §8.3.)

Why it is hard. There is no unified long-term memory mechanism, and the field moves fast. A benchmark tied to one memory design is obsolete when the design changes.

What the report proposes as directions. Benchmarks that test context-aware, time-aware storage and retrieval; that check whether inappropriate data is filtered before it enters the context window; and that treat personal memory as versioned, subject to revision, decay and re-contextualization. Above all, datasets in modular, interoperable formats that encode time relationships between CI parameters and stay independent of any agent's memory architecture.

OP 10.4 · Leakage that accumulates

The question. Benchmarks check one run at a time. Nobody can yet measure how privacy loss spreads across many interactions over an agent's life (§10.4, p. 64).

Two settings. Non-adversarial: each run respects the norm, but the sum does not. Adversarial: an attacker never tries a single exfiltration and instead rebuilds identity or secrets from long observation of agent traces, as in linkage or attribute inference.

What exists. The report cites SEABench (Das et al., 2026), which runs self-evolving agents through sequences where earlier interactions change persistent state and later tasks check whether that state is still safe. It calls this a useful starting point. Still missing: realistic methods and standard metrics for how much privacy each interaction costs, and for the "rate of leakage" across flows.

A direction the authors suggest. A benchmark where the attacker assembles a "shadow profile" from hundreds of non-sensitive interactions, including inferential leaps where the agent confirms something the user never meant to share. Defenses to test against it could include abstraction layers, added noise and anonymization.

OP 10.5 · Privacy-utility trade-offs that move over time

The question. Benchmarks assume that what is sensitive today stays sensitive. CI says otherwise, and the report gives a vivid case: a traumatic event from youth may need strict filtering in early adulthood and later become a "bonding story" the person wants told (p. 64). Needs also differ by life stage, a teenager's need for forgetting versus a professional's need for archiving.

Why it is hard. Both errors cost something. Over-protecting old data makes a sterile agent without personal depth. Under-protecting it can cause reputational or psychological harm. The report says there is no standard way to find the right balance as it moves over a 10- or 20-year horizon, and current models have no framework for adjusting retrieval as the user's identity changes.

Direction. Datasets with "contextual metadata" that let a test check whether an agent notices a topic moving from off-limits to shareable, across simulated decades of a user's life. The report notes this means moving from objective accuracy toward subjective alignment, which is harder to score.

OP 10.6 · Evaluations that carry across environments

The question. CI benchmarks are welded to their harness: a QA format with tiered vignettes, a particular chat-and-email interface, a particular web-form schema (pp. 64–65). Change the agent framework or the benchmark's infrastructure and the measurement can silently break, though the privacy question has not changed. It is hard to separate what is tested (CI compliance) from how it is run. So benchmarks get rebuilt from scratch and always lag the deployed system.

Why it is hard for agents. Redeployment changes the CI parameter space itself. New tools bring new senders and recipients, new environments bring new information types, and new protocols bring new transmission principles. Long multi-step plans across environments grow the space of possible violations combinatorially. Self-evolving agents may even change their own tools or how they summarize their context (the report cites Kontonis et al., 2026).

What a good answer would do. Our reading: express a scenario in terms of CI parameters and expected flows, and keep harness adapters separate, the way OP 10.3 asks memory datasets to stay independent of memory architecture.

OP 10.7 · Agents in evolving environments

The question. No benchmark tests whether an agent notices that the world changed and goes back to fix earlier sharing decisions (p. 65). The report's triggers include shifts in political climate, company policy or positioning. Its example responses include deleting user profiles, filing right-to-be-forgotten requests and opting out of data-sharing practices.

Why it matters. A persistent agent that manages data on a user's behalf is responsible not only for each decision when it is made but for whether past decisions are still appropriate. The report says no benchmarks for this exist yet, and calls them critical.

Two scenarios it sketches. (1) The transmission principle changes: a company that received the user's data adopts a new sharing policy after a new agreement, a new government policy or an acquisition. (2) The recipient changes: a government's political leaning shifts, and sharing political views, sexual orientation or finances with government bodies now carries a different risk.

Connections

OP 10.3 and 10.5 point back to the report's treatment of stale and re-contextualized memory in Teaching models contextual reasoning (§8.3). Test awareness, central to OP 10.1, also undermines trust calibration in Trust, stop buttons, accountability. OP 10.1 is about measuring Contextual Security. The deletion and right-to-be-forgotten actions in OP 10.7 are the subject of PII, minimization, deletion, anonymity, and the question of where norms come from and how they change is taken up in Governance.

Further reading, verified

• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory — Mireshghallah et al. (2024) · ConfAIde, the first row of Table 2.
• PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action — Shao et al. (2024) · the vignette-to-trajectory benchmark the report singles out for its broad norm coverage and metric suite.
• CI-Bench: Benchmarking Contextual Integrity of AI Assistants on Synthetic Data — Cheng et al. (2024) · reviewed in App. D; errors broken down by functional stage.
• AirGapAgent: Protecting Privacy-Conscious Conversational Agents — Bagdasarian et al. (2024) · reviewed in App. D; separates context-preserving from context-hijacking adversaries.
• SPILLage: Agentic Oversharing on the Web — Roh et al. (2026) · one of the two benchmarks the report calls most comprehensive multi-modal agent testing.
• MPCI-Bench: A Benchmark for Multimodal Pairwise Contextual Integrity Evaluation of Language Model Agents — Wang & Zhang (2026) · the same image tested in conflicting contexts.
• AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents — Debenedetti et al. (2024) · the report's example of a constrained simulated environment (§10.1).
• Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection — Naseh & Mireshghallah (2025) · cited for why synthetic evaluations can mislead.