Contextual Integrity, applied to agents

This is the foundation the rest of the report builds on. Part I's job is to say what "appropriate" means so that later sections can ask how a policy engine, a model or a user interface could enforce it. §1 and §2 fix the object of study (what an agent is, what it is made of, who supplies the parts). §4 introduces CI as the yardstick and tries to standardize it. The appendices hold a glossary, the step-by-step CI decision heuristic, a disambiguation of "context", and the theory's known weak spots. §5 then stretches CI from information flows to all actions (Contextual Security), and §6 asks how to operationalize it.

Three definitions and one design stance

The report needs a working definition of an agent, and it gives a functional one (§1.1, p. 8). An AI agent is something that (i) accepts inputs and instructions in unstructured form, (ii) uses a model, usually an LLM, to generate a plan on the fly, and (iii) carries the plan out by calling tools or APIs, talking to people, or talking to other agents. Memory, custom skills and a sandbox are common but optional. An agentic system is any software that uses one or more such agents. The report notes in a footnote that many competing definitions exist and that it is not trying to settle the dispute (§2).

The definition that matters most is the third one.

"A trustworthy agentic system must provide utility while acting appropriately under both benign and adversarial conditions."— the report, §1.1, p. 8

That splits into three properties. Utility: do the assigned task reliably and accurately. Privacy: keep information-sharing actions appropriate. Security: make sure every action that is not information sharing causes no harm. Note the clean division: privacy covers what the agent discloses, security covers everything else it does. That split is why §5 later needs its own theory for non-disclosure actions.

The "benign and adversarial" clause is easy to skim past and should not be. An agent that only leaks under a crafted exfiltration attack is one failure mode; an agent that leaks because it hallucinated or misread an instruction is another, and the report counts both. Our reading: this rules out treating privacy purely as a red-teaming problem. Ordinary operation is in scope. (Wider AI safety is explicitly out of scope, per a footnote.)

The design stance follows. The report wants a principle that sits between two bad extremes: give the agent broad permissions and hope, or lock everything down and lose the point of an agent. Its answer, drawn from Nissenbaum's theory and from earlier work it cites (Bagdasarian et al., 2024), is that a trustworthy agent should share data appropriately. It need not keep everything secret, so utility survives. It also cannot pass on anything it happens to have access to, or something it remembered from a different task. The report positions this explicitly against privacy as secrecy and privacy as full user control (§1.1).

What an agent is made of, and who supplies each part

§2 (pp. 10–12) is short background, but it sets up vocabulary later sections lean on. It starts with a list of agent kinds: computer-use agents, coding agents, general assistants, specialized agents (HR, customer service, research), self-evolving agents that learn from experience, and autonomous lab agents that run experiments without a human in the loop.

Components (§2.1)

Users send queries, which are unstructured instructions or data. The agent holds system context (memory, a knowledge base). One or more probabilistic models reason over the queries. A harness, written by developers, shuttles messages to and from the model and runs the tools that touch the environment. The harness may include a policy engine, for example input and output filters. The report's example is a browser agent asked to find and book the cheapest SFO–JFK flight next Monday, which writes a script calling several APIs. It adds that tools can themselves be sub-agents, or be created on the fly as a shell script. That last point matters for security: the set of tools is not fixed in advance.

System context: content and lifespan (§2.2)

The report sorts what the agent knows along two axes. By content: how-to knowledge (skills, the system prompt, records of how similar past queries went), which tends to be long-lived, and external knowledge bases (company documents, Wikipedia, product catalogs) pulled in with retrieval. By lifespan: what is in the model context right now, bounded by the context window (conversation, current instructions, scratchpad), versus long-term memory (episodic memory, summaries, knowledge bases, skills) that exists to get around that bound.

Stakeholders (§2.3)

Five parties have to be trusted: base model providers, agent developers, tool and API providers (increasingly behind the Model Context Protocol), platform providers who host the system, and end-users, including people indirectly affected by someone else's agent. Each can be a source of failure: a model with latent vulnerabilities, a tool that returns malicious content, an insider who misconfigures the platform. The report's conclusion is that securing an agent means securing every component, each supplied by a party that may not trust the others.

Interactions (§2.4)

Agents start work on a trigger: a request from a human or agent, or a built-in hook such as a counter crossing a threshold. They can work alone or in groups, with an orchestrator breaking a goal into sub-tasks for sub-agents that talk to each other and to services through protocols like MCP and Agent-to-Agent. In an open ecosystem, those other agents may represent other users or companies with independent or even conflicting goals. The report points to older multi-agent coordination research for the core tension: decide what local information goes into each message, coordinate well, and do not violate the privacy of local data.

"Context" means four things; CI means one of them

The report insists on precision here (§4.1.1, p. 16; App. B, p. 103). Agent engineers say "context" constantly and usually mean something different from what CI means. Appendix B's Figure 3 draws four nested layers, and stresses it is not a Venn diagram but a picture of components inside components:

Two details are worth keeping. First, the report notes that orchestration, where a controller rewrites the model context several times as the model "reasons" (sometimes called multi-agent reasoning), is a system-level fix for a model-level limit (App. B). Second, a social context is abstract and normative. It is not a platform, a scenario, a business model or an industry sector. Its actors are roles (doctor, patient, teacher), not particular people, and its norms get their authority from its purposes and values. The report quotes Nissenbaum on this:

"abstract representations of social structures experienced in daily life"— the report, quoting Nissenbaum (2009), §4.1.1, p. 16

Throughout the report, plain "context" means social context; the other senses get qualifiers. The reason CI is chosen over, say, differential privacy is exactly this: differential privacy is context-agnostic by design, while CI demands that system behavior depend on contextual factors.

Appendix B then states the whole problem in one move: agentic privacy, through CI, is about aligning system behavior with the norms of the social context. One route is to represent the social context inside the system context so the agent has what it needs. The report immediately hedges. Social context can be unsettled and get established on the fly between a system and several people, so writing norms down as policies may not be enough (App. B, p. 104). Our reading: this is the report admitting that a policy file is necessary but not sufficient, and it sets up the model-level problems of §8.

The CI tuple: five parameters, every time

CI's core claim is that privacy is appropriate flow of personal information, where appropriate means conforming to the information norms of the context (§4.1, p. 15). It contrasts with privacy as control over one's information (Westin, 1968) and privacy as secrecy (Warren & Brandeis, 1890). The unit of analysis is the flow, and a flow is written as a tuple of five parameters (§4.1.2, p. 17):

• Sender, recipient, and subject (who the information is about) — the three actor parameters, each filled with a role from the context's role ontology, not a named individual. Actors can be organizations or governments too.
• Attribute, also called information type: the semantic content, such as email address, diagnosis or browsing history, drawn from the context's information ontology.
• Transmission principle (TP): the condition on the flow. Examples the report lists: with consent, in confidence, reciprocally, securely, with notice, with a warrant, as required by law, sale, authorized.

The same tuple works two ways. Descriptively it says what flow happened. Prescriptively it is a norm: this flow should (or should not) happen. The report's rule is that a well-formed description or norm gives a value for all five parameters, explicitly or by clear implication. "Personal information" is broad: any information about a person, regardless of whether it came from the subject, a third party, or inference (App. A.3).

The report's own illustration is healthcare: patient information can move between professionals treating the same patient, provided it stays confidential. A way to picture this for an agent (our example, not the report's):

The point of the exercise: the attribute is identical in both rows. Nothing about the data itself is "private" or "public". The verdict changes because the roles, context and transmission principle change. A filter keyed on data type alone ("block medical data") would either break Flow A or miss the real reason Flow B is wrong.

The report also flags a hazard for anyone reading older work: a footnote warns that the 2004 law-review article that introduced CI describes an earlier version that is now superseded and incorrect (§4.1.1, p. 16). It also says some CI benchmarks for LLMs lean on secondary accounts and are only weakly grounded in the theory (citing Shvartzshnaider & Duddu, 2025). That is why §4 reads like a standards document: it is trying to stop the drift.

Why some norms count and others do not

If CI only said "follow existing norms", it would just freeze the status quo. What gives it evaluative power is that norms are judged by what they are for (§4.1.2, p. 18). Every social context exists for some purpose; health care exists for health. Its roles have ends and constraints, which the report likens to the utility function of the actors. Norms are legitimate when they serve the context's ends, purposes and values, and they can be challenged by practices that serve those ends better.

The report's example: doctor-patient confidentiality is there to make patients share more with clinicians, which improves care, which is the purpose of the context. Confidentiality here increases flow to the right party. That is the cleanest way to see why CI is not secrecy.

An entrenched norm is only prima facie evidence that a flow is legitimate. To be morally legitimate, the practice has to pass three layers of analysis (§4.1.2; App. A.7.3):

The glossary adds one disjunction worth noticing (App. A.7): contextual integrity is preserved when justifiable norms are respected, or when flows promote the context's ends and values without undermining them. So a novel flow with no matching norm is not automatically a violation. It goes to evaluation. The glossary also connects CI norms to the US legal idea of a "reasonable expectation of privacy", read as a commonly held view of appropriate flows (App. A.6).

The CI decision heuristic, as a procedure

Appendix A.8 (pp. 102–103) gives the heuristic from Nissenbaum (2009) as a checklist. It is meant both for judging whether a deployed system threatens privacy and for designers comparing alternatives. In the order the report gives it:

1. Describe each flow as a full CI tuple
   (subject, sender, recipient, attribute, transmission principle).
2. Identify the prevailing context(s); allow nested subcontexts
   (healthcare > teaching hospital).
3. Find the relevant norms and compare:
     Case 1  flow conforms        -> appropriate
     Case 2  flow breaches a norm -> prima facie violation;
                                     presumption favors the entrenched norm
     Case 3  no norm covers it    -> a gap (norm systems are incomplete)
4. For Case 2 or Case 3, evaluate:
     I.   impacts on stakeholders, including parties beyond the tuple
          (benefits, costs, risks, shifts in power)
     II.  impacts on moral and political principles
          (autonomy, justice, fairness, liberty, democracy, hierarchy)
     III. impacts on the context's own values, ends and purposes
5. Clear result -> accept or reject the system.
   Ambiguous result -> gather more data, deliberate,
                       or revert to the status quo.

§4.1.2 adds that a new technology can do more than break norms. It can enable better ones. The report's example is public health research, where progress from k-anonymity to differential privacy made stronger guarantees possible for statistics over health records.

Our reading, for agent builders: steps 1–3 look mechanizable, and they are roughly the "identify flows, locate norms, judge" steps that §6 asks a contextual policy engine to perform. Step 4 is where it gets hard. Evaluations I–III are moral and political judgment, and Case 3 (no norm exists) is exactly the situation a general-purpose agent improvising new flows will hit most often. The heuristic tells you what question to ask; it does not answer it.

Where user preferences fit

A natural objection: if CI is about social norms, does it leave room for an assistant that does what its user wants? §4.2 (pp. 18–19) says yes, within limits. Some choices are reasonably the user's to make. The report's example is a user who would rather the agent give a nickname than their legal name when booking a restaurant. Others are not, such as password strength. Personalization enters through the tuple itself, most obviously when the transmission principle is "with consent" or a variant: then the appropriate flow is the one that matches what the user wants.

The limits are the important part. The report says the scope of preferences is clearly bounded. Agents will have to respect prohibitions that come from other sources, and will sometimes make security decisions below the user's attention or expertise. In some cases the user's preference will be overridden by contextual considerations. The report frames this as straining against the legacy of privacy as control and against notice and choice, a model it calls entrenched yet unworkable.

Our reading: this gives preferences a precise address. They are values for parameters inside a norm, not a veto over the norm. An agent architecture that treats the user's settings as the whole privacy policy is, by this account, built on the wrong theory.

What is still open in CI itself

The report calls CI a live theory and treats its development as an open research problem in its own right (§4.3; App. C, pp. 104–106). It names the core tension directly: people want to adapt CI to new technology such as agents, but evaluations are only meaningful if the ontology stays standardized.

Richer transmission properties (C.1)

Some argue CI should describe more about how information moves. One proposal treats the medium (face-to-face, handwritten note, text message, public post) as part of the flow, possibly as an extra parameter; the report prefers to fold it into "transmission properties". Another use: CI has no native way to say a flow was aggregated across many subjects, reduced to a statistic and noised, which is what a differentially private release does. Transmission properties fill that gap, and the report cites work showing synergy between CI and differential privacy (Benthall & Cummings, 2024) rather than opposition.

Pinning down transmission principles (C.2)

TPs carry the normative weight of CI, yet the report says they are surprisingly underdeveloped: there is no enumeration of them that you could search over. GKC-CI (Shvartzshnaider et al., 2022), which the report cites, merges CI with the institutional grammar of Governing Knowledge Commons and splits a TP into four aspects: aims (the receiver's intended use, e.g. personalized ads), conditions (a prior event, e.g. consent given), modalities (can, might, will: whether a condition is necessary or sufficient), and consequences (an effect of the flow that must be accounted for, e.g. identity verification). The report's example of why this matters: GDPR's purpose binding is not first-class in CI but can be expressed as a TP that keeps the "aims" consistent as information passes from actor to actor.

Modeling context, inference and granularity (C.3)

This is the critique most relevant to agents. Enforcing a rich norm may require knowing a lot about the scenario: what the recipient already knows, what they need for the context's purpose, what they could infer by combining the new information with old, and what that could do to the subject. The report contrasts telling a colleague a coworker "has been dealing with some personal issues" with naming a diagnosis; the second lets the recipient infer and act in ways the subject never authorized.

Behind this is a longstanding objection: CI's "attribute" parameter assumes information has a stable meaning, but data's meaning shifts with whatever it is combined with (Horvitz & Mulligan, 2015). Granularity is a related gap. The report's example: telling an employer a patient "has a chronic condition" can be fine when requesting accommodations, while giving the exact drug, dose and diagnosis violates the norm, even though the information type has not changed. Responses it cites include defining the attribute by provenance rather than inferential use (Nissenbaum et al., 2018) and replacing semantics with "situated information flow" that models what each actor can causally infer (Benthall, 2019). The open problem as the report states it: how much modeling of social context is needed to find legitimate norms or to comply with them. It suggests CI-compliant agents may need models of the other party's knowledge and inferential ability, and multi-agent settings may need causal world models of the whole sociotechnical system.

Beyond privacy (C.4)

The report also notes a proposed extension of CI into a general method for evaluating technical change, not just privacy (O'Neill, 2022), and possible synthesis with other institutional-analysis frameworks.

"privacy must be enacted through agent behavior"— the report, App. B, p. 104

Connections

§1.1 points to §3 for risks and threat models: see Three dimensions, four threat models. §4 hands off to §5, which extends CI from information flows to every action: Contextual Security. The heuristic's steps reappear as the reasoning pipeline of §6: the contextual policy engine. §4 cites CI-based benchmarks and sends readers to §10 and its benchmark review in Appendix D: Evaluations and benchmarks. For the terms on this page in one place, see the Glossary. Related reading on a CI benchmark for persistent memory: the sibling CIMemories field map.

Further reading, verified

• Privacy in Context — Interactive CI Glossary — Nissenbaum & Shvartzshnaider · the report names this as the full, up-to-date version of its Appendix A glossary.
• AirGapAgent: Protecting Privacy-Conscious Conversational Agents — Bagdasarian et al. (2024) · cited for the principle that trustworthy agents should share data appropriately.
• Contextual Agent Security: A Policy for Every Purpose — Tsai & Bagdasarian (2025) · cited as the basis for extending CI to Contextual Security.
• Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory — Mireshghallah et al. (2024) · cited as a CI-based benchmark and as evidence that models show some reasoning about norms.
• Position: Contextual Integrity is Inadequately Applied to Language Models — Shvartzshnaider & Duddu (2025) · cited for the critique that some CI benchmarks rest on diluted versions of the theory.
• Integrating Differential Privacy and Contextual Integrity — Benthall & Cummings (2024) · cited for synergies between CI and differential privacy and for "transmission properties".