Mental models and the end of notice-and-choice
Part II of the report splits the problem into three levels: the system around the agent, the model inside it, and the user in front of it (§1.2). This page is the user level. The two earlier component pages ask how to fence and steer the agent; this one asks what the person is supposed to understand and decide. The report frames §9 as three needs: accurate mental models, a shift from static consent to dynamic and contextual control, and user trust (§9, p. 50). The first two are covered here. Trust, stop buttons and accountability (§9.3) have their own page. The premise is blunt: if agents could do every task flawlessly and provably, users could step out of the loop. They cannot, so the open question is what role the user should play and what role they want.
OP 9.1 · How users picture agents, and how to keep that picture current
The problem asks two things. First, how do people's ideas about privacy and security differ when the thing in front of them is an agent rather than an app or a website? Second, what mechanisms would help them form a correct picture, and then correct and refresh it as the agent changes (OP 9.1, p. 50)? The second half matters as much as the first. A mental model is not formed once; it goes stale as the agent's model, memory and tools change underneath it.
The report gives five reasons the old picture fails, one per subsection.
1. The workflow is open-ended and brittle (§9.1.1)
A script or a program runs a fixed sequence. An agent picks its steps at run time from the current context, so the user cannot know ahead of time which tools it will call or which data it will touch (also noted in §7.5). Worse, the plan is "jagged": the language model that writes it is sensitive to small input changes, so the same task can produce different plans for different users or on different days. That inconsistency hides errors. When the agent does something unexpected, the user cannot tell whether it made a mistake or simply took another route (§9.1.1, p. 51).
2. The interface is social, and the parties multiply (§9.1.2)
Agents talk in natural language, so people treat them like people. The report says this leads users to apply social privacy reasoning (what would I tell a friend?) instead of the institutional expectations they bring to software, which produces misplaced trust and riskier sharing. It also gives the agent a kind of influence over beliefs that a settings page never had, and the report notes that attackers could exploit this to manipulate users through their own agent.
The second shift is in who is involved. Classic systems have a known set of parties, often authorized through something like OAuth 2.0, which the report cites. An agent can talk to other people and other agents on the user's behalf, which makes the interaction many-to-many, with a different level of trust for each party. It also processes collaborative data, such as shared email threads, meeting transcripts and other people's documents, so the person typing the prompt is no longer the only data subject. The key line here is about authentication:
"authenticated channels do not imply governable behavior."— the report, §9.1.2, p. 51
Our reading: a login tells you who is on the other end, not what will be said. Because the agent writes each exchange on the fly, and stochastically, knowing that a channel is authenticated says little about which information will cross it.
3. Risk can come from a trusted agent (§9.1.3)
Users have learned that danger comes from malicious software running on a trusted platform. An agent can hurt them without being compromised at all, just by making a mistake while doing a sensitive task. The report's example is an agent that flips a social media account from private to public. The other new risk is merged contexts. A multi-tool plan can cross personal, work and social domains, and stray events or ambiguous data can pull those contexts together, so something from a private setting ends up in a public one. The report links this to the model-level question of reasoning under ambiguity (§8.2).
4. The agent changes over time (§9.1.4)
Two things drift. Memory builds a profile of the user, often without direct supervision, which leaves a large footprint the user cannot easily see, and which may not match their actual preferences. The underlying model also changes. The report cites research showing that different models and prompting methods leak private information at different rates, while users tend to treat "the AI" as one thing. A new model release, or a cheaper downgrade, can change behavior without the user noticing. Its suggestion, framed as something that "could help," is to surface these differences to users (§9.1.4, p. 52).
5. Merged contexts, infinite memory and bystanders (§9.1.5)
This subsection gathers four failures that make the earlier ones concrete.
- Bridged silos. The report's example: an agent that ties a personal dinner request to a work calendar can reveal medical dietary restrictions to an employer. Each context had its own norms, and the agent ran them together.
- Infinite memory. An instruction given years ago can outlive the life it was written for. After a marriage or a child, old permissions may no longer fit. The report's ideal is an agent that abstains or asks for fresh context (also §8.3).
- Bystanders. When an agent scans a user's messages and photos, it handles data about friends, family and colleagues who never used the system. The user is in effect making a consent decision on their behalf.
- Fluency mistaken for discretion. People assume a fluent agent understands unwritten rules, such as not mentioning a job search to the boss. The report says the system treats input as raw data unless told otherwise, so users may lower their guard and share more than they would with a tool they saw as a tool.
What a good answer to OP 9.1 would have to do, in our reading: measure how agent mental models actually differ from app models (the report poses this as a question, not a finding), and design ways to update the user's picture when the agent's model, memory or behavior shifts, not just at first use. For memory specifically, the sibling CIMemories field map covers a benchmark on how agents misuse remembered facts across contexts.
OP 9.2 · Thinking past notice and choice
Notice and choice is the familiar model: a service tells you what it does with your data and gives you options. The report reminds us that it was in trouble well before agents (§9.2, p. 52). Data practices got too complex to follow. Empirical work doubts that privacy notices leave readers better informed, and philosophical work doubts that consent in these settings means much. The concrete evidence it cites spans Windows User Account Control prompts, iOS and Android app permissions, cookie dialogs and browser permission prompts: across all of them, asking too often undermines the very goal of informing users and giving them control.
Agents make this worse. They are unpredictable and act faster than a person can follow. The report concludes that if consent is not reliably tied to what people actually want, notice and consent becomes not just impractical but "normatively unmoored," meaning it loses its claim to legitimacy, not just its usability.
Why Contextual Integrity goes further
Contextual Integrity (CI) is the report's lens throughout. Here the report makes a sharper point: CI's challenge to notice and choice is not only that consent dialogs work badly. CI also questions whether individuals should be the ones making fine-grained decisions about personal data at all (§9.2, p. 53). Two reasons are given. One person's privacy choices can affect others (the report cites Viljoen, 2021; Barocas & Levy, 2020; Barocas & Nissenbaum, 2014). And collective, social and contextual values deserve weight and may override what a single person prefers. So CI leans on substantive norms rather than on possibly uninformed individual clicks.
OP 9.2 then asks: if notice and choice was questionable for ordinary software, it is unlikely to work for agents, so what frameworks give users agency and transparency without overwhelming them or gutting the agent's usefulness? Note the two-sided constraint. A design that drowns the user fails, and so does one that makes the agent useless.
OP 9.3 · Appropriateness when many people are affected
If an agent follows CI, it has to judge whether each action is appropriate, whether that means asking for consent or following a rule. That judgment needs a view of everyone the action touches, not only the person who asked (§9.2.1, p. 53). OP 9.3 asks how appropriateness can be determined when users, bystanders and other third parties are all in the picture.
Stakeholders and bystanders
The report is careful that this is not new. It cites earlier work showing that camera-based augmented reality headsets affect people nearby and that home IoT devices can play a role in domestic abuse. Its claim is that agents amplify the issue. Two of its examples:
- A user asks the agent to email someone. If the email carries adversarial content picked up elsewhere, such as a prompt injection, the recipient is now exposed. The harm lands on a third party.
- An agent with access to a wearable camera or microphone can capture and share images or conversations of people who never agreed to anything.
As a design habit, the report points to Butler et al. (2025), who urge designers to check regularly that they have accounted for everyone their system affects. It then asks, as an open question, whether agents could consider stakeholders beyond the "default persona" of the imagined typical user (a term from Sim et al.), including at-risk and vulnerable groups.
A way to picture this: today's permission check has one principal, the account holder. A good answer to OP 9.3 would need a check with several principals, some of whom are not present, cannot be asked, and may have interests that conflict with the user's.
Where consent still matters, and four ways it breaks
CI does not throw consent out. In CI terms, "with consent" can be the transmission principle that makes a given flow appropriate (the report points back to §4). So the report asks how to make consent work inside an agent, and lists four failures of static consent (§9.2.2, p. 54):
| Failure | What goes wrong | The report's example |
|---|---|---|
| Foreseeability | You cannot agree to steps nobody knows yet. Fixed features already strained this; agents that choose their own steps break it, since neither user nor developer can fully predict the path. | An agent infers your location from receipts and calendar invites. "Manage my schedule" may license combinations, like private email cross-referenced with public social media, that no one foresaw. |
| Inference | Consent covers the raw inputs, but agents combine inputs with other knowledge to produce new insights the consent never mentioned. | Health conditions or socioeconomic status inferred from grocery purchase history. |
| Binary grants | Allow/Deny cannot express conditions. Squashing a conditional instruction into one button either cripples the agent or hands it too much authority. | "Buy plane tickets," but the user would want to reconsider if the price is too high or the layover too short. |
| Volume and fatigue | Agents spawn many sub-tasks. Asking at each one kills the point of background work and trains users to click "accept" without thinking. Asking rarely and broadly grants too much. | Confirming every email read or routine reply draft. |
"signing a blank check for future behaviors and data combinations"— the report, §9.2.2, p. 54
Our reading: the last row is a trap with no good setting. Frequent, narrow prompts produce fatigue and rubber-stamping; rare, broad prompts produce over-broad authority. Tuning the frequency alone cannot escape it, which is why the report moves to changing what is asked and when.
From static permission to dynamic control
Where consent is what legitimates an action, the report suggests using the agent's own automation to decide how closely it should be supervised. The goal is to replace all-or-nothing grants with a spectrum of control set by identifiable factors (§9.2.3, p. 54). Three directions follow, all presented as approaches to explore.
Risk-scaled mixed initiative
The agent asks targeted questions when intent is unclear, such as which "Sarah" to email or what tone a message should take. That ties consent to the present context instead of an instruction from months ago. Other actions need no confirmation at all. The idea is a mixed-initiative agent that adjusts its autonomy to the risk, treating the user's attention as a scarce resource and spending it only where an error would justify a human stepping in. The report's three tiers:
- Low stakes, like organizing a playlist: proceed automatically.
- High stakes, like a financial transfer: require explicit confirmation.
- Risk to third parties: possibly block outright. Here something other than the user's consent wins.
The report cites the Janus permission-management playground (Brigham et al., 2027) as a platform for experimenting with how much to involve the user.
Generative interfaces
When a just-in-time prompt is unavoidable, the agent could build an interface for that specific request rather than reuse one generic dialog. The report suggests showing the request as an information flow: who is involved, what type of data, what could go wrong. The level of detail could vary from sketches and block diagrams to concrete vignettes, like a guest trying to open a camera feed late at night. The aim is lower cognitive load per decision (§9.2.3, p. 55).
Personal preferences and permission prediction
There is a tension the report names directly: CI doubts individuals' entitlement to fine-grained decisions, yet personal preferences can still be one layer in a hierarchy of policies, alongside legal rules, institutional policies, social norms and developer guidance. Prior work on automated permission assistants decided requests from a user's past choices, similar users' choices and context. For agents, the report describes preliminary work showing that prior decisions predict future ones with high accuracy (it cites Flemings et al., 2026 and Fawaz et al., 2026, and gives no figure). Agents could also read implicit grants from the prompt itself: "send my current location to Alice" implies GPS and contacts access for that one task.
OP 9.4 · Letting users supervise automation through policies
Agents move faster than users can follow, so approving each action is not realistic. OP 9.4 asks whether users can instead write policies that steer the agent, how to elicit those policies, and how users can maintain, edit and verify them over time (OP 9.4, p. 55). The report also flags an open question of how policies should be represented and how to make agents actually follow them.
Onboarding: getting the policy out of the user's head
Users are not necessarily technical and do not have time for long setup. The report sketches strategies with different trade-offs between usability and utility:
| Strategy | Upside | Downside |
|---|---|---|
| Walk the user through realistic scenarios, then compress their answers into policies | Policies that predict what the user would do | Cumbersome for the user |
| Learn from past behavior, by recall or from logged decisions | Little effort; matches what the user did | Past choices may reflect bad interfaces, deceptive patterns or resignation |
| Adaptive interview that models how the user decides | Listed as a further option | Not elaborated in the report |
Our reading: the second row means learning from history can launder bad consent. Clicks made from fatigue or under a dark pattern become policy.
Maintenance: keeping the policy right
Policies need editing for two reasons: the agent did something the user did not want, or the user's preferences changed with life events or new technology. That requires an editable representation. The report suggests the real policy could be in a formal language while users read and edit it in natural language. It names two maintenance mechanisms:
- In-the-moment disambiguation: the agent pauses to ask when intent is ambiguous or contexts conflict.
- Retrospective editing: the user reviews past actions and fixes the rule behind a wrong inference.
Our reading: a policy that is never revisited is the stale-permission failure of §9.1.5 in another form.
When a click no longer means consent
The section ends on a point that is easy to miss and hard to fix. Automating consent may strip the user of agency, and it collides with infrastructure that assumes a human click proves intent (§9.2.3, pp. 55–56). The report lists three places that assumption is built in: OAuth consent screens that need a click on "Allow," payment flows where the user confirms details (reinforced in Europe by PSD2 Strong Customer Authentication), and Terms of Service where clicking "I Agree" forms a binding contract.
An agent in an authenticated browser session clicks, fills forms and navigates just as a person would. When it clicks "Buy Now," the system logs that the user consented, even if the user never explicitly did.
"If the "click" no longer carries its consensual meaning, how does this interact with the current legal, regulatory, and security infrastructure"— the report, §9.2.3, p. 56
Two gaps follow. Technically, most authentication protocols, browsers and APIs cannot signal whether a human or an agent is clicking. Legally, the California Consumer Privacy Act lets a user authorize an "agent" to make privacy requests, but that agent is another person or a business. The report says no framework exists for a non-human agent making security and privacy decisions such as consent.
Our reading: this is where the user-level problem turns into an identity problem. Fixing it likely needs a way to mark actions as agent-made and to tie them to a specific delegation, which is the territory of the identity and authorization page.
Connections
The report ties this section to several others. Open-ended workflows are the same problem developers face in AI as the last-mile developer (§7.5). Reasoning under ambiguity and abstaining on stale context are model-level questions in Teaching models contextual reasoning (§8.2, §8.3). "With consent" as a transmission principle comes from Contextual Integrity, applied to agents (§4). A hierarchy of policies is the input a contextual policy engine must resolve, and the human-or-agent click gap meets identity, authorization, accounting (our links, not the report's). The third user-level need, trust, continues in Trust, stop buttons, accountability.
Further reading, verified
- You've Been Warned: An Empirical Study of the Effectiveness of Web Browser Phishing Warnings — Egelman, Cranor & Hong (2008) · cited for the finding that prompting too often undermines informed control.
- User-Driven Access Control: Rethinking Permission Granting in Modern Operating Systems — Roesner et al. (2012) · cited on OS permission prompts and on reading implicit grants from user actions.
- Security Fatigue — Stanton, Theofanos, Prettyman & Furman (2016) · cited for prompt and warning fatigue.
- A Relational Theory of Data Governance — Viljoen (2021) · cited for why one person's data choices affect others.
- Sketch-based Access Control: A Multimodal Interface for Translating User Preferences into Intent-Aligned Policies — Monteiro & Das (2026) · cited for sketch-level visualization and eliciting user policies.
- Text-Based Personas for Simulating User Privacy Decisions — Fawaz et al. (2026) · cited as preliminary work on predicting users' permission decisions.
- RFC 6749: The OAuth 2.0 Authorization Framework — Hardt (2012) · cited as the fixed-party authorization model and the "Allow" consent screen.
- California Consumer Privacy Act (CCPA) — California Attorney General · the report notes its "authorized agent" covers people and businesses, not AI agents.