Trust, stop buttons, accountability
Section 9 is the report's user-level chapter. Its first two parts (§9.1–9.2) argue that users' mental models of agents are shaky and that click-through consent does not scale to autonomous execution. §9.3 picks up the obvious next question: if users cannot approve every step, on what basis should they let an agent act at all? The report's answer has three strands. Users need ways to check what the agent really did. They need to build trust gradually and in the right domain. And they need two kinds of backstop: controls that can halt or reverse an agent mid-task, and a legal and economic framework for when harm happens anyway. The section opens by pointing to user control, auditing and accountability as the levers (Such et al., 2014, which the report cites). It is a section of questions, not results.
Trust versus trustworthiness, and audit paths you can verify
The report separates two ideas that everyday speech blurs (§9.3.1, p. 56). Trust is something the user does. Trustworthiness is a property the system has. You want both, and the two mismatches are not equally bad.
- Trusted but not trustworthy. The user has been led to believe the agent protects some property when it does not. The report flags this as the worse case, and notes it can be produced on purpose by a manipulative design.
- Trustworthy but not trusted. The agent would behave well, but the user holds back. The report calls this less harmful; the cost is lost opportunity.
So the research question is not "how do we raise trust" but "how do we let users assess trustworthiness, so that trust follows it."
The appeal, and the limit, of asking the agent
Agents speak natural language, so the tempting transparency tool is a conversation. Instead of a settings page, you ask "Why did you send that?" or "What do you know about my health?" and correct it in the same chat where you give it tasks. The report treats this as attractive if it can be made reliable.
The catch is the gap between what an agent says it did and what it executed. The report cites evidence that agents asked about their own privacy behavior sometimes answer wrongly, including inventing details or denying that they touched sensitive data (Zhang et al., 2025c).
"Because the open-ended nature of LLMs makes them unreliable narrators of their actions, conversational transparency is insufficient."— the report, §9.3.1, p. 56
The proposed direction is a trusted path: the audit record is produced by something other than the untrusted task model. The report gives examples rather than a design: a dedicated permission assistant, or visible UI indicators, each backed by an immutable log of actions actually taken. One way to build that, which the report cites, is hardware-backed confidential execution (Abdollahi et al., 2026, AgenTEE). There the agent runtime, the inference engine and third-party apps each sit in their own confidential virtual machine, and every interaction between them goes over a mediated, secure channel. The mediation point is what gives you a record the model cannot rewrite.
A way to picture this: conversation stays the friendly front end, but the answer to "what did you send?" is read from the log, not composed by the model from memory. The model may explain; the log testifies.
"What if" scenarios and trust built in stages
Trust, the report stresses, is situational: for a purpose, in a setting, at a time (§9.3.2, pp. 56–57). Trusting an agent as a "gofer" for meeting scheduling says nothing about trusting it as a "coach" for mental health advice, an example drawn from work on trust in healthcare voice assistants (Zhan et al., 2024b). Learning where an agent can be trusted by trying it live risks costly mistakes. Hence the question: could users stress-test an agent in a simulated environment ("what would you do if...") before relying on it in a new domain?
How people build trust: three stages
The report borrows a model from organizational psychology (Lewicki & Bunker, 1996). People start new relationships with low-stakes tasks and hand over more as things go well. Lewicki and Bunker describe three stages:
| Stage | What it rests on | Agent analogue (our reading) |
|---|---|---|
| Calculus-based | Deterrence. Early, low-harm interactions test whether trust is warranted. | Let the agent draft, not send; read, not write. |
| Knowledge-based | Enough shared history that you can predict the other's behavior. | You know how it handles your calendar and where it tends to slip. |
| Identification-based | Understanding deep enough that each can act on the other's behalf. | Delegation without step-by-step review. |
The report also cites earlier work (Deutsch, 1949) suggesting that identification-based trust is what lets one person act as another's "agent." The implication it draws: user trust in an AI agent will take a history of interactions long enough that the user can understand and predict it.
Simulation as an accelerator, and why it is hard
A sandbox lets a user push the agent through realistic scenarios, watch what it does, and form a trust model, with no real harm if it misbehaves. The report lists three obstacles (p. 57):
- Usability. The interface has to be approachable for novice and non-technical users, not a test harness.
- Realism of scenarios. They must match the user's actual use, for a personal assistant probably including the user's own context. Users cannot be expected to write these scenarios themselves.
- Realism of behavior. Models increasingly detect when they are being tested (Abdelnabi & Salem, 2025, which the report cites). An agent that senses, even implicitly, that it is in a simulation may not show its real-world behavior. The test then calibrates trust against the wrong agent.
The third point is the sharp one. It turns a trust-building tool into a possible source of the bad mismatch from the previous section: trust earned in a test the agent recognized.
The lock-in side effect
If trust takes a long calibration history, that history becomes an asset tied to one vendor. The report notes that moving established trust to another vendor's agent is an open problem. Our reading: this is a portability question similar to data portability, except that what must move is evidence about behavior, which may not transfer when the underlying model changes.
OP 9.5 · Delaying, stopping, and reversing agent actions
What it asks. Can users get real guarantees that "stop" halts everything, and that sensitive actions can be held, previewed or undone (§9.3.3, pp. 57–58)? The report notes that some systems already accept a "stop" message, but it is unclear the model is bound to obey it. It cites a 2026 press account of an agent that ran out of control in a researcher's inbox (Bort, 2026).
Why it is hard for agents. A stop button on a single process is easy. An agent is not one process. It has a planner, and at any moment it may be running several synchronous and asynchronous tools and sub-agents, some with effects outside the system. A "stop" that the planner reads as one more instruction is advice, not a control.
Delayed execution
For actions that are highly sensitive or irreversible, the report's examples are deleting production logs, starting financial transfers and deactivating accounts (drawing on Ramesh et al., 2026, WebSP-Eval). There, the agent could hold the action and either change its plan or escalate to the user for explicit confirmation. The report is clear that this is a trade-off: every delay buys control at the cost of autonomy. Our reading: this links back to the confirmation-fatigue problem in §9.1–9.2. Delay works only if it is reserved for the few actions that warrant it.
Undo and reversibility
An "undo" window lets the user reverse a sensitive operation before it becomes permanent. That needs someone to keep pre- and post-action state, much like rollback in a database. The report admits the obvious limit: many tools and external systems an agent touches have no such capability. One direction it offers is to simulate the action's effects first, in a synthetic environment or a world model. That is a preview, not an undo.
A deterministic circuit breaker
The strongest proposal is a system-level circuit breaker that works deterministically, has a usable interface, and conclusively halts the agent. On "stop," the report says:
"the agent should stop generating new actions and send “kill” signals to all processes it has spawned"— the report, §9.3.3, p. 58
Ideally it then reports back which processes it managed to terminate and which it did not. The report also suggests breakers at the platform level, for systems where many agents communicate, in case unexpected collective behavior needs time to be examined. It compares this to stock-exchange circuit breakers.
What a good answer would have to do (our reading of the problem statement):
- Put the breaker outside the model, so halting does not depend on the model choosing to comply.
- Track the full tree of spawned sub-agents and tool calls, including asynchronous ones, so the kill reaches all of them.
- Give an honest status report, including side effects already in flight that cannot be recalled.
- Classify actions by reversibility ahead of time, so delay and preview are applied where undo is impossible.
OP 9.6 · Accountability frameworks for AI agents
What it asks. Agents will make mistakes that cause harm despite every protection. What should accountability look like, and what would contracts, transparency, liability and incentives mean for an agent (§9.3.4, pp. 58–59)?
The starting observation. No domain is error-free, and society already has machinery for that. Software has SLAs, liability frameworks and incident-response protocols that define acceptable error and remediation. The physical world has tort law, insurance and professional licensing, which compensate victims and hold operators accountable. The question is how much of that carries over.
The contractor analogy
The report finds the closest fit in legal agency and human contractors. Like a contractor, an agent is given a goal by a principal and the freedom to pick the means. Principal-agent theory, which the report cites via Prause (2026), describes the problems that come with handing over decision authority: misaligned objectives, information asymmetry, incentives and accountability. With human contractors, law constrains the relationship through four instruments. The report walks through how each one breaks for an AI agent:
| Instrument | What it does for a human contractor | Where it breaks for an AI agent |
|---|---|---|
| Contracts | Binding scope of work, deliverables, payment and termination terms. | An agent has no legal capacity to contract. The "contract" becomes explicit policies placed in the context window on every task. Nothing deterministically guarantees the model follows them or reads them as a human would. |
| Transparency | Mandatory reporting and third-party audits reduce information asymmetry. | In some ways easier: you can watch chain-of-thought and execution logs live. But the model is opaque, interpretability tools are not usable by laypeople, and explanations can be post-hoc rationalizations that sound right and are invented (Peng et al., 2026; Turpin et al., 2023; Lanham et al., 2023), which gives a false sense of transparency. |
| Liability | The contractor bears legal and financial consequences for negligence, breach or self-dealing. | An agent cannot be punished or sued. Liability could move to model developers, application providers or deployers, but assigning it is murky when harm stems from an ambiguous request, untrusted input, a hallucination or a broken third-party API. |
| Incentives | Pay and reputation tie the contractor's gain to the principal's goal. | The agent's "incentives" are an opaque objective function plus whatever context it was fed, neither visible to the user. Risks: reward hacking against the objective, and misalignment from prompt injection or from clashes between accumulated context and the direct prompt. |
The report ends the section on the hardest piece: how to turn a human principal's interests into quantifiable metrics is still an open research question.
What a good answer would have to do
Our reading: the four rows are not independent. The "contract" is only as strong as the transparency that shows whether it was followed, and that is exactly where the §9.3.1 problem returns: self-reports do not count. Liability needs a record of who supplied which input, so any framework depends on the same tamper-resistant logs the trusted-path idea calls for. And incentives cannot be fixed by contract language if they live in weights and context the user never sees. A credible framework probably combines a technical layer (enforced policies, verifiable logs, provenance of inputs) with a legal layer that assigns liability among the humans and companies in the chain. The report does not propose such a combination; it frames the question.
Connections
The confirmation-fatigue and consent arguments that make these backstops necessary are on Mental models and the end of notice-and-choice. Immutable action records and hardware-backed isolation overlap with accounting and sandboxing on Identity, authorization, accounting. The worry that models behave differently under test carries straight into Evaluations and benchmarks. Unfaithful reasoning traces come up again for model training on Teaching models contextual reasoning. Platform-level circuit breakers for many agents belong with Shared norms, delegation, composition, and who sets and enforces accountability rules is the subject of Governance.
Further reading, verified
- AgenTEE: Confidential LLM Agent Execution on Edge Devices — Abdollahi et al. (2026) · the report's example of a trusted path: agent, inference and apps in separate confidential VMs with mediated channels.
- The Hawthorne Effect in Reasoning Models: Evaluating and Steering Test Awareness — Abdelnabi & Salem (2025) · evidence that models detect tests, which undercuts simulation-based trust calibration.
- WebSP-Eval: Evaluating Web Agents on Website Security and Privacy Tasks — Ramesh et al. (2026) · cited for sensitive, irreversible actions such as deactivating accounts.
- Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting — Turpin et al. (2023) · why reading the reasoning trace is not the same as transparency.
- Measuring Faithfulness in Chain-of-Thought Reasoning — Lanham et al. (2023) · cited alongside Turpin on the false sense of transparency.
- Measuring and Mitigating Post-hoc Rationalization in Reverse Chain-of-Thought Generation — Peng et al. (2026) · the report's citation for post-hoc rationalization.