WALLACE ET AL. 2024 → MANYIH 2026four core papers · three neighbors · three bridges · 13 pages here
The instruction hierarchy in one picture: every note gets read, but a note's rank depends on who wrote it, and a printout from the web does not get to give orders.
Instruction hierarchy, mapped
A language model reads one long stream of text: the developer's system prompt, the user's message, a web page it fetched, the output of a tool. Every one of those can contain something phrased as an order. The instruction hierarchy is the idea that the orders should not count equally, and the research question is how to make a model actually behave that way. This map walks the papers that built the idea, compares their four approaches, and connects it to three sibling maps on privacy, Claude's constitution and model memory.
The one-sentence version: in April 2024 Wallace and colleagues at OpenAI argued that prompt injection, jailbreaks and system-prompt extraction share one root cause, a model that treats all text as equally privileged, and trained GPT-3.5 Turbo to follow a ranking (system > user > tool output) that obeys lower-ranked instructions only when they agree with higher-ranked ones; system-prompt-extraction robustness rose from 32.8% to 95.9%. Later work tried three other routes to the same behavior: mark each token's rank in the architecture (Instructional Segment Embedding), make the model reason about rank before answering (VerIH), and let the ranking have many tiers set at inference time (ManyIH, where the best of thirteen frontier models scores 42.7%). The open problem the papers leave is who assigns the rank, which is where contextual integrity and Claude's principal hierarchy come in.
Tick a page when you have read it; this browser remembers.
01 · PART A · THE PAPERS
Four ways to teach a model whose orders count
One page per paper, each in the same order: the problem it names, the method, the data, the results, and the limits the authors admit. Two more pages cover the neighbors every one of these papers cites: the "data is not instructions" defenses and OpenAI's written chain of command.
Train it
Synthetic conflicts
Wallace et al.: generate aligned and misaligned instructions at each level, fine-tune on the right behavior.
Build it in
A rank per token
ISE: add a learned "segment" vector to every token saying system, user, data or output.
Reason about it
Think, then obey
VerIH: reinforcement learning with checkable rewards on system–user conflicts.
Generalize it
Many tiers, set at run time
ManyIH: up to 12 privilege levels written into the prompt; a benchmark frontier models mostly fail.
The anchor paper. Why system prompts, user messages and tool outputs should not carry equal weight, the aligned-versus-misaligned rule, how training data was synthesized for each attack, the robustness gains (including on attacks it never trained on) and the over-refusal cost.
The architectural route: tag every input token with its source and add a learned embedding for that tag, so the rank is present at every position rather than only in a few delimiter tokens. Gains on two benchmarks, and an honest appendix showing it does nothing against adaptive jailbreaks.
arXiv:2511.04694VerIH, 7,192 samplesGRPO with verifiable rewards
The reasoning route: rewrite half of an instruction-following dataset so the user conflicts with the system prompt, train with reinforcement learning on checkable constraints, and let the model think about precedence first. Roughly 20-point gains on conflicts, and safety improvements without any safety data.
arXiv:2604.09443853 tasks · up to 12 tiersbest model 42.7%
The generalization: real agents get orders from system prompts, config files, skill files, users, tools and other agents, so two or three fixed roles are not enough. A privilege tag in the prompt, a benchmark built around it, and the finding that accuracy falls as tiers grow and shifts when you merely change how privilege is written.
arXiv:2402.06363 · arXiv:2410.05451two-level special case
The simplest hierarchy: one channel for the developer's instruction, one for data that must never be obeyed. Reserved delimiter tokens and a filtering front end (StruQ), then preference optimization that cuts most tested attacks, including optimized ones, below 10% success (SecAlign).
Model Spec, 2026-08-18 versionroot > system > developer > user > guideline
Where the hierarchy is written down as policy rather than trained from a paper: six authority levels, "no authority" for tool outputs and quoted text, delegation by unquoted instructions, and practical rules for when an AGENTS.md file should be followed.
02 · THE COMPARISON
Fixed tiers, many tiers, reasoning, architecture
The four papers answer different questions with different models and benchmarks, so their headline numbers cannot be lined up directly. This page compares what each assumes, where the rank comes from, what it costs, and what it leaves open.
A side-by-side table, the shared evaluation trap (over-refusal), why "more tiers" and "more reasoning" are separate axes, and the one assumption every paper makes: that something trustworthy has already labeled each piece of text with its source.
03 · BRIDGES
Into the sibling maps
The instruction hierarchy answers "whose order wins". Three sibling maps ask neighboring questions: which information flows are appropriate (contextual integrity), how a model should weigh its principals (Claude's constitution), and what to do with facts a model remembers (CIMemories). Each bridge page says where the ideas meet and where they do not.
to: Agentic privacy, mappedtransmission principles vs. privilege tiers
A privilege tier decides whether an instruction is obeyed; a contextual-integrity norm decides whether a flow of information is appropriate. Why an agent needs both, why indirect prompt injection is a low-privilege sender trying to set a flow, and how the CAPS report's policy engine keeps untrusted text away from the rules.
to: Constitutional AI, mappedprincipals · non-principals · conversational inputs
Anthropic, operators and users as principals with different trust; tool results and documents as information, not commands; the user protections an operator cannot override; and why the constitution calls its ordering "not a strict hierarchy".
to: CIMemories, mappedwhere a remembered fact sits
A memory prefix is text the model wrote about you in an earlier conversation, pasted in front of the current one. Is it an instruction, a user message or data? Why none of the papers assigns it a tier, what CIMemories measures instead, and a proposal for treating memory as a sender with its own rank.