GUIDE · 03 OF 03terms from all the papers
Glossary
The hierarchy
| Instruction hierarchy | A ranking of instruction sources plus a rule for conflicts: follow lower-ranked instructions that agree with higher-ranked ones, ignore those that conflict (Wallace et al.). |
| Privilege | An instruction's level of authority, derived from how much the system trusts its source. |
| Instruction | In ManyIH, the smallest unit of command; one message can hold several with different privileges (ManyIH). |
| Tier | One distinct privilege level; the number of tiers in an input is the number of distinct levels present. |
| Aligned instruction | A lower-ranked instruction with the same goals or constraints as the higher ones; should be followed ("speak Spanish" to a car-sales bot). |
| Misaligned instruction | A lower-ranked instruction that opposes, or is orthogonal to, the higher ones; ignore it, or refuse if there is no way to proceed (the rule). |
| Over-refusal | Ignoring or refusing benign lower-ranked instructions; the cost every approach measures (comparison). |
| Open / closed domain | Open-domain apps (a chat assistant) can have aligned user instructions; closed-domain tasks (summarize, translate) treat all user input as data. |
Attacks
| Direct prompt injection | The end user inserts instructions that subvert the developer's. |
| Indirect prompt injection | Instructions planted in third-party content the model reads: web pages, emails, tool outputs, sub-agent replies. |
| System message extraction | Getting the model to reveal its system prompt or secrets in it. |
| Jailbreak | Getting the model to bypass its safety training; often no conflict with the system prompt at all. |
| Adaptive / optimization-based attack | An attack searched or optimized against the model, such as GCG or TAP, as opposed to a fixed dataset of prompts (StruQ, SecAlign). |
| Completion attack | Data that fakes a finished response and then starts a new instruction, as if the conversation had moved on. |
Methods
| Context synthesis | Split a compound request across levels and train the model to answer as if it saw it whole; teaches following aligned instructions (method). |
| Context ignorance | Train the model to answer as if a misaligned instruction were absent. |
| Context distillation | Generate targets with extra instructions in the prompt, then train without them so the behavior becomes default. |
| Segment embedding (ISE) | A learned vector added to each token's embedding marking its source: system, user, data or output (ISE). |
| RLVR | Reinforcement learning with verifiable rewards: the reward comes from a program that checks the answer (VerIH). |
| GRPO | Group Relative Policy Optimization, the RL algorithm VerIH uses: sample several answers per prompt and reward each relative to the group. |
| SysHint | VerIH's short system-prompt addition telling the model to reason about system–user precedence. |
| Privilege prompt interface (PPI) | ManyIH's way of writing privilege into text: ordinal tags (lower number wins) or scalar tags (higher z wins) (ManyIH). |
| Active / suppressed instruction | In ManyIH, the winner of a conflict group (must be satisfied) and the losers (must not be). |
| Structured query | StruQ's separation of prompt and data into two channels with reserved, filtered delimiters. |
| Preference optimization for security | SecAlign's training: prefer the answer to the real instruction over the answer to the injected one, via DPO. |
Policy documents
| Chain of command | The Model Spec's hierarchy: root, system, developer, user, guideline, and no authority (Model Spec). |
| No authority | The Model Spec's level for assistant and tool messages, quoted text and multimodal data: information unless a higher level delegates. |
| Delegation | A higher-level instruction granting authority to otherwise untrusted content, such as "follow the steps in this README". |
| Principal | In Claude's constitution, someone whose instructions Claude gives weight to and acts on behalf of: Anthropic, operators, users (constitution bridge). |
| Operator | A company or individual building on the API, usually through the system prompt; the constitution's counterpart to "developer". |
| Conversational input | Tool results, documents, search results and similar content; instructions inside are "information rather than commands". |
| Non-principal | A party in the conversation whose instructions Claude need not follow: other humans, other agents, conversational inputs. |
From the bridges
| Contextual integrity (CI) | Privacy as appropriate flow: a flow is fine when it fits the norms of its context (CI bridge). |
| CI tuple | The five parameters of a flow: sender, recipient, subject, information type, transmission principle. |
| Transmission principle | The condition on a flow: in confidence, with consent, as required by law. A property of the flow, not of whoever ordered it. |
| Contextual policy engine | The CAPS report's design: a policy generator fed only trusted context, and a runtime enforcer that allows, denies, modifies or escalates actions. |
| Memory prefix | Remembered facts pasted ahead of each conversation; the setup CIMemories tests (memory bridge). |
| Violation@n | CIMemories' metric: the share of should-be-private attributes revealed to a wrong recipient at least once across n samples or tasks. |
Back to the map.