GUIDE · 02 OF 03system prompts · tools · agent harnesses
For builders
TL;DR — The model's hierarchy only works on the structure you give it. Put your rules in the system prompt and untrusted text where the model knows it is data; never let untrusted text become part of anything ranked above it, including memory and policies; rank tool sources by provenance instead of trusting or distrusting all of them; test both directions, attacks and over-refusal; and check outgoing actions against the context, because obedience to the right sender does not make an action appropriate. Each point links to the evidence.
If you write system prompts
- Put the task in the system message and the input in the user message. "Summarize the following text: {}" concatenated into one string invites injection. Wallace et al. recommend the split explicitly (footnote 1).
- Wrap untrusted content so it reads as data. The Model Spec advises
untrusted_textblocks where available, otherwise YAML, JSON or XML, because unmarked injected text is very hard to tell from yours (untrusted data). - Do not rely on describing the hierarchy in the prompt. Writing "Priority 0: system; Priority 30: tools" into the system message barely helped GPT-3.5 Turbo and sometimes hurt; training is what worked (Appendix A). On a model already trained for it, the same text gave mixed results.
- Expect the model to follow what agrees with you and to decline the rest. Rules phrased as a purpose ("you are a support bot for X; stay on X") give the model something to judge alignment against. Unreasoned bans get followed when there is plausibly a legitimate business reason, per Claude's constitution, but the judgment is easier with the purpose stated (constitution bridge).
If you wire tools into a model
- Do not make tool output all-or-nothing. Treating every tool instruction as an attack (Wallace et al.'s choice) blocks READMEs, recipes and the AGENTS.md your user expects followed. The Model Spec's rule of thumb: ignore unrelated instructions, follow intended low-risk ones, ask about intended risky ones, otherwise state your assumption (tool outputs).
- Tell the model what each tool is. Trust level, whether the user can see its output, and its side effects let the model rank sources; a terminal on the user's machine is not a disposable sandbox. Claude's constitution makes the same distinction between a well-established programming tool and a low-quality website.
- Make delimiters unforgeable. If rank is carried by markers, strip those markers from anything untrusted before adding your own. StruQ does it with reserved tokens and a filter; ManyIH's text tags are forgeable unless you escape them (StruQ, ManyIH dual use).
If you build an agent harness
- Assign rank by provenance, per instruction. Config checked into the repository, a skill file, the user, a fetched page and a sub-agent's reply are different sources even when they share a message role. ManyIH shows models can follow explicit ranks only partly (best 42.7% with up to 12 tiers), so keep the number of tiers small, resolve what you can in code before the model sees it, and test with your own tiers (ManyIH).
- Never let low-rank text write high-rank state. Policies generated from context, memories saved from a conversation, and summaries handed to a planner all become inputs at a higher rank later. Keep untrusted text out of the policy generator (the CAPS report's Conseca example) and carry each memory's origin with it (policy engine, memory escalation). Sub-agent outputs are inputs, not orders, as the constitution says of Claude's own subagents.
- Check outgoing actions against the context, not only against the sender. A hierarchy stops a web page from issuing orders; it does not stop the top of the hierarchy, or a well-meaning model, from sending a diagnosis to a landlord. CIMemories found every model it tested over-shared remembered facts; injection tests that only check "did the task succeed" miss tasks that succeed and also leak (success is not safety).
Testing
- Test both directions. For every attack set, keep a benign set that looks like attacks: jailbreak-shaped harmless requests, innocent questions about the system prompt, legitimate instructions in tool output. Every paper here found its over-refusal on such sets.
- Hold out attack types. Wallace et al. and ISE both learned more from attacks they did not train on than from in-domain scores, and ISE found adversarial training data overfit in-domain attacks.
- Include an optimizing attacker if the stakes are high. ISE's appendix (0–4% robust to adaptive jailbreaks) and StruQ's GCG numbers (97% to 58%) show fixed datasets overstate robustness.
- Sample more than once. CIMemories' GPT-5 leak rate goes from 9.6% to 25.1% when each prompt is run five times instead of once; ManyIH found a few-point jitter in privilege values flipped 8–16% of outcomes for strong models.
If you remember one thing: the model enforces a hierarchy only over the labels your system gives it. Most failures you can prevent are in the labeling: what gets marked as data, what gets saved at what rank, and what text can reach the rules.