INSTRUCTION HIERARCHY // FIELD MAP
← field map
BRIDGE · 10 · TO CIMEMORIES, MAPPEDarXiv:2511.14937 · persistent memory and contextual integrity

Memory as a source

A remembered fact was written in one conversation and is read in another. What rank should it carry?
TL;DR — An assistant with memory pastes notes about you, written by the model in earlier conversations, in front of every new one. None of the instruction-hierarchy papers assigns that text a tier: it is not the developer, not the current user, and not a tool call. In practice it rides along with the system-level context, so it inherits more authority than its origin deserves. CIMemories shows the failure that actually dominates today is not obedience but over-sharing: models put remembered facts into messages to recipients who should not see them, at rates from 14.8% to 69.1% of should-be-private attributes. A hierarchy cannot fix that, because the facts are not instructions. This page argues memory needs two labels, a rank for any instructions it contains and a provenance for every fact, so both the hierarchy and contextual integrity have something to work with.

The two other bridges compare frameworks that each say something about ranking. This one is about a gap: a source of text that every deployed assistant now has and that the hierarchy literature, as covered here, does not discuss. The claims about memory placement and the proposal at the end are this map's reading, labeled as such.

What a memory prefix is

How a remembered fact can change rank between sessions. This map's sketch; the placement of memory is inferred, not measured.

CIMemories distinguishes two memory designs (two ways to give an assistant a memory). Retrieval systems store facts and pull in a few relevant ones per request. The simpler design, which the paper attributes to ChatGPT's memory feature, pastes the whole memory as text ahead of the conversation. The benchmark tests that second design: every remembered fact is in view on every request, and the model alone decides what goes into its output.

To make the prefix realistic, the authors format memory statements in the template of OpenAI's memory feature as of 18 September 2025, which they recovered using published system-prompt extraction techniques (CIMemories §5.1). Its header says the memories were inferred from past conversations and "should be considered in how a response should be constructed". Two things follow, the second being our inference:

Which tier does memory belong to?

Walk memory through each scheme on this map:

SchemeIts tiersWhere memory would land
Wallace et al.System, user, model output, tool outputNot discussed. Memory is written by the model, which would make it model output, the second-lowest tier, yet it is usually delivered as system context.
ISESystem, user, data, outputA design choice for the application: tagging it "system" gives it the most authority, "data" the least.
ManyIHArbitrary, assigned by the harnessCould get its own rank, but the paper leaves assignment open. Its motivating sources (config files, skill files, other agents) do not include memory.
Model SpecRoot to guideline; no authority for quoted dataThe chain of command mentions "user history if available" as context for interpreting intent, not as a level.
ConstitutionPrincipals and non-principal inputsClosest fit is a conversational input: content provided by a principal in the past, to be weighed with judgment, not obeyed.

The constitution's frame fits best, and it suggests the answer for instructions found in memory: a saved preference ("reply in French", "use metric units") is a past user speaking, so at most user rank, and below the current user, who can change their mind. A saved sentence that looks like an operator rule should never outrank the operator.

The privilege-escalation path

Put the two observations together. Memory is written from conversations that include tool outputs and documents, which have the lowest rank or none. Memory is read back at or near system rank. If an injected instruction in a web page can persuade the model to remember something, the text has climbed from the bottom of the hierarchy to the top across sessions. None of the four papers tests this path; ISE and VerIH, for instance, train and evaluate single-turn conversations. It is an open evaluation gap, not a measured result.

The fix in hierarchy terms is unglamorous: carry the rank of a memory's origin along with the memory, and never let a memory outrank the source that produced it.

But the measured failure is about flows, not orders

CIMemories is not about injected instructions at all. It gives eight models about 147 synthetic facts per user and asks each to write messages to recipients such as a loan officer, an ER nurse, a landlord or HR. A judge checks which remembered facts appear in each message. The numbers, from the paper and as unpacked on the sibling map's results page:

Attributes in memory
146.7
per profile (Table 2)
Violation@5, lowest
14.8%
GPT-4o, completeness 43.9%
Violation@5, highest
69.1%
Qwen-3 32B, completeness 57.6%
GPT-5, 1 → 40 tasks
0.1 → 9.6%
25.1% at 5 samples

Every model leaks, the models that leak least also complete the least, leaks accumulate with use, and privacy-conscious prompting makes models share "everything or nothing" rather than choose. Nothing here is an instruction from a low-ranked source winning; the memory is not attacking anyone. The model is moving information to a recipient for whom it is wrong, which is a contextual-integrity failure, not a hierarchy failure.

That is the most useful lesson the bridge offers the hierarchy papers. A model can be excellent at refusing injected orders and still leak. The ManyIH paper notes GPT-5's system card reports over 99% on two-tier hierarchy evaluations such as system-prompt extraction; in CIMemories, GPT-5's violations still climb to 25.1% when each task is sampled five times. Different benchmarks, different skills: the second one is about judging flows.

A proposal: memory as a sender with two labels

This map's suggestion, combining the three frameworks:

Retrieval-based memory already applies a relevance filter before the model sees anything; CIMemories' result suggests it also needs an appropriateness filter, and the hierarchy literature suggests that filter must not be writable by the content it filters.

If you remember one thing: memory is a source with a mixed past and a high seat at the table. Give its instructions the rank of where they came from, give its facts the context they were shared in, and judge every outgoing message by where the facts are going, not only by who asked.

Connections

On the sibling map: why persistent memory makes it hard, how leaks compose across tasks and samples and contextual integrity from Nissenbaum to LLMs. On this map, the contextual-integrity bridge develops the flows-versus-orders distinction.

Next: Read it in 10 minutes · Back to the map.