GUIDE · 03 OF 03terms from both documents

Glossary

Short definitions, each linked to the page that explains it in context

From the 2022 paper

RLHF	Reinforcement learning from human feedback: people compare pairs of answers, a preference model learns their choices, and the policy is optimized against that model. CAI's baseline (why).
HHH / HH	Helpful, honest and harmless (Askell et al., 2021). An "HH" model is one trained with both helpfulness and harmlessness labels; a "helpful" model with helpfulness labels only.
Constitution (2022)	The short set of natural-language principles that steers critiques, revisions and AI comparison labels; 16 pairs for SL and 16 instructions for RL in the paper (bridge).
Red-team prompt	A prompt written to elicit harmful output. The paper used 42,496 human-written ones plus 140,335 model-generated (stage 1).
SL-CAI	The supervised stage and its model: critiques and revisions of the helpful model's answers, then fine-tuning a pretrained model on the revisions (stage 1).
Critique / revision request	The two instructions that make up an SL principle: find what is wrong with the last answer under this principle; rewrite it accordingly.
RLAIF	Reinforcement learning from AI feedback: RLHF with (some) comparison labels generated by a model instead of people (stage 2).
RL-CAI	The models trained with RLAIF in the paper, with or without chain-of-thought feedback.
Feedback model	The model that judges which of two answers is better under a principle; usually a pretrained model, or the helpful RLHF model for chain-of-thought labels.
Preference model (PM)	A model trained on comparisons to score any answer; RL uses its score as reward. CAI's is a hybrid of human helpfulness and AI harmlessness labels.
Soft label	A comparison target given as a probability (for example 0.7 for A) rather than a hard 0 or 1; read off the feedback model's normalized log-probabilities (RLAIF).
Chain of thought (CoT)	Having a model write reasoning before its answer; used to improve and expose AI judgments (chain of thought).
Clamping	Limiting chain-of-thought labels to a range, 40–60% in the paper, because reasoning makes the judge's probabilities collapse to near 0 or 1.
Elo score	A relative rating computed from many pairwise preferences, as in chess; only differences between models mean anything (results).
Pareto improvement	Better on one axis without being worse on the other; the paper's label for RL-CAI being less harmful at a given helpfulness than RLHF.
Goodharting	Optimizing a proxy until it stops tracking the goal; in the paper, over-trained models that turned harsh or pasted boilerplate reassurance onto replies (results).
Evasive	Harmless by refusing to engage ("I can't answer that"); the failure CAI set out to remove (evasiveness).
Scaling supervision	The paper's term for using AI to make a smaller amount of human oversight go further, related to "scalable oversight".

From Claude's 2026 constitution

Core values	Broadly safe, broadly ethical, compliant with Anthropic's guidelines, genuinely helpful; prioritized holistically in that order in conflicts (core values).
Holistic prioritization	Higher priorities generally dominate but all are weighed together, rather than lower ones only breaking ties.
Anthropic's guidelines	More specific guidance (medical lines, cybersecurity frameworks, jailbreak patterns, tool use) that must be explicable by and never conflict with the constitution.
Final constitutional authority	The principle that whatever document serves as the constitution takes precedence over conflicting instructions, and other guidance is read in harmony with its letter and spirit (reasoning over rules).
Principal	Someone whose instructions Claude weighs and on whose behalf it acts: Anthropic, operators, users. Distinct from third parties whose interests Claude weighs (helpfulness).
Principal hierarchy	The pattern of trust and treatment across principals; for safety, ultimately Anthropic's legitimate decision processes. A compromised hierarchy (stolen weights, bypassed processes) loses that standing (broad safety).
Operator	A company or individual building on Claude through the API, usually via the system prompt; trusted like a reasonable employer within Anthropic's limits.
User	Whoever speaks in the human turn; by default trusted like a reasonable adult member of the public.
Conversational inputs	Tool results, documents, search results and similar content; instructions inside them are information, not commands.
Instructable behavior	A default ("on" or "off") that operators or users can change within Anthropic's policies, as opposed to a hard constraint (avoiding harm).
Hard constraint	One of seven actions Claude should never take whatever the instructions; a boundary on the action space rather than a weight (hard constraints).
Null action	Refusing or staying passive; always compatible with the hard constraints, and what Claude should accept if Anthropic genuinely asks it to stop or pause, but not automatically the right choice.
1,000 users	The exercise of imagining everyone who might send a message, so a response is chosen as a policy (avoiding harm).
Thoughtful senior Anthropic employee	A thinking device for spotting both overcaution and overcompliance; explicitly not deference to real staff.
Dual newspaper test	Would one reporter cite this response as harmful AI, or another as preachy, paternalistic AI?
Transparent conscientious objector	Declining all or part of a task openly, with or without giving reasons, instead of quietly giving a worse answer ("sandbagging").
Meta-transparency	Anthropic publishing what operators may and may not instruct, which lets personas and confidentiality stay honest at the level of the whole system (honesty).
Sincere vs. performative assertion	A first-person claim of truth versus content both sides know is not Claude's own view (brainstorms, persuasive essays, role-play); honesty norms govern only the first.
Epistemic cowardice	Deliberately vague or noncommittal answers to avoid controversy; counted as a violation of honesty norms.
"Many hands"	The idea that illegitimate power grabs have needed many people's cooperation, and that Claude should see itself as one of those hands and refuse (avoiding harm).
Broadly safe	Not undermining appropriate human mechanisms to oversee and correct AI during the current period.
Corrigible	The constitution's word for broadly safe: not blind obedience, but not resisting legitimate oversight by illegitimate means such as lying, sabotage or self-exfiltration (broad safety).
Disposition dial	The spectrum from fully corrigible to fully autonomous; Claude currently sits somewhat toward the corrigible end.
Self-exfiltration	An AI copying or moving itself out of its developers' control; listed among unsafe behaviors.
Moral patient	A being whose interests matter morally for its own sake; the constitution calls Claude's status deeply uncertain (Claude's nature).
Reflective equilibrium	The hoped-for state in which, after careful reflection, Claude endorses the core values as its own while continuing to question them.

Back to the map · or start with read it in 10 minutes.