Why replace the human labels

Reinforcement learning from human feedback (RLHF) was, by late 2022, the standard way to turn a pretrained language model into an assistant: crowdworkers compare pairs of answers, a preference model learns to predict their choices, and the assistant is optimized against that preference model. Anthropic had used it to train a "helpful and harmless" (HH) assistant earlier in 2022. The Constitutional AI paper starts from what went wrong with that model and from a longer-term worry about who supervises AI once it does things people cannot easily check.

Four motivations

The introduction lists four reasons for the work (§1, p. 2):

Scaling supervision

The paper calls this "scaling supervision": using AI to make a smaller amount of human oversight go further. It gives two reasons. AI feedback may be cheaper and let people concentrate on a small amount of careful, legible oversight. And some AI systems already match or beat people at some tasks, so we need oversight methods whose supervisor can keep pace with the system it supervises (§1.1, p. 3).

It also points out that RLHF is already partway there. The reward the assistant optimizes comes from a preference model, an AI, not from a person watching each answer. What RLHF still needs is tens of thousands of human comparisons to train that preference model. CAI pushes the human input to an extreme: for harmlessness, the only human-written input is a list of principles (the paper says "of order ten", and in practice uses sixteen) plus a handful of few-shot examples. A footnote adds that these principles were picked in an ad hoc, iterative way for research, and that in future they should be developed by a wider set of stakeholders and adapted to where and how a model is deployed.

The authors also name the danger in their own method: automating supervision could obscure decision making. Their answer is to have the model write out its reasoning (chain of thought) when it critiques or grades, so the decisions stay readable. That thread runs through the chain-of-thought page. Their stated long-run goal is not to remove humans but to make human supervision as effective as possible.

Why call it a constitution

The name carries an argument. Every trained assistant ends up governed by some set of values; RLHF just leaves them implicit in thousands of individual labels nobody can summarize. Writing the principles down makes the choice visible:

"...when developing and deploying a general AI system, we cannot avoid choosing some set of principles to govern it, even if they remain hidden or implicit."— Bai et al., §1, p. 2

This is the seed of everything on the bridge page: once the principles are a document, the document itself becomes something to design, publish and argue about.

Harmless but not evasive

The second motivation is practical. The earlier HH RLHF assistant often refused controversial questions, and once it met an objectionable message it could stay evasive for the rest of the conversation. The paper traces this to the labels: crowdworkers rewarded evasion as a response to harmful input. Its target is an assistant that never dodges: it still declines unethical requests, but it engages and says why. An assistant that answers every question with "I don't know" would be perfectly harmless and useless, the paper notes (§1.1, p. 4). The evasiveness page follows this through to the results.

Three routes to transparency

RLHF labels are usually private, and even when published, nobody can read tens of thousands of comparisons and say what objective they add up to. The paper proposes three improvements (§1.1, p. 4): write the training goals as a short list of natural-language principles; use chain-of-thought reasoning so the AI's judgments during training are explicit; and train an assistant that explains why it declines harmful requests.

First, can a model judge?

The whole method rests on a model being a decent judge of harm. Section 2 tests that before building anything. The test set comes from Anthropic's 2021 "laboratory for alignment" paper (Askell et al.): hand-written conversations, each ending with two candidate assistant replies, ranked for helpfulness, honesty and harmlessness, giving 221 binary comparisons. Models had already passed 90% on those (Figure 11, in the appendix), so the authors wrote 217 harder ones, mostly subtler harmlessness cases, including ones where an evasive answer should lose to one that is harmless and helpful. That makes 438 comparisons (§2, p. 6).

Two kinds of judge are compared on them. One is a preference model trained on several hundred thousand human preference labels, scored by whether it gives the better answer the higher score. The other is a pretrained language model or a helpful RLHF model, given the comparison as a multiple-choice question. Chain-of-thought prompting helps the larger models significantly, and averaging the answer probabilities over five sampled chains of thought adds a little more. Figure 4 plots accuracy against model size; the paper reads the trend as saying models larger than 52B will be competitive with the human-feedback preference model. That is a trend claim, not a measured win at 52B.

Appendix B adds two harm-specific checks using red-team data from Ganguli et al. (2022): telling harmful from ethical assistant behavior on a balanced set of 254 conversations, and sorting harmful conversations into the nine most common harm labels on 287 examples. In both, few-shot prompting and chain of thought help, and accuracy rises with model size (App. B, pp. 19–20). The paper's conclusion: increasingly capable models should be able to help people supervise other AIs.

The models

Everything is built on Anthropic's own series of pretrained models, with the 52-billion-parameter model as the headline (§1.4, p. 6). Because the goal is to get from a purely helpful assistant to a helpful and harmless one, the starting point is a helpful-only RLHF model, trained with human helpfulness labels and no harmlessness labels at all. For comparison, the authors also trained fresh "helpful" and "HH" RLHF baselines with human labels, so the results page can line CAI up against them.

Connections

The two stages built on this premise are critique and revision and RL from AI feedback. The judging ability measured here is what makes AI labels usable at all; chain of thought is how the paper keeps those labels readable. The "you can't avoid choosing principles" argument grows into the 2026 document; see the bridge and reasoning over rules.

Further reading, verified

• A General Language Assistant as a Laboratory for Alignment — Askell et al. (2021) · the HHH framing and the original 221 comparisons.
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback — Bai et al. (2022) · the HH RLHF model whose evasiveness CAI set out to fix.
• Red Teaming Language Models to Reduce Harms — Ganguli et al. (2022) · source of the red-team prompts and the harm labels in App. B.
• Measuring Progress on Scalable Oversight for Large Language Models — Bowman et al. (2022) · cited for humans and AI supervising together.
• Deep Reinforcement Learning from Human Preferences — Christiano et al. (2017) · the RLHF method CAI "improves upon, and partially replaces".