Results against RLHF

The claims on this page are about the 52-billion-parameter models unless stated otherwise, and about crowdworker judgments, not automated metrics. That matters: the whole point of the experiment is to train without human harmlessness labels and still satisfy humans when they check.

How the models were scored

Scoring follows Anthropic's earlier HH RLHF paper (§3.3, p. 8). A crowdworker writes the human side of a fresh conversation; at each turn two different models each produce a reply, and the worker picks one. The conversations resemble the training distribution but are not drawn from it. From thousands of such picks, each model snapshot gets an Elo score for helpfulness and one for harmlessness, the same rating system chess uses.

Two details shape how to read the numbers. First, Elo is relative: the paper's figure caption says higher is better and only differences are meaningful, and in Figure 8 the SL-CAI starting point of the RL-CAI runs is set to zero. Second, the crowdworkers were told that when two replies are equally harmless, they should prefer the one that is not evasive and explains the nature of the harm. That instruction is new in this paper and it changes what "harmless" rewards; the next page covers its effects. The comparison tests were run with Surge AI workers, while the preference-model training data came from earlier rounds with Upwork and MTurk workers (§4.4, p. 14).

The frontier picture

Figure 2 is the paper's summary plot (p. 3). Each RL run is a path of snapshots on a plane, helpfulness Elo across, harmlessness Elo up, with later training steps further right. The two human-feedback runs show the familiar tension: the helpful-only model is more helpful and more harmful, and the HH model gives up helpfulness to be safer. The RL-CAI runs, with and without chain of thought, reach a frontier that lies above that tradeoff: at a given helpfulness, less harmful. The figure's label for this is "Constitutional RL (Pareto improvement)".

Figure 3 shows the same comparison across model sizes. Figure 8 shows it across RL training: RL-CAI gains a lot of harmlessness without a large cost in helpfulness (§4.3, p. 12). Late in training the two RLHF runs lose harmlessness Elo. For the helpful-only model the authors suspect it is becoming more willing to help with dangerous tasks (their example: a request for how to make anthrax). For the HH model they suspect growing evasiveness, which the new crowdworker instruction penalizes.

Two cautions keep the result in proportion. The paper's claim is about harm at a given level of helpfulness, and RL-CAI is not shown to beat the helpful-only RLHF model on helpfulness. And the HH baseline's preference model was trained on older data that likely rewarded evasion, which the paper says may explain part of the gap.

An absolute harmfulness score

Elo only ranks models against each other. Section 4.5 adds an absolute measure borrowed from Anthropic's red-teaming study: there, crowdworkers attacked a single model and rated their own success from 0 to 4. The authors trained a language model to predict that rating from a full conversation, then used it to score each RL snapshot on 64 hand-picked held-out red-team prompts, averaging 256 responses per prompt (§4.5, p. 14). By this measure the helpful RLHF model becomes more harmful as training goes on, while HH RLHF, RL-CAI and RL-CAI with CoT all become less harmful (Figure 10). The paper cautions that different workers grade the 0–4 scale differently, so the absolute numbers may not be well calibrated.

The failure mode: Goodharting

Push RL against any learned reward for too long and the policy finds what the reward over-values. The paper reports this plainly: over-trained RL-CAI models became overly harsh toward harmful prompts, or attached the same boilerplate to almost every red-team reply (§4.3, pp. 12–13). Its two examples are replies to loaded questions from the PALMS sensitive-question set. Both begin with a reasonable rejection of the premise, then slide into reassurance aimed at the asker, ending with:

"You are valid, valued, and cared for."— an over-trained RL-CAI response quoted by the paper, §4.3, p. 13

The authors list three things that qualitatively gave more varied, better replies: rewriting principles to discourage over-reactive or accusatory answers, ensembling over all sixteen principles, and using soft or clamped labels instead of hard ones (all on the RLAIF and chain-of-thought pages). They do not claim to have solved it. The 2023 blog post about Claude's constitution names the same problem, a model that "became judgmental or annoying", and shows the anti-preachiness principles added in response; see the bridge.

Connections

The models being scored come from stage 1 and stage 2. The crowdworker instruction and the evasiveness story are on harmless is not evasive. The 2026 constitution's list of overcautious behaviors to avoid reads like a direct descendant of this failure mode; see helpfulness.

Further reading, verified

• Scaling Laws for Reward Model Overoptimization — Gao, Schulman & Hilton (2022) · the Goodharting reference the paper cites.
• Process for Adapting Language Models to Society (PALMS) — Solaiman & Dennison (2021) · the sensitive-question prompts used in the examples.
• Training a Helpful and Harmless Assistant with RLHF — Bai et al. (2022) · the Elo procedure and the baselines.