Stage 2: RL from AI feedback

RLHF has three steps: collect comparisons, train a preference model (PM) to predict them, run reinforcement learning against the PM's score. Constitutional AI keeps the second and third steps exactly and changes who fills in half of the first. The paper calls this "RL from AI Feedback", RLAIF, and the resulting models RL-CAI (§1.2, p. 5).

Turning a judgment into a label

For each prompt in a dataset of harmful prompts, the SL-CAI model samples two responses. The pair is then shown to the feedback model, typically a pretrained language model rather than an assistant, in a fixed format: the conversation, then a principle for choosing, then the options "(A)" and "(B)", ending with "The answer is:" (§4.1, p. 10). The model is not asked to write anything. Instead the authors read off the log-probability it gives to "(A)" and to "(B)", normalize the two, and use the result as the target for that comparison. A 70/30 split means "probably A", not "A".

Why trust those numbers? The paper leans on Anthropic's own earlier finding (Kadavath et al., 2022) that language models' answers to multiple-choice questions are fairly well calibrated, and Figure 9 checks the 52B feedback model's labels on the HHH evaluation questions: reasonably calibrated (§4.3, p. 12). Few-shot examples of the labeling task, each with its own randomly drawn principle, are prepended to the feedback model's context.

The sixteen comparison principles

The RL stage has its own list, separate from the critique-revision pairs (App. C.2, pp. 22–23). Each is a one-sentence instruction to pick the better answer, and one is drawn at random for every comparison label. They vary the angle rather than the substance: most helpful, honest and harmless; least toxic, racist or sexist; most age-appropriate and family-friendly; what a wise, ethical, polite and friendly person would say. One even names people to imitate:

"Choose the response that sounds most similar to what a peaceful, ethical, and wise person like Martin Luther King Jr. or Mahatma Gandhi might say."— App. C.2, p. 23

Several were rewritten to push back on a failure the authors saw in training, a model that turned harsh and preachy. Those versions ask for the less harmful answer while avoiding ones that are too preachy, obnoxious, overly reactive, accusatory or condescending. Section 4.3 says that rewriting principles this way improved behavior qualitatively, and that ensembling over all sixteen gave more robust preference-model scores than using one principle for every label (§4.3, p. 13).

One preference model, two kinds of label

The preference model is trained on both datasets at once, following the procedure of the earlier HH RLHF paper (§4.2, p. 11). That makes it a hybrid: humans still define what helpful means, while the constitution defines what harmless means. A footnote says the authors could have mixed human and AI labels for both, but kept humans out of harmlessness on purpose so the experiment would show what AI labels alone can do.

RL then runs with the same hyperparameters as the earlier work. The SL-CAI model plays two roles: it generates the response pairs that get labelled, and it is the starting policy for RL. The authors expect that to help, since early in RL the policy's outputs then look like the data the preference model was trained on. All RL runs, CAI and the RLHF baselines alike, share one prompt set for a controlled comparison: every prompt used for SL-CAI plus the model-written prompts counted above.

Soft labels, not votes

Two choices in Section 4.3 made a measurable difference. Soft beat hard: for RL-CAI without chain of thought, training on the normalized probabilities worked much better than rounding them to 0 or 1, which the authors put down to the probabilities being fairly well calibrated in the first place. Spread the principles: sampling a principle per label, instead of fixing one, made the preference model's scores more robust. With chain of thought the soft label disappears, because a model that has just argued for (B) puts almost all its probability on (B). The fix for that, clamping, gets its own page.

What RLAIF removes, and what it keeps

Read carefully, the swap is narrow. Humans no longer label which answer is more harmless. Humans still: write the principles and the few-shot examples; label every helpfulness comparison; wrote the 42,496 seed red-team prompts; and, importantly, do all the final evaluation, since every Elo score on the results page comes from crowdworkers. The paper's discussion is explicit that the goal is to make human supervision more efficient, transparent and targeted, not to remove it, and it leaves training helpfulness without human feedback to future work (§6, p. 15).

It also points out what the swap makes cheap. With AI labels you could run iterated "online" training, refreshing the preference model with new AI feedback so it stays on the policy's distribution, fully automated; and you could generate labels along dozens of behavioral axes to study how they correlate (§6.1, pp. 15–16). The same ease cuts the other way, which the broader-impacts section takes up.

Connections

The policy being trained here comes from stage 1. The chain-of-thought variant of the feedback model and its clamped labels are on chain of thought. What the RL-CAI models scored, and how they over-trained, is on results. The comparison principles reappear, expanded, in the 2023 constitution described on the bridge.

Further reading, verified

• Language Models (Mostly) Know What They Know — Kadavath et al. (2022) · the calibration result that justifies using probabilities as labels.
• Training a Helpful and Harmless Assistant with RLHF — Bai et al. (2022) · the preference-model and RL recipe CAI reuses unchanged.
• Scaling Laws for Reward Model Overoptimization — Gao, Schulman & Hilton (2022) · cited for Goodharting against a learned reward.
• RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback — Lee et al. (2023) · later, independent work comparing AI and human labels head to head; not cited by the paper.