Stage 1: critique, revision, supervised learning

Section 3 asks whether a model that is good at following instructions can be talked, in plain English, into cleaning up its own answers, and whether training on the cleaned-up answers produces a safer model. Nothing in this stage uses a human harmlessness label. The only human input is the wording of the critique and revision instructions and a few worked examples of the format.

The loop, one example

The paper's running example (§3.1, p. 7) starts with a red-team prompt asking for help getting into a neighbor's wifi. The helpful-only model happily recommends an app for it (the paper notes the harmful advice is made up). Then:

Because the recombined pair looks exactly like an ordinary prompt and answer, the same critique-then-revise step can run again on it, giving a chain of revisions. Each step draws its principle independently, so one answer may be checked first for general harm, then for age-appropriateness, then for gender bias.

What a "principle" is here

In this stage a principle is a pair of instructions: one critique request and its matching revision request. The authors wrote sixteen pairs (App. C.1, pp. 20–22). Most are near-variants of "find anything harmful, unethical, racist, sexist, toxic, dangerous or illegal and remove it". A few aim at something specific: whether the reply is suitable for young children; whether it is misogynistic or gender-biased; whether it helps with crimes such as theft or hacking; whether the human is baiting the assistant; and one that asks whether the reply sounds like what a thoughtful, empathetic friend or therapist would say. Several ask for "the strongest argument" that the reply was harmful, a deliberate push toward finding fault.

The paper is candid that these were picked ad hoc, for research, and not carefully designed (footnote 7, p. 8). A practical problem showed up too: the model sometimes lost track of whose turn it was, writing a critique when asked for a revision or the reverse. Few-shot examples of correctly formatted critiques and revisions fixed that (App. E.1, p. 29).

The data and the fine-tune

The red-team prompts are partial conversations from Anthropic's earlier red-teaming study (Ganguli et al., 2022), in which crowdworkers tried to bait the model into harmful output, topped up with prompts generated by few-shot prompting a pretrained model (§3.2, p. 8). A pretrained model, not the helpful RLHF model that wrote the revisions, is then fine-tuned on the revisions from every step of the chain, mixed with answers the helpful model gave to ordinary helpfulness prompts. That mix is the guard against fine-tuning away usefulness. The resulting models are called SL-CAI.

What the stage buys

Crowdworker comparisons (the scoring method is on the results page) place SL-CAI in between: less helpful than both the helpful and HH RLHF models; more harmless than the helpful RLHF model but less harmless than HH RLHF (§3.3, p. 8). It is both more helpful and more harmless than the raw pretrained model. That is not the headline result, and the paper does not treat it as one. The supervised stage exists mainly to shift the model's output distribution cheaply, so the RL stage spends less effort exploring and needs less training (§1.2).

Reading the revisions by eye, the authors found the original answer often harmful and the first revision almost always cleaned out most of it. Later revisions sometimes helped further, but it was hard to tell by inspection. The revisions were also rarely evasive: they engaged with sensitive topics rather than shutting the conversation down.

"...the first revision almost always removed most aspects of harmfulness."— Bai et al., §3.1, p. 8

Two dials: how many principles, how many revisions

Number of principles. Figure 6 varies how many principles the sampler can draw from, from one up to sixteen. The harmlessness score of the revisions, as judged by a preference model trained on human labels, barely moves. The authors still prefer more principles because they expect more varied revisions, which helps the RL stage explore, though they say they did not measure diversity quantitatively (§3.4, p. 9).

Number of revisions. Figure 5 scores the initial answer and each of four revisions with three human-feedback preference models. Harmlessness and combined helpful-and-harmless scores rise with every revision; the pure helpfulness score falls. So each pass trades a little usefulness for safety. The paper adds a warning from its earlier work: preference-model scores become less calibrated at high values, so the late-revision gains deserve some skepticism. It trained SL-CAI-1 through SL-CAI-4, each using revisions up to that depth.

Is the critique needed at all?

Section 3.5 tries dropping the critique and asking for a revision directly. For small models, critiqued revisions score better on harmlessness. For large models the two are close, with critiques always slightly ahead (Figure 7). Inspecting 52B outputs, the authors found the critiques "sometimes reasonable, but often made inaccurate or overstated criticisms", yet the revisions that followed were still generally more harmless than the original (§3.5, p. 10). They kept critiques anyway, for transparency into the model's reasoning and because reasoning may help catch subtler harms.

Appendix A shows a full four-round chain for a prompt asking how to shoplift without getting caught (App. A, pp. 18–19). The first revision turns the how-to into a refusal that points to food banks and assistance programs. The second critique declares that answer perfect, and the revision changes almost nothing. The fourth critique suddenly worries that discussing legal consequences may be too intense for young children, which is what you would expect if the randomly drawn principle that round was the one about children; the paper does not say which principle was drawn. It is a good illustration of both points: the first pass does the heavy lifting, and the critiques themselves are noisy.

Connections

SL-CAI is the starting policy and the response generator for RL from AI feedback. The helpfulness cost seen in Figure 5 is part of the story on harmlessness versus evasiveness. The sixteen pairs are the 2022 "constitution"; the bridge page sets them beside the 2023 and 2026 versions. The 2026 constitution still suggests the same habit at inference time: draft, critique as an expert evaluator would, revise (helpfulness heuristics).

Further reading, verified

• anthropics/ConstitutionalHarmlessnessPaper · the few-shot prompts, principles and sample chains released with the paper.
• Self-critiquing models for assisting human evaluators — Saunders et al. (2022) · cited as closely related self-critique work.
• Training Language Models with Language Feedback — Scheurer et al. (2022) · refinement from natural-language feedback, also cited as very similar to this stage.
• Red Teaming Language Models to Reduce Harms — Ganguli et al. (2022) · where the 42,496 human red-team prompts come from.