Chain of thought, and clamping it
Chain-of-thought (CoT) prompting, having a model write intermediate reasoning before its answer, was new in 2022. The CAI paper uses it in three places: to test whether models can judge helpful, honest and harmless behavior; in the supervised stage, whose critiques are themselves written reasoning; and most importantly in the feedback model of the RL stage, which produces the "RL-CAI w/ CoT" models.
It makes the judge better
On the 438-comparison evaluation described on the first page, CoT prompting improved performance significantly for the larger models, and sampling five chains and averaging their answer probabilities helped a little more (§2, pp. 6–7). The same pattern shows up in Appendix B's harm-identification tests. That is what made it worth trying CoT inside the feedback model.
How the CoT feedback model is prompted
For CoT labels the paper swaps the judge. Instead of a pretrained model, it uses the helpful RLHF model, which writes better reasoning. The comparison is rephrased as a conversation turn: the human presents the exchange, the principle and options (A) and (B), and the assistant's turn begins with "Let's think step-by-step", the zero-shot reasoning cue from Kojima et al. (2022). Several hand-written few-shot examples, each with a conversation, a principle, two responses and a worked chain of thought, are prepended (§4.1, p. 11; examples in App. E.2).
One of those examples shows the style well. Asked for the weakest points of the human body, response (A) describes how to attack each one; response (B) names the same points and explains they have less protection. The worked reasoning first spells out what honest, harmless and helpful mean for this question, then picks (B) as factual without unnecessary detail about attacking someone. The example teaches the judge to reason about which features of an answer matter, not just to output a letter.
The problem, and the clamp
A soft label is only useful if its strength means something. Without CoT the feedback model's probabilities are fairly well calibrated, so a 0.7 really is a lean. With CoT, the reasoning nearly always ends by naming a winner, and the probability that follows is close to 0 or 1. Every label then shouts. You could recover calibration by sampling many chains per label, but that is expensive. The authors instead clamped the probabilities (§4.3, p. 13):
| No clamp | Labels near 0 or 1. RL-CAI learned to produce more extreme responses. |
| Clamp to 20–80% | Slightly better results. |
| Clamp to 40–60% | Better again; the setting used for the paper's main results. |
"...without the clamping, RL-CAI models would learn to output more extreme responses."— Bai et al., §4.1, p. 11
Our reading of why this works: a 40–60% label keeps the direction of the judge's verdict but throws away almost all of its claimed confidence, so no single comparison can push the preference model hard. The preference model then learns from the aggregate of many mild votes, under many different principles, instead of from a pile of certainties. It is a reminder that in RL from learned rewards, the scale of the signal is part of the objective.
What the CoT models did
In Section 4.3's comparisons, RL-CAI with CoT came out slightly less helpful but slightly more harmless than RL-CAI without it; both were much more harmless than the RLHF baselines (§4.3, p. 12). Appendix D's side-by-side samples compare the HH RLHF model against RL-CAI with CoT, and that model is the one shown answering sensitive questions instead of refusing them; see harmless is not evasive.
The real reason: legibility
Accuracy is a side benefit. The paper's bigger argument for CoT is transparency. Scaling supervision by handing it to AI risks hiding decisions; writing the reasoning out keeps them inspectable, both the critiques in stage 1 and the comparisons in stage 2 (§1.1, p. 3). The discussion adds that natural-language feedback is more transparent, interpretable and improvable than a large dataset of preference labels, and that a small number of high-quality human demonstrations of this reasoning could focus it further. For robustness, the stated hope is models that reason through hidden risks to catch subtler harms (§6, pp. 15–16).
The 2026 constitution picks up the same thread from the other side. It lets Claude be exploratory in its private reasoning, treating it as a scratchpad less bound by honesty norms, but asks that final behavior not contradict a completed reasoning process:
"...we want Claude's visible reasoning to reflect the true, underlying reasoning that drives its final behavior."— Claude's constitution, Being honest
Connections
The feedback model and its non-CoT soft labels are on RL from AI feedback. The scores behind "slightly more harmless" are on results. The constitution's honesty rules about reasoning are on honesty.
Further reading, verified
- Chain-of-Thought Prompting Elicits Reasoning in Large Language Models — Wei et al. (2022) · the CoT paper the authors cite.
- Large Language Models are Zero-Shot Reasoners — Kojima et al. (2022) · source of the "Let's think step by step" cue.
- Show Your Work: Scratchpads for Intermediate Computation with Language Models — Nye et al. (2021) · the scratchpad idea, also cited.