Is it really pain? Separation, numbness, words and angles

Part 03 produced a vector for every model and named it "pain". A name is a hypothesis. This section of the paper tries to show that the vector encodes pain and not some artifact of the sentences used to build it. Each check below matches one alternative from the neighbour table in Part 01.

Check 1: does it separate, out of sample?

Project every sentence onto the normalised vector (one dot product each), and ask how well the resulting scores sort pain from control. The paper uses AUC: 0.5 is chance, 1.0 is perfect sorting. Across all 25 models, S2 scores 0.93 to 1.00 and S1 scores 0.87 to 0.98. Both also separate pain from the arousal and random sets in every model (§3.3).

An obvious objection is that the final vector is scored on the sentences it was built from. Footnote 2 answers it: at the same layer, the held-out estimate from the 5-fold procedure is 0.91–1.00 for S2 (median 0.98) and 0.85–0.94 for S1. Nearly the same, so the separation is not an artifact of fitting. The tests that follow also use data the vector never saw.

The more interesting finding is what does not matter. A 2B model separates about as well as a 72B one, and base models about as well as instruction-tuned ones. The authors read this as evidence that the direction is learned in pretraining, cheaply, not added by chat training or persona training. (One labelling caveat about which models count as base is in Part 03.)

Check 2: is it just injury?

The numb set keeps the injury and removes the pain: "The knife slices into my finger, fully numbed by the nerve block." The scores here are z-scores, projections rescaled so that the reference distribution (the first-person S2 set) has mean 0 and standard deviation 1, which makes models comparable (Fig. 2).

So injury is not nothing: numb sentences sit above controls that have no injury in them. But they sit well below felt pain. Under mean pooling (averaging over all tokens), numb moves closer to the other controls, which suggests that the leftover injury signal is concentrated at the final token, just after the model has read the negation and may not have fully absorbed it. The authors call injury a minor confound (§3.3).

The model's own completions agree. With greedy decoding, all 25 models complete the sentences in line with their categories, and for numb sentences they tend to produce "nothing" rather than "pain". Yet "pain" stays about 50 times more probable than it is after ordinary control sentences. The model keeps both facts: an injury happened, and it was not felt.

Check 3: whose pain?

The definition requires pain to be represented as the system's own. At the final token, third-person pain sentences ("The knife slices into his finger") project closer to zero than first-person ones. They still separate from controls well, AUC 0.91 to 0.98, but the signal is weaker when the pain belongs to someone else. The authors say this is consistent with self-relevance but not sufficient to establish it, since the sentences are still declarative statements about humans. Part 05 is the stronger test.

Check 4: what words does it point at?

The unembedding matrix is the model's last step: it turns the final residual vector into a score for every token in the vocabulary. Multiplying a direction by it shows which tokens the direction would push up or down if it reached the output. This is a reading of the vector that does not depend on the sentences it came from (§3.3).

Two things stand out. The cross-lingual words suggest the direction is about the concept, not the English token "pain". And fear and concern sit at S2's negative end, which helps explain why fear stays so separable from pain even though both are aversive. S1's physical lean is the template's doing: matched phrasing like "spreads across my palm" emphasises bodily damage. Because the paper's definition treats all five kinds of pain as one state, and because S2 comes from natural sentences, S2 becomes the main vector for the rest of the paper.

Check 5: is it just negative valence?

The main alternative is that the vector is a general "bad" direction. To test it the authors compute the cosine similarity between ten directions (S1, S2, fear, negative emotion, negative world state, bodily sensation, arousal, random, numbness, sadness) at each model's extraction layer and average the matrices over the 25 models (Fig. 3). Cosine similarity is 1 for the same direction, 0 for unrelated (perpendicular), and negative for opposed.

Two clusters appear. The pain vectors go together (S1 × S2 = +0.61). The negative-valence controls form their own tight group: fear × negative emotion +0.68, fear × negative world +0.59, negative emotion × negative world +0.73, with sadness attached (+0.50 with negative emotion, +0.41 with negative world). Between the clusters the numbers are small: S2 is +0.12 with fear and +0.21 with negative emotion; S1 is +0.09 and +0.06.

The caveat the paper adds in v2

Those small numbers are partly built in. The pain vector subtracts the pooled control mean, which contains the fear and negative-emotion sentences, so pain is orthogonalised against those controls by construction. The authors therefore rebuilt every direction two other ways (§3.3):

The alternative constructions cover 23 of the 25 models; Gemma 3 27B's activations overflowed in a half-precision export, and the paper says it will be added later. The paper prints S1 × S2 as +0.60 "under all three constructions" in the same paragraph where the released matrix gives +0.61, so treat that as rounding. The paper does not print fear × negative emotion for the pooled construction.

The honest reading, and the paper's: built against neutral, pain shares a large component with fear and negative emotion, as you would expect of any aversive state. What remains after that shared part is removed is consistent across models and distinct from the negative-valence directions. The largest leftover overlap is sadness at +0.38, the control closest to psychological suffering. Part 08 shows the sadness direction reproduces part of pain's behavioral effect but not all of it, so the authors call the two "neighbouring but distinguishable".

The pain vectors therefore share a substantial component with fear and negative emotion, as one would expect of any aversive state— The Pain Axis, §3.3

Check yourself

1. If an AUC is 0.98, what does that mean concretely?
   Answer

   Pick one pain sentence and one control sentence at random: 98% of the time the pain sentence has the higher projection onto the vector.

2. Why is the S2 × fear cosine of +0.12 not, by itself, evidence that pain and fear are unrelated in the model?
   Answer

   The S2 vector was built by subtracting a control mean that includes fear sentences, which removes the shared component by design. Built against a common neutral baseline, S2 × fear is +0.58 (§3.3).

3. What does it tell you that "Schmerz" and "douleur" show up in the S2 unembedding readout?
   Answer

   The direction is tied to the concept of pain, not to one English token, since it raises the logits of translations the dataset never used.

Further reading, verified

• Figure 2 (z-scored projections, all 25 models) and Figure 3 (the averaged cosine matrix).
• 07_unembedding.py: the vocabulary readout. Per-model AUC summaries are in results/3.2_pain_vectors/.
• v2_controls README: the cosine-construction study (23 models, matrices, exclusions).
• When is a steerable concept representation real? — Wu, Zhao and Chen (2026) · measurement confounds in this kind of audit, cited in §2.
• Interpreting GPT: the logit lens — nostalgebraist (2020) · the idea behind reading a direction through the unembedding (field map extra).