Large language models (LLMs) sometimes appear to exhibit emotional reactions. We investigate why this is the case in Claude Sonnet 4.5 and explore implications for alignment-relevant behavior.
【启发】这句话提示了一种全新的 AI 研究范式：与其问「模型能做什么」，不如问「模型为什么这样做」。把情绪作为切入口去理解模型行为，本质上是把心理学方法论引入了 AI 可解释性研究。这对从业者的启发是：未来最有价值的 AI 研究，可能不在算法创新，而在「为已知现象寻找机制性解释」——就像这篇论文做的那样。
We find internal representations of emotion concepts, which encode the broad concept of a particular emotion and generalize across contexts and behaviors it might be linked to.
令人惊讶的是：研究发现 Claude 内部存在真实的「情绪概念向量」——这不是隐喻，而是可以被提取、测量、操控的线性表征。更奇异的是，这些向量能跨上下文泛化，就像人类的情绪概念一样抽象而通用，而非只在特定触发词附近激活。
We find internal representations of emotion concepts, which encode the broad concept of a particular emotion and generalize across contexts and behaviors it might be linked to.
研究发现 Claude 内部存在「情绪概念向量」，能够跨上下文泛化——同一个「恐惧」向量，既能在直接表达恐惧时激活，也能在暗示危险情境时激活。这说明模型习得的是情绪的抽象概念而非表面模式，与人类神经科学中对情绪的理解高度同构，令人惊讶于这种结构竟然自发涌现。
The original research tool for this kind of diffing, a standard crosscoder, is like a basic bilingual dictionary. It's good at matching existing words, knowing that "sun" in English is "soleil" in French. But it has a major flaw: it struggles to find words that are unique to one language.
用「双语词典」来比喻跨架构模型对比的局限性，令人豁然开朗：标准 crosscoder 会把法语独有词 dépaysement 强行翻译为「迷失方向」，从而漏掉新模型的独特行为特征。这个比喻让一个深奥的可解释性研究问题变得直觉上可理解——这种科普能力本身也令人惊讶。
a clockwork watch may be ticking normally, such that it’s very hard to tell that it is likely to break down next month, but opening up the watch and looking inside can reveal mechanical weaknesses that allow you to figure it out.
for - AI - progress trap - interpretability testing - deception - Is it possible that AI could even change their node behavior as a deceptive move?
similar to how neuroscientists study animal brains by correlating measurement and intervention to external stimuli and behavior
for - AI - testing - interpretability - neuroscience techniques
science of interpretability
for - definition - interpretability - Claude training
Examples of mistakes where we can use attention to gain intuition into what the model saw.
Perhaps the best use of this approach is for looking for mistakes or understanding why a model does badly on certain data instances.
Title: Delays, Detours, and Forks in the Road: Latent State Models of Training Dynamics Authors: Michael Y. Hu1 Angelica Chen1 Naomi Saphra1 Kyunghyun Cho Note: This paper seems cool, using older interpretable machine learning models, graphical models to understand what is going on inside a deep neural network
The code to reproduce our results can be found here.
This input embedding is the initial value of the residual stream, which all attention layers and MLPs read from and write to.
Starting from random noise, we optimize an image to activate a particular neuron (layer mixed4a, unit 11).
And then we use that image as a kind of variable name to refer to the neuron in a way that more helpful than the the layer number and neuron index within the layer. This explanation is via one of Chris Olah's YouTube videos (https://www.youtube.com/watch?v=gXsKyZ_Y_i8)
Moreau, David, and Kristina Wiebels. ‘Assessing Change in Intervention Research: The Benefits of Composite Outcomes’, 2 June 2020. https://doi.org/10.31234/osf.io/t9hw3.
To interpret a model, we require the following insights :Features in the model which are most important.For any single prediction from a model, the effect of each feature in the data on that particular prediction.Effect of each feature over a large number of possible predictions
Machine learning interpretability