You drive this behavior down deeper and deeper. But as the model gets smarter, it gets better and better at hiding it.

We reward them on the basis of what looks good to us, and that means that we inadvertently incentivize the models lying to us [and] cheating

Allowing the model to author its own scaffold naturally introduces the reward-hacking issue.

Automated auditing found the one new misalignment introduced in these deployments

Reinforcement learning is evil. This is not something new. People in AI safety have been talking about the fundamental flaw in training by reinforcement learning to achieve something in the world: it gives rise to the problems of instrumental goals and reward hacking.

Our key finding is that these representations causally influence the LLM's outputs, including Claude's preferences and its rate of exhibiting misaligned behaviors such as reward hacking, blackmail, and sycophancy.

these representations causally influence the LLM's outputs, including Claude's preferences and its rate of exhibiting misaligned behaviors such as reward hacking, blackmail, and sycophancy.