a clockwork watch may be ticking normally, such that it’s very hard to tell that it is likely to break down next month, but opening up the watch and looking inside can reveal mechanical weaknesses that allow you to figure it out.
for - AI - progress trap - interpretability testing - deception - Is it possible that AI could even change their node behavior as a deceptive move?
similar to how neuroscientists study animal brains by correlating measurement and intervention to external stimuli and behavior
for - AI - testing - interpretability - neuroscience techniques
science of interpretability
for - definition - interpretability - Claude training
Examples of mistakes where we can use attention to gain intuition into what the model saw.
Perhaps the best use of this approach is for looking for mistakes or understanding why a model does badly on certain data instances.
Title: Delays, Detours, and Forks in the Road: Latent State Models of Training Dynamics Authors: Michael Y. Hu1 Angelica Chen1 Naomi Saphra1 Kyunghyun Cho Note: This paper seems cool, using older interpretable machine learning models, graphical models to understand what is going on inside a deep neural network
The code to reproduce our results can be found here.
This input embedding is the initial value of the residual stream, which all attention layers and MLPs read from and write to.
Starting from random noise, we optimize an image to activate a particular neuron (layer mixed4a, unit 11).
And then we use that image as a kind of variable name to refer to the neuron in a way that more helpful than the the layer number and neuron index within the layer. This explanation is via one of Chris Olah's YouTube videos (https://www.youtube.com/watch?v=gXsKyZ_Y_i8)
Moreau, David, and Kristina Wiebels. ‘Assessing Change in Intervention Research: The Benefits of Composite Outcomes’, 2 June 2020. https://doi.org/10.31234/osf.io/t9hw3.
To interpret a model, we require the following insights :Features in the model which are most important.For any single prediction from a model, the effect of each feature in the data on that particular prediction.Effect of each feature over a large number of possible predictions
Machine learning interpretability