Anthropic have been working on the interpretability of neural networks for a long time. Their past SAE (Sparse Autoencoder) method has already been adopted by OpenAI and Google, and now they offer a new way to "parse" AI into thoughts - Circuit Tracing.
🟢 How does it work?
🍒 They take an off-the-shelf language model and select a task.
😘 Replace some components of the model with simple linear models (Cross-Layer Transcoder).
😘 Train these replaced parts to mimic the original model, minimizing the difference in output.
🍒 Now you can see how information "flows" through all the layers of the model.
😘 Based on this data, an attribution graph is built - it shows which attributes influence each other and form the final answer.
🟢 What interesting things were discovered in Claude's brain?
🟠 The LLM "thinks ahead." For example, when she writes a poem, she plans the rhyme scheme in advance, even before she starts a new line.
🟠 Math is not just about memorization. Turns out the model is actually calculating, not just retrieving memorized answers.
🟠 Hallucinations have a cause. A specific "answer is known" trigger is found. If it is triggered in error - the model starts making things up.
🟠 Fun fact: if you tell the model the answer to a problem right away, it will think backwards - come up with a plausible path to that answer.