Anthropic have been working on the interpretability of neural networks for a long time. Their past SAE (Sparse Autoencoder) method has already been adopted by OpenAI and Google, and now they offer a new way to "parse" AI into thoughts - Circuit Tracing.
đ˘ How does it work?
đ They take an off-the-shelf language model and select a task.
đ Replace some components of the model with simple linear models (Cross-Layer Transcoder).
đ Train these replaced parts to mimic the original model, minimizing the difference in output.
đ Now you can see how information "flows" through all the layers of the model.
đ Based on this data, an attribution graph is built - it shows which attributes influence each other and form the final answer.
đ˘ What interesting things were discovered in Claude's brain?
đ The LLM "thinks ahead." For example, when she writes a poem, she plans the rhyme scheme in advance, even before she starts a new line.
đ Math is not just about memorization. Turns out the model is actually calculating, not just retrieving memorized answers.
đ Hallucinations have a cause. A specific "answer is known" trigger is found. If it is triggered in error - the model starts making things up.
đ Fun fact: if you tell the model the answer to a problem right away, it will think backwards - come up with a plausible path to that answer.
#claude #AI