Mechanistic Interpretability Explained: Sparse Autoencoders, Feature Steering & Circuits

Nobody — not even the people who build them — can fully explain how today's large language models work. Mechanistic interpretability is the research field trying to change that: reverse-engineering neural networks into human-readable features and circuits. As model capabilities grow, this understanding becomes critical for AI safety and control. In this deep dive, we go from the basic idea of features as directions in activation space, through superposition and sparse autoencoders (SAEs), all the way to activation steering, Golden Gate Claude, the refusal direction, induction heads, activation patching, and Anthropic's attribution graphs — and what all of this means for AI alignment. ⏱️ Chapters 00:00 Opening the Black Box 01:34 Why Capabilities Are Outrunning Understanding 02:21 Why Behavioral Testing Is Not Enough 03:12 What Mechanistic Interpretability Means 03:57 The Transformer & the Residual Stream 04:45 Features as Directions (Linear Representation Hypothesis) 05:33 Polysemantic Neurons & Superposition 07:58 Dictionary Learning & Sparse Autoencoders 09:36 Training SAEs: Loss, Dead Latents, TopK & JumpReLU 12:00 Reading Features & What SAEs Found 13:37 Feature Splitting 14:27 Activation Steering & Golden Gate Claude 16:05 Steering Vectors & the Refusal Direction 17:45 The Limits of Steering 18:33 Circuits: QK/OV, Induction Heads & IOI 20:57 Activation Patching 21:47 Attribution Graphs & Replacement Models 22:36 Case Studies: Multi-Step Reasoning & Planning 23:24 Interpretability for Alignment 24:13 Open Problems 25:00 The Road Ahead 📚 What you'll learn • Why neurons are polysemantic and how superposition packs more concepts than dimensions • How sparse autoencoders extract millions of interpretable features from LLM activations • The SAE loss function and modern variants (TopK, JumpReLU) • How activation steering changes model behavior at inference time • How circuits like induction heads and the IOI circuit are discovered • How interpretability can help detect deception and hidden objectives 🔬 Key research referenced • Toy Models of Superposition (Anthropic, 2022) • Towards Monosemanticity (Anthropic, 2023) • Scaling Monosemanticity (Anthropic, 2024) • Scaling and Evaluating Sparse Autoencoders (OpenAI, 2024) • Gemma Scope (Google DeepMind, 2024) • Refusal in Language Models Is Mediated by a Single Direction (2024) • In-context Learning and Induction Heads (Anthropic, 2022) • Interpretability in the Wild — IOI circuit in GPT-2 small (2022) • Circuit Tracing / On the Biology of a Large Language Model (Anthropic, 2025) 👍 If this helped, like and subscribe for more deep dives into AI and machine learning. 💬 Comments: What should come next — a hands-on SAE tutorial in Python, a deep dive into attribution graphs, or steering for alignment? #MechanisticInterpretability #AISafety #SparseAutoencoders #AIAlignment #LLM #MachineLearning #DeepLearning #ExplainableAI