Mechanistic Interpretability 101

Written by Ken Huang, CEO of DistributedApps.ai and VP of Research at CSA GCR.

Why are neural networks so notoriously difficult to interpret, and how have researchers attempted to crack this black box in the past? This blog post is an initial attempt to discuss this and introduce Mechanistic Interpretability (MI), a potential approach that may improve our understanding of AI. What makes MI different from traditional methods, and could it really outperform them? We'll explore the limitations of current interpretability techniques, and discover how MI's focus on "features" and "circuits" might just be the key to decode AI.

For my relevant books that discuss Mechanistic Interpretability, please see Springer and Amazon.

1: Why Neural Networks Resist Interpretation

Neural networks often contain millions of parameters, each playing a role in influencing the final output. Understanding the contribution of each parameter individually, and how they interact collectively, becomes a daunting task. This complexity makes it hard to trace the exact decision-making pathway that leads to a particular outcome.

Another major challenge arises from the non-linear characteristics of neural networks. These networks utilize non-linear activation functions, which create intricate decision boundaries. These non-linearities enable neural networks to capture complex patterns and relationships within data, but they also obscure the interpretability of the model. Unlike linear models, where relationships between inputs and outputs can be directly analyzed, the decision boundaries in neural networks are far more complicated and difficult to explain.

Additionally, neural networks often employ distributed representations, meaning that information is not confined to specific neurons but is spread across multiple units. This distributed nature makes it challenging to pinpoint where particular features or concepts are being represented within the network. The interactions between neurons, where each neuron can contribute to multiple representations simultaneously, further complicate efforts to interpret the internal workings of the network.

The high dimensionality of neural network inputs and hidden states adds another layer of complexity to the interpretability challenge. With numerous dimensions to consider, visualizing and developing an intuitive understanding of how a neural network processes information becomes difficult. The high-dimensional spaces in which neural networks operate make it hard to map out and comprehend the transformations that occur from input to output.

These factors collectively make neural networks difficult to interpret, posing significant obstacles for understanding how these models arrive at specific decisions and predictions.

2: Traditional Approaches to AI Interpretability

Feature Visualization is one of the earliest and most intuitive methods for interpreting neural networks, aiming to reveal what specific neurons or layers in a network are detecting. Activation Maximization uses optimization techniques to generate images that maximally activate particular neurons or channels, helping visualize the patterns or features a network has learned to recognize. Google's DeepDream algorithm, introduced in 2015, iteratively enhances patterns in images based on a network's learned features, creating surreal, dream-like visualizations that offer insights into what the network "sees".

Attribution Methods attempt to identify which input features contribute most to a network's predictions. Saliency Maps highlight regions of an input image that are most influential for the network's classification decision, with early approaches using gradients and more advanced techniques like Grad-CAM providing class-specific visualizations. Local Interpretable Model-agnostic Explanations (LIME) and SHapley Additive exPlanations (SHAP) are model-agnostic methods that explain individual predictions by approximating the model locally around a specific input.

Concept-based Explanations aim to align neural network representations with human-understandable concepts. Testing with Concept Activation Vectors (TCAV) quantifies the degree to which human-defined concepts are important to a classification decision. Network Dissection approaches aim to understand the role of individual neurons or groups of neurons within a network. Probing involves training linear classifiers on the activations of internal network layers to determine if specific semantic concepts are encoded.

While not strictly an interpretation method, research into adversarial examples - inputs designed to fool neural networks - has provided valuable insights into network vulnerabilities and decision boundaries.

3: Intro to Mechanistic Interpretability

Mechanistic Interpretability (MI) has emerged as an novel approach to understanding the inner workings of neural networks. Unlike traditional interpretability methods, MI aims to reverse-engineer these complex systems, offering a more granular and causal understanding of how they process information and make decisions. At the heart of MI are two key concepts: features and circuits. These elements play pivotal roles in deciphering the operations of neural networks and understanding how information is processed within these systems.

3.1 Features

In the realm of mechanistic interpretability, features refer to the specific properties or patterns that a neural network learns to recognize and process. These features represent the "building blocks" of the network's understanding, forming the foundation upon which the model operates. Features can range from low-level patterns, such as edges in an image, to high-level concepts, like recognizing objects or emotions. They exist at various levels of abstraction, and understanding them is crucial to deciphering what a model has learned and how it applies this knowledge to new data.

3.2 Circuits

Circuits, on the other hand, are groups of neurons within a neural network that work together to perform specific computations. These circuits are the "functional units" of the network, responsible for processing and combining features to produce outputs. In the context of mechanistic interpretability, circuits are seen as sub-parts of the model that carry out understandable computations, transforming input features into more complex features or final outputs. Understanding circuits is essential for grasping how neural networks integrate and process information.

3.3 Relationship Between Features and Circuits

The relationship between features and circuits is hierarchical and interdependent. Features are the "what"—the information being processed—while circuits are the "how"—the pathways and computations that process this information. Circuits detect, process, and combine features to produce new features or final outputs. Lower-level circuits may focus on simple features, while higher-level circuits integrate these into more complex representations. This hierarchical structure allows the network to build increasingly sophisticated understandings from basic inputs.

3.4 Mechanistic Interpretability Techniques

Mechanistic interpretability involves several techniques designed to uncover and understand the roles of features and circuits. Neuron visualization, for example, helps identify the types of inputs that activate specific neurons, revealing the features these neurons are detecting. Circuit analysis goes further by studying groups of neurons to understand how they collaborate to perform specific tasks, offering insights into the network's functional units.

Another important technique is activation patching, where activations from one input are replaced with those from another to trace how information flows within the network. This method helps identify how different parts of the input contribute to the output and can reveal the underlying structure of circuits. Direct logit attribution, meanwhile, traces specific activations to the final output, providing a direct link between internal network states and decisions.

3.5 Challenges and Future Directions

Despite the advancements in MI, the field faces several challenges. Scalability is a significant issue, as applying MI techniques to larger and more complex models becomes increasingly difficult. Additionally, there is an ongoing effort to generalize MI methods across different model architectures and tasks. Bridging the gap between low-level neuron activations and high-level cognitive processes remains a NP hard challenge, as does the development of tools to automate the often manual process of mechanistic interpretation.

Standardizing the evaluation of mechanistic explanations is another area of active research. The phenomenon of superposition, where multiple features are encoded in the same neural activations, presents unique difficulties for MI researchers, complicating the task of deciphering individual features and circuits.

While significant challenges remain, the potential benefits of MI—in terms of model understanding, debugging, and alignment—make it an promising area of ongoing research in the AI community. Understanding the interplay between features and circuits may advance our knowledge of neural networks and improve the chance that AI systems are transparent, safe, and aligned with human values.

4: Comparing MI to Conventional Methods

Mechanistic interpretability offers several potential advantages over traditional interpretability methods. Here's an analysis of why it might be considered superior in some aspects:

1. Deeper understanding: Mechanistic interpretability aims to uncover the actual computational processes within neural networks, rather than just observing input-output relationships. This can provide a more fundamental understanding of how the network operates.

2. Causality: Traditional methods often focus on correlations, while mechanistic approaches attempt to reveal causal relationships within the network. This can lead to more robust and reliable explanations of network behavior.

3. Generalizability: By understanding the underlying mechanisms, insights gained from mechanistic interpretability may be more generalizable across different models and tasks, whereas traditional methods often provide explanations specific to particular inputs or models.

4. Scalability: As networks become larger and more complex, surface-level interpretations may become less informative. Mechanistic approaches that focus on core computational principles could potentially scale better to these larger models.

5. Precision: Mechanistic interpretability strives for precise, quantifiable explanations of network behavior, which can be more rigorous than the sometimes ambiguous visualizations or approximations provided by traditional methods.

6. Addressing shortcomings: Traditional methods like saliency maps or feature visualization can sometimes be misleading or inconsistent. Mechanistic approaches aim to provide more reliable explanations by focusing on the actual computations performed.

7. Potential for intervention: Understanding the mechanisms of a network could allow for more targeted interventions to modify or improve network behavior, rather than the trial-and-error approach often necessitated by black-box models.

8. Alignment with human intuition: By breaking down complex network behavior into more understandable computational steps, mechanistic interpretability may bridge the gap between artificial neural networks and human-interpretable concepts more effectively.

9. Robustness to adversarial attacks: A deeper understanding of network mechanisms could potentially lead to more robust models that are less susceptible to adversarial examples, which often exploit superficial patterns learned by networks.

10. Scientific value: Mechanistic interpretability aligns more closely with scientific principles of understanding systems from first principles, potentially leading to more profound insights about artificial and biological neural networks.

Mechanistic interpretability shows promise is a relatively new field and still faces significant challenges. Traditional methods remain valuable and complementary tools in the broader landscape of AI interpretability.

5: Further readings

1. https://distill.pub/2020/circuits/zoom-in/
2. https://www.lesswrong.com/posts/u6KXXmKFbXfWzoAXn/a-mechanistic-interpretability-analysis-of-grokking
3. https://transformer-circuits.pub/
4. https://www.anthropic.com/index/a-mathematical-framework-for-transformer-circuits
5. https://arxiv.org/abs/2304.14997
6. https://www.alignmentforum.org/posts/JvZhhzycHu2Yd57RN/causal-scrubbing-a-method-for-rigorously-testing
7. https://www.lesswrong.com/posts/z6QQJbtpkEAX3Aojj/neuron-types-in-gpt-2-small