Mechanistic Interpretability for AI Safety — A Review

# Mechanistic Interpretability for AI Safety — A Review

A comprehensive review of mechanistic interpretability, an approach to reverse engineering neural networks into human-understandable algorithms and concepts, focusing on its relevance to AI safety.

### Authors

### Affiliations

[Leonard Bereska](https://leonardbereska.github.io/)

University of Amsterdam

[Efstratios Gavves](https://www.egavves.com/)

University of Amsterdam

### Published

July 10, 2024

Understanding AI systems’ inner workings is critical for ensuring value alignment and safety. This review explores mechanistic interpretability: reverse engineering the computational mechanisms and representations learned by neural networks into human-understandable algorithms and concepts to provide a granular, causal understanding. We establish foundational concepts such as features encoding knowledge within neural activations and hypotheses about their representation and computation. We survey methodologies for causally dissecting model behaviors and assess the relevance of mechanistic interpretability to AI safety. We examine benefits in understanding, control, and alignment, along with risks such as capability gains and dual-use concerns. We investigate challenges surrounding scalability, automation, and comprehensive interpretation. We advocate for clarifying concepts, setting standards, and scaling techniques to handle complex models and behaviors and expand to domains such as vision and reinforcement learning. Mechanistic interpretability could help prevent catastrophic outcomes as AI systems become more powerful and inscrutable.

## Introduction

As AI systems rapidly become more sophisticated and general

- **Sparks of Artificial General Intelligence: Early experiments with GPT-4** [\[PDF\]](http://arxiv.org/pdf/2303.12712.pdf)

S. Bubeck, V. Chandrasekaran, R. Eldan, J. Gehrke, E. Horvitz, E. Kamar, P. Lee, Y.T. Lee, Y. Li, S. Lundberg, H. Nori, H. Palangi, M.T. Ribeiro, Y. Zhang.

CoRR. 2023.

[DOI: 10.48550/arXiv.2303.12712](https://doi.org/10.48550/arXiv.2303.12712)
- **Managing AI Risks in an Era of Rapid Progress** [\[PDF\]](http://arxiv.org/pdf/2310.17688.pdf)

Y. Bengio, G. Hinton, A. Yao, D. Song, P. Abbeel, Y.N. Harari, Y. Zhang, L. Xue, S. Shalev-Shwartz, G. Hadfield, J. Clune, T. Maharaj, F. Hutter, A.G. Baydin, S. McIlraith, Q. Gao, A. Acharya, D. Krueger, A. Dragan, P. Torr, S. Russell, D. Kahneman, J. Brauner, S. Mindermann.

CoRR. 2023.

[DOI: 10.48550/arXiv.2310.17688](https://doi.org/10.48550/arXiv.2310.17688)

\[1, 2\]

, advancing our understanding of these systems is crucial to ensure their alignment

- **AI Alignment: A Comprehensive Survey** [\[PDF\]](http://arxiv.org/pdf/2310.19852.pdf)

J. Ji, T. Qiu, B. Chen, B. Zhang, H. Lou, K. Wang, Y. Duan, Z. He, J. Zhou, Z. Zhang, F. Zeng, K.Y. Ng, J. Dai, X. Pan, A. O'Gara, Y. Lei, H. Xu, B. Tse, J. Fu, S. McAleer, Y. Yang, Y. Wang, S. Zhu, Y. Guo, W. Gao.

CoRR. 2024.

[DOI: 10.48550/arXiv.2310.19852](https://d

... (truncated, 98 KB total)