Transformer Debugger - An open-source tool from OpenAI for understanding and analyzing the internals of large models.

What is Transformer Debugger?

Transformer Debugger (TDB) is a tool developed by OpenAI's Superalignment team to help researchers and developers gain a deeper understanding and analysis of the internal structure and behavior of Transformer models. Transformer models are a deep learning architecture widely used in Natural Language Processing (NLP), particularly in tasks such as machine translation, text generation, and understanding.

TDB's core functionality is to provide a way to quickly explore model structure without writing code. It combines automatic interpretability technology and sparse autoencoders, enabling users to intuitively view and analyze specific behaviors of the model, such as why the model chooses to output a specific token given a prompt, or why the model's attention mechanism focuses on certain parts of the input text.

GitHub source code address:https://github.com/openai/transformer-debugger

Features of Transformer Debugger

• No-code model explorationTDB allows users to explore model structures without writing code, making the research and debugging process more intuitive and efficient.
• Forward transmission interventionUsers can intervene in the model's forward propagation process to observe how different operations affect the model's output, thereby gaining a better understanding of the model's decision-making process.
• Component-level analysisTDB can identify and analyze the latent representations of specific components that significantly contribute to the model’s behavior, such as neurons, attention heads, and autoencoders.
• Automatically generate explanationThe tool can automatically generate explanations that show the reasons for the activation of specific components, helping users understand the internal workings of the model.
• Visual interfaceThrough Neuron viewer, a React-based application, TDB provides a user-friendly interface for displaying and analyzing information about model components.
• Backend supportThe Activation server acts as a backend server, providing the necessary data support for TDB, including reading and serving data from public Azure buckets.
• Model and dataset supportThe open-source content includes a simple inference library for the GPT-2 model and its autoencoder, as well as some well-organized activation dataset examples to facilitate user experimentation and analysis.

How to install and configure the Transformer Debugger

1. First, make sure that Python/pip and Node/npm are installed on your computer.
2. Using a virtual environment is recommended (optional).

   # 如果已在虚拟环境，先取消激活
   deactivate
   # 创建新的虚拟环境
   python -m venv ~/.virtualenvs/transformer-debugger
   # 激活新的虚拟环境
   source
   ~/.virtualenvs/transformer-debugger/bin/activate

3. After setting up the environment, follow these steps:

   git clone git@github.com:openai/transformer-debugger.git
   cd transformer-debugger
   
   # 安装神经元解释器
   pip install -e .
   # 安装pre-commit钩子
   pre-commit install
   
   # 安装神经元查看器
   cd neuron_viewer
   npm install
   cd ..

4. To run the TDB application, follow the instructions to set it up.Activate server backendandNeuron Viewer Front End.