Qwen-Scope - Alibaba Tongyi's open-source toolkit for interpreting large models - AiBoss

What is Qwen-Scope?

Qwen-Scope is an open-source toolkit for interpretable large models from the Alibaba Tongyi Qianwen team. Based on Sparse Autoencoder (SAE) technology, it extracts interpretable features from the hidden layers of Qwen3/Qwen3.5 series models. The model can transform complex parameter calculations into human-understandable concepts and rules, which can be used for post-analysis. Through feature-level intervention, it enables inference control, data processing, and model optimization, becoming a practical interface connecting the model's internal structure with downstream development.

Qwen-Scope's main functions

SteeringWithout explicit natural language commands, targeted modifications to language, entities, and styles, as well as badcase repairs, can be achieved by enabling or disabling specific SAE features.
Data classification and synthesisBased on a small amount of seed data, toxicity/safety-related features are discovered, enabling classification with zero additional trainer; inactive features are identified and supplementary samples are synthesized in a targeted manner to cover long-tail capabilities.
Model training optimizationIt locates abnormal activation features such as language mixing and repeated generation, and assists in optimizing model behavior during supervised fine-tuning (SFT) and reinforcement learning (RL) stages.
Evaluation redundancy analysisCalculate the feature activation patterns between different evaluation sets, determine the redundancy and capability coverage of the evaluation sets, and guide the selection of high-coverage, low-cost test samples.

Qwen-Scope's technical principles

Sparse Autoencoder (SAE): Insert SAEs into the residual flow of each Transformer layer of Qwen, and decompose the high-dimensional activation vectors into sparse, interpretable feature dictionaries by imposing sparsity constraints.
Top-k activation and reconstructionEach layer is trained separately for the SAE. The encoder maps activations to overcomplete latent representations and retains only the k largest activations (k=50/100) for reconstruction, ensuring that features are highly decoupled.
Contrast Feature RecognitionConstruct positive and negative sample sets, compare their SAE average activation differences, and identify the feature directions most relevant to the target attributes (such as toxicity, Chinese language, classical writing style).
Feature Intervention FormulaDuring inference, the residual flow is modified by h′ ← h + αd, where d is the SAE feature direction and α controls the intervention intensity, with positive values enhancing the feature and negative values suppressing it.

How to use Qwen-Scope

Access Experience PlatformVisit the Hugging Face online space.
Select model weightsLoad the corresponding SAE weights based on the target model (e.g., Qwen3-8B, Qwen3.5-27B).
Input prompts to observe activationEnter the prompt word to view the SAE feature activation heatmap and ranking.
Identify target features: Locate anomalies or target feature IDs (such as Chinese feature 6159, classical Chinese feature 36398).
Adjusting the intensity of interventionSet the feature intervention coefficient α to positively enhance or negatively suppress a specific feature.
Verify control effectCompare the model output before and after the intervention to confirm that the bad case was repaired or the style transfer was successful.
Integrated training processBy incorporating the SAE signal into the SFT/RL loss function, directional model optimization can be achieved.

Key information and usage requirements for Qwen-Scope

PublisherAlibaba / Tongyi Qianwen Team
Coverage model: Qwen3-1.7B/8B, Qwen3-30B-A3B, Qwen3.5-2B/9B/27B/35B-A3B (7 in total)
Model typeDense Model + Hybrid Expert (MoE) Architecture
SAE weights14 groups, covering all Transformer layers.
Training dataEach model's pre-training data sampled 0.5B units.
Feature Dimension32K / 64K / 80K / 128K
Represents the number of reconstructed features50 or 100
Online experienceHugging Face and ModelScope are both online.

Qwen-Scope's core advantages

From "understanding" to "improving"Beyond traditional ex-post analysis, interpretability is transformed into the core engine driving model evolution.
Zero-weighted modification interventionDuring the inference phase, the direction of features can be directly manipulated, and the output behavior can be changed without fine-tuning or updating model parameters.
Data efficiency and low dependencyHighly discriminative features can be discovered with only a small amount of seed data (about 200 pairs), and the classification F1 score can reach 0.90+, significantly reducing the labeling cost.
Targeted and precise optimizationFor low-frequency bad cases such as language mixing and repeated generation, the SFT stage can accurately locate abnormal features and reduce the Chinese mixing rate from 0.81% to 0.22%.
Evaluation cost optimizationFeature coverage analysis identifies redundancy in the evaluation set, helping to select high-coverage samples and reduce evaluation overhead.

Qwen-Scope project address

HuggingFace model library:https://huggingface.co/collections/Qwen/qwen-scope
Technical Papers: https://qianwen-res.oss-accelerate.aliyuncs.com/qwen-scope/Qwen_Scope.pdf

Comparison of Qwen-Scope's similar competing products

Comparison Dimensions	Qwen-Scope	Gemma Scope
Publisher	Alibaba / Tongyi Qianwen	Google DeepMind
Coverage model	Qwen3 / Qwen3.5 series (7 models)	Gemma 2/3 series
Architecture support	Dense Model + MoE	Dense Model
SAE Architecture	Top-k SAE	JumpReLU SAE
Open source scale	14 sets of SAE weights	400+ SAEs, 30 million+ features
Core Applications	Inference control, evaluation analysis, data synthesis, training optimization	Mechanism explanation, security analysis, circuit tracing
Data Synthesis	Feature-driven synthesis improves data energy efficiency by approximately 15 times.	Mainly relies on traditional synthesis schemes
Evaluation and Analysis	Supports benchmark redundancy and coverage analysis	Emphasis on feature visualization and interactive exploration
Chinese support	Native support, including distinctive style features such as classical Chinese.	Primarily geared towards English-speaking scenarios
Interactive platform	Hugging Face / Magic Dash Community	Neuronpedia

Application Scenarios of Qwen-Scope

Reasoning Control and Repair: Fixed the issue of unexpected mixing of Chinese and other languages under English prompts; enabled style transfer such as modern text to classical Chinese text.
Secure Data GovernanceMultilingual toxic content classification based on feature discovery; targeted synthesis of safe training data, achieving near-safe alignment results with 120k real data using only 4k synthetic data.
Model training assistanceThe SFT stage uses SAE-assisted loss to suppress anomalous activation; the RL stage increases the sampling frequency of anomalous responses by manipulating repetitive correlation features, thereby accelerating convergence.
Evaluation set optimization: Analyze the feature overlap matrix among benchmarks such as GSM8K, MATH, and MMLU-Pro, eliminate redundant evaluation sets, and improve evaluation efficiency.
Model interpretability researchIt provides an open-source SAE foundation for academia and industry, supporting mechanism explanation, circuit tracing, and root cause analysis of illusions and biases.