SALMONN-2 - A general-purpose audio language model open-sourced by Tsinghua University and others.

What is SALMONN-2?

SALMONN-2 is a general-purpose audio language model jointly open-sourced by Tsinghua University, Shanghai Artificial Intelligence Laboratory, and University of Cambridge. It is built on ByteDance's SPEAR unified self-supervised audio encoder and multi-layer feature fusion adapter, using Qwen3 as the text base. With approximately 18,000 hours of supervised data, it achieved the best performance among open-source models of the same scale on the three major comprehensive benchmarks of MMAU-Pro, MMAR, and MMSU. It is the first system in the general-purpose ALLM to realize advanced capabilities such as sound event detection, audio forgery detection, speech quality assessment, and multimodal contextual speech recognition.

Main functions of SALMONN-2

• General Audio UnderstandingIt supports cross-domain tasks such as speech recognition, audio description, music understanding, emotion recognition, and speaker verification.
• Sound Event Detection (SED)It can locate and output the start and end time periods of environmental sound events, such as dog barks and footsteps.
• Audio deepfake detection: Determine the authenticity of the speech and locate the time interval of suspected synthesized or edited segments.
• Speech Quality Assessment (SQA)It perceives low-level acoustic features such as noise, distortion, and clarity, and provides quality scores and explanations.
• Multimodal Contextual Speech Recognition (MICL)By combining text spelling with reference audio pronunciation, the accuracy of recognizing uncommon words and rare personal names can be improved.

Technical Principles of SALMONN-2

• Unified self-supervised audio encoderSALMONN-2 uses SPEAR as a single audio front-end. The encoder is trained on large-scale unlabeled audio data through self-supervised learning, and can simultaneously capture semantic, pronunciation, timbre, speaker and acoustic environment information. It replaces the traditional Whisper+BEATs dual encoder solution, and maintains a more balanced cross-domain capability while simplifying the architecture.
• Multi-layer Feature Fusion (MLF) AdapterSPEAR encodes information at different levels: shallow layers preserve acoustic/pronunciation details, middle layers carry timbre/speaker information, and deep layers accumulate semantic concepts. The MLF adapter first performs layer normalization and splicing on the hidden states of each layer, then performs learnable downprojection and frame grouping compression, and finally maps the fused hierarchical representation to the language model embedding space, so that the model can flexibly call different levels of acoustic cues according to task requirements.
• Timestamp injection mechanismThe model converts timestamps into natural language text (e.g., ... <2.0 seconds>The audio sequence is directly inserted and interleaved with the audio embedding into the language model, enabling the model to complete the time localization task within a unified generation framework without the need for an additional dedicated timestamp embedding layer.
• Multimodal Context Learning (MICL) TrainingIn contextual speech recognition tasks, the model not only receives a text bias vocabulary, but also simultaneously acquires the reference pronunciation audio of these words as multimodal context. During training, a strategy of randomly discarding interference words and target words is introduced to prevent over-biasing, enabling the model to learn to make reasonable use of contextual cues with the support of acoustic evidence.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use SALMONN-2

• Access the repository and clone the codeAccess the GitHub repository https://github.com/bytedance/SALMONN/tree/salmonn2,use git clone Download the code to your local machine.
• Create runtime environment:implement conda create -n salmonn2 python=3.10 Create and activate the virtual environment, and upgrade pip, setuptools, and wheel.
• Install dependencies and projectsRun in the root directory of the repository pip install -r requirements.txt and pip install -e . --no-depsComplete the installation of SALMONN-2 and its runtime dependencies.
• Download pre-trained weightsDownload model checkpoints via HuggingFace CLI:hf download marcoyang/SALMONN-2-8B --repo-type model --local-dir /path/to/salmonn-2-hf。
• Load the model and inference:use AutoProcessor and AutoModelForCausalLM Load local weights, pass in the audio file and instruction text, and call... model.generate() It can obtain audio comprehension results.

SALMONN-2's core advantages

• Data efficiencyUsing approximately 18,000 hours of supervised data, far less than the hundreds of thousands to millions of hours commonly used by competitors of similar scale, significantly reduces annotation costs.
• A unified front end provides a more balanced approachA single SPEAR self-supervised encoder replaces dual encoders, achieving a more balanced performance across speech, ambient sound, music, and paralinguistic tasks.
• Complete utilization of hierarchical informationThe MLF adapter integrates features from all encoder layers, avoiding the loss of key acoustic/timbre details by using only the last layer.
• Expanding audio analysis capabilitiesFor the first time in general ALLM, the system supports previously neglected voice analysis tasks such as SED, forgery detection, and SQA.
• ScalableAfter expanding the text base from 8B to 30B-A3B, the scores of the three comprehensive benchmarks continued to improve, validating that the architecture has good scale-up potential.

SALMONN-2 project address

• GitHub repository：https://github.com/bytedance/SALMONN/tree/salmonn2
• HuggingFace model library：https://huggingface.co/marcoyang/SALMONN-2-8B
• arXiv technical paper：https://arxiv.org/pdf/2607.17079

Comparison of SALMONN-2 with similar competing products

Application scenarios of SALMONN-2

• Smart Meeting AssistantIt can transcribe meeting content in real time and accurately identify foreign names and technical terms by combining pronunciation examples of participants.
• Audio content reviewAutomatically detects unusual audio events in live streams or podcasts and identifies deepfake voices to prevent fraud.
• Voice quality monitoringAutomated quality scoring of customer service calls and recording studio footage, identifying noise and distortion segments.
• Multimedia Archive RetrievalGenerate timestamped event descriptions for historical audio and radio programs, enabling precise content-based retrieval.
• Assistive devices for the hearing impairedIt provides real-time notifications to hearing-impaired users of ambient sound events (such as doorbells and alarms) in the form of text and timestamps.