SAM Audio - Meta's open-source audio segmentation model

What is SAM Audio?

SAM Audio is an open-source audio segmentation model from Meta that uses multimodal cues, including text, visual, and temporal cues, to separate specific sounds from complex audio mixtures. Its core technology is Perception Encoder Audiovisual (PE-AV), based on Meta's open-source Perception Encoder model. This model fuses audiovisual information and performs precise temporal annotation, achieving high-precision audio separation. Users can utilize SAM Audio through simple text descriptions (such as "guitar sound"), clicking on the sound-producing object in the video, or marking the time range of a sound's appearance.

SAM Audio's main functions

• Multimodal audio separation: Isolate specific sounds from complex audio mixes through text descriptions, visual selections (objects in the video), or time segment markers.
• Supports multiple audio tasksIt can handle various audio separation tasks such as speech, music, and general sound effects, and is suitable for different scenarios.
• Audio reviews without referenceIt provides SAM Audio Judge, an objective evaluation model that does not require reference audio to assess audio separation quality from a perceptual perspective.
• Real-world benchmarkingSAM Audio-Bench, the first real-world audio separation benchmark, is released, covering a variety of audio domains and cue types, and supports no-reference evaluation.
• High-efficiency real-time processingIt runs faster than real-time processing, with a real-time factor of approximately 0.7, making it suitable for large-scale audio processing.
• Assisting accessibility technologyCollaborate with relevant organizations to explore applications of accessibility technologies, such as hearing aid integration, and promote the inclusive development of audio AI.

SAM Audio's Technical Principles

• Perceptual encoder audio-visual module (PE-AV)Based on Meta's open-source Perception Encoder model, PE-AV can extract frame-by-frame video features and align them with audio representations, fusing audiovisual information to provide accurate time annotations and semantically rich feature representations for audio separation.
• Stream matching diffusion Transformer architectureIt employs a generative modeling framework to encode mixed audio and cue information into a shared representation, generating the target audio track and the remaining audio tracks, and supports cue input in multiple modalities.
• Large-scale multimodal data trainingBy training with a mixture of real and synthesized audio data, covering a variety of sound events such as speech, music and general sound effects, and combining advanced audio synthesis strategies, the robustness and generalization ability of the model are improved.
• No-reference audio evaluation modelSAM Audio Judge assesses audio quality based on the perceptual dimension, without requiring a reference audio track, providing an objective evaluation standard that is closer to the human auditory experience.
• Real-world benchmarkingSAM Audio-Bench, as the first real-world audio separation benchmark, covers a variety of audio tasks and modal cues, supports no-reference evaluation, and ensures the effectiveness of the model in practical applications.

SAM Audio's project address

• Project official websitehttps://ai.meta.com/samaudio/
• Github repositoryhttps://github.com/facebookresearch/sam-audio

SAM Audio Application Scenarios

• Audio cleanup and background noise removalRemove unwanted background noise from audio files, such as removing dog barking or traffic noise in podcast recordings, to improve audio quality.
• Creative Media ProductionIt helps creators extract specific instrument sounds from songs, separate vocals, or remix audio to enhance creative expression.
• Accessibility technologyCollaborate with hearing aid manufacturers to explore applications in hearing-assistive devices to help people with hearing loss better understand audio content.
• Video editingIn video production, visual cues can be used to separate the sound of specific objects, such as clicking on a guitarist in a video to extract the guitar sound, thus improving the flexibility of video editing.
• Audio Analysis and ResearchIt provides tools for audio analysis, helping researchers isolate and study specific sound events, and is applicable to fields such as music analysis and sound ecology.