Stable Audio 3 - An open-source audio generation model series from Stability AI

What is Stable Audio 3?

Stable Audio 3 is a new generation of open-source audio generation models from Stability AI. Based on a stream-matching latent space diffusion architecture, it supports text-to-music and sound effects conversion, audio editing, and continuation. Stable Audio 3 offers Small, Medium, and Large models. The Small version can run locally on a MacBook Pro and generate audio up to 2 minutes long, while Medium and Large models support high-quality generation of over 6 minutes. All models are trained using fully licensed data, with open Small and Medium weights, and support for LoRA fine-tuning, achieving fast inference in less than 2 seconds.

Main features of Stable Audio 3

• Text-to-audio generationThe system can generate instrumental music or ambient sound effects by prompting users with English text, and the output duration can be precisely controlled down to the second.
• Variable length audio synthesisThe model allocates the latent space sequence length according to the proportion of request duration, avoiding computational and memory waste caused by a fixed maximum length.
• Partial audio editing and repairThis feature enables partial inpainting of audio using single or multiple segments of masking, replacing the target area while preserving the original segment.
• Audio intelligent continuation extensionUsing a causal masking mechanism to coherently continue existing audio, short recordings can be expanded into complete works of more than 6 minutes.
• LoRA style tweaksFor the first time, LoRA training documents and Small and Medium weights are available, allowing users to efficiently adapt custom styles using their own audio libraries.
• End-to-end local deploymentVersion 3.0 Small supports fully offline operation on consumer devices such as the MacBook Pro, enabling privacy-preserving creation without a network connection.

The technical principles of Stable Audio 3

• Semantic-Acoustic AutoencoderBased on the SAME architecture, 44.1kHz stereo is mapped to a 256-dimensional latent space through a 4096x downsampling, balancing high-fidelity reconstruction and semantic structure encoding.
• Flow matching latent space diffusionThe flow matching training paradigm is used to perform diffusion generation in a compact latent space, and the optimal transport coupling of small batches is combined to improve training stability and efficiency.
• Post-competition training accelerationAfter pre-training and ODE distillation warm-up, adversarial post-training is introduced to compress the number of inference steps to an extremely low level, enabling the generation of long audio in less than 2 seconds on the H200.
• Differential Attention TransformerThe Diffusion Transformer integrates differential attention, adaptive layer normalization conditional injection, and memory embedding to enhance the modeling accuracy of long audio sequences.
• Variable-length inference mechanismBreaking through the fixed sequence length limitation of traditional diffusion models, this technology makes the latent space length proportional to the actual request duration, significantly reducing the cost of short audio generation.

How to use Stable Audio 3

• Obtain model weightsVisit the Hugging Face Stability-AI/stable-audio-3 repository and download the 3.0 Small, Small SFX, or Medium weight files.
• Configure the runtime environmentClone the stable-audio-tools code repository and install dependencies, ensuring that PyTorch and CUDA or Apple Metal backends are correctly configured.
• Loading Models and EncodersInstantiate the SAME autoencoder and the corresponding scaled diffusion Transformer in the script, and load the pre-trained weights into GPU memory.
• Write and generate promptsEnter English prompts describing the target audio style, instruments, and mood, and set the precise output duration (in seconds).
• Execution reasoning generationCall the model generation interface. The model will perform variable-length inference based on the duration parameter and output a 44.1kHz stereo WAV file.

The core advantages of Stable Audio 3

• Fully licensed commercial friendlyThe entire model is trained on licensed and Creative Commons data. The Community License allows users to freely own and commercially generate content.
• Native support for consumer hardware: 3.0 Small with open weights and can run offline on MacBook Pro M4, is the first lightweight model to support local full-song composition.
• Ultra-long high-quality generationThe Medium and Large models support the generation of high-quality audio exceeding 6 minutes, a significant improvement over the previous generation Stable Audio Open's 47-second limit.
• Ultimate reasoning efficiencyAfter adversarial training and optimization, the Large model generates 6 minutes and 20 seconds of audio in less than 2 seconds on an H200 GPU and only a few seconds on a MacBook Pro.
• Zero-annotation flexible editingIt can achieve single-segment and multi-segment editing and continuation without additional training data labeling, and can be directly integrated into the real creative workflow through random and causal masks.

Project address for Stable Audio 3

• Project official websitehttps://stability.ai/news-updates/meet-stable-audio-3-the-model-family-built-for-artistic-experimentation-with-open-weight-models
• GitHub repositoryhttps://github.com/Stability-AI/stable-audio-3
• HuggingFace model libraryhttps://huggingface.co/collections/stabilityai/stable-audio-3
• arXiv technical paper: https://arxiv.org/pdf/2605.17991

Stable Audio 3 Comparison with Similar Products

Application scenarios of Stable Audio 3

• Game and movie sound effectsIt can quickly generate interactive sound effects, ambient sounds, and background music, and precisely match the rhythm and emotional needs of the scene through local editing.
• Short video and advertising musicGenerates customized instrumental clips with precise duration, avoiding manual trimming and directly adapting to short videos, promotional videos, and podcast transitions.
• Music creation assistanceIt helps musicians generate motivational ideas, expand unfinished drafts, or replace parts of songs, significantly accelerating the arrangement iteration cycle.
• Local privacy-sensitive creationsVersion 3.0 Small supports fully offline operation, meeting the stringent requirements of film studios and independent musicians for data privacy and network isolation.
• Personalized brand voice: Fine-tune your company's own audio assets with LoRA to create consistent UI sound effects, brand notification sounds, and exclusive music styles.