Audio-SDS - NVIDIA's Extended Text Conditional Audio Diffusion Model

What is Audio-SDS?

Audio-SDS, an innovative technology from NVIDIA's AI research team, extends Score Distillation Sampling (SDS) technology to text-based conditional audio diffusion models, bringing a significant breakthrough to the field of audio processing. Without retraining the model, any pre-trained audio diffusion model can be transformed into a versatile tool, widely applicable to tasks such as sound effect generation, sound source separation, FM synthesis, and speech enhancement. Audio-SDS guides audio generation through text prompts, supports high customization, and meets both creative and industrial needs.

Main functions of Audio-SDS

• Sound effect generationIt can generate various environmental or creative sound effects based on text prompts, such as explosions and wind sounds, to help game development and virtual reality (VR) applications.
• Audio source separationIt accurately extracts target audio tracks from mixed audio, suitable for music production and video post-processing. For example, it can perform fully automated source separation of real-world audio without manually labeling sources or using specialized datasets.
• Physical information affects sound simulationIt can guide physical information to influence sound simulation, such as simulating the sound of objects colliding.
• FM Synthesis Parameter CalibrationSupports high-quality frequency modulation synthesis for expressive sound design.
• Speech enhancementImproves voice clarity and is suitable for audio editing software and smart voice assistants.

The technical principle of Audio-SDS

• Pre-trained audio diffusion modelAudio-SDS is based on a pre-trained audio diffusion model. This model can generate high-quality audio samples that contain rich prior audio knowledge.
• Text conditional guidanceThe audio generation process is guided by text prompts. These prompts are encoded as conditional vectors that instruct the audio diffusion model to generate audio that conforms to the description.
• Fractional distillation sampling (SDS)During audio generation, SDS optimizes model parameters by calculating the difference between the generated audio and the target audio, making the generated audio closer to the target audio. Specifically, SDS achieves this through the following steps:
  • Noise additionAdd random noise to audio samples to generate noisy audio.
  • Loss CalculationThe algorithm calculates the difference between the noisy audio and the real audio, and optimizes the parameters using gradient descent to minimize the difference between the predicted noise and the real noise.
  • Optimization ObjectiveThe loss function of SDS is based on the probability density distribution of the diffusion model, and the parameters are optimized by minimizing the KL divergence between the noise distribution and the true distribution.
• Multi-functional extensionAudio-SDS eliminates the need for retraining the model, transforming a pre-trained audio diffusion model into a versatile tool suitable for various tasks such as sound effect generation, sound source separation, FM synthesis, and speech enhancement.
• Efficient ReasoningThe optimized SDS algorithm maintains high-quality output while reducing computational complexity and improving the feasibility of real-time applications.

Audio-SDS project address

• Project official website:https://research.nvidia.com/labs/toronto-ai/Audio-SDS/
• arXiv technical paper:https://arxiv.org/pdf/2505.04621

Application scenarios of Audio-SDS

• Sound effect generationAudio-SDS can generate a variety of realistic environmental or creative sound effects based on text prompts, such as explosions, wind, and rain, providing immersive sound design for movies, games, and virtual reality (VR) applications and enhancing the user experience.
• Audio source separationIn music production and video post-processing, Audio-SDS can accurately extract target audio tracks from mixed audio, such as separating vocals from accompaniment, making it easier for music producers to mix or create new music.
• Audio editingThis tool provides efficient tools for music producers and content creators, lowering the barrier to entry for professional audio processing. Creators can generate high-quality audio content through simple text descriptions, without requiring complex audio editing skills.
• Music EducationExtracting a cappella audio tracks can be used to create karaoke accompaniment and also helps with music education, such as transcribing and learning.
• Smart HomeIt automatically recognizes various sounds in the home environment, such as a baby crying or a leaking faucet, thus improving the level of smart living.