AudioX-Turbo - An audio generation framework jointly developed by Noiz AI and Tsinghua University

What is AudioX-Turbo?

AudioX-Turbo is a unified and efficient audio generation framework developed by Noiz AI in collaboration with the Hong Kong University of Science and Technology and Tsinghua University. It supports the generation of high-quality sound effects and music from arbitrary combinations of text, video, and audio inputs. The model is based on a 2.7B multimodal diffusion Transformer architecture. Through distribution matching distillation and adversarial distillation, it compresses the inference steps from 50–200 to 4, generating 10 seconds of audio on a single RTX 4090 in just 0.24 seconds. The team constructed the IF-caps-Pro dataset with approximately 9.2 million samples, achieving precise timestamp control for the first time.

Main functions of AudioX-Turbo

• Text to audio (T2A)It generates ambient sounds and action sound effects based on plain text prompts, and supports precise description of sound type, style and scene.
• Text-to-Music (T2M)Generates music clips with specified style, instrument configuration, rhythm, and mood based on text commands, and supports complex music theory descriptions.
• Video to Audio (V2A)Automatic dubbing and Foley for silent videos, matching motion sound effects and ambient sounds in real time based on the content of the video.
• Video to Music (V2M)It analyzes the emotions, rhythm, and dynamics of a video to automatically generate synchronized background music, supporting multiple music styles.
• Text + Video Joint Generation (TV2A / TV2M)It combines video footage with text commands for precise control, such as specifying "thunder at the 3rd second and guitar at the 5th second," to achieve timestamp-level precise sound effects/music generation.
• Audio completion and repair It can complete, repair, or transfer styles based on contextual audio segments, and supports intelligent continuation and noise reduction of audio content.
• Image to audio (zero-sample)It does not require special training for images and can infer and generate corresponding ambient sounds or scene sound effects based on the content of static images.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

AudioX-Turbo's technical principles

• Multimodal diffusion Transformer architectureThe model employs a 2.7B multimodal diffusion Transformer, built upon the Flow Matching framework. Text is processed by a T5 encoder to extract semantic features, video by a Synchformer to extract temporal visual representations, and audio by VAE compression into latent space representations. These three modalities are then uniformly projected and input into a shared Transformer for cross-attention fusion, enabling joint generation of text, video, and audio under arbitrary combinations.
• Distribution matching distillation and counter-distillationTo compress the 50–200 sampling process of the teacher model into 4 steps, the team adopted a two-stage distillation strategy: First, the student model was trained to approximate the flow field distribution of the teacher model through distribution matching distillation (DMD), so that a single-step prediction could match the results of multiple steps; then, a diffusion discriminator was introduced to perform adversarial distillation, further refining the audio details under the 4-step constraint. The student model even surpassed the teacher model in some objective metrics.
• Strong instruction dataset IF-caps-ProThe team constructed the IF-caps-Pro dataset with approximately 9.2 million samples. The core innovation lies in structured annotation: each data point includes precise timestamps, the number of events, instrument type, and the order of appearance. This enables the model not only to understand static combinations like "guitar + drum beats," but also to accurately execute complex timing instructions such as "first a cicada chirps for 3 seconds, then add the guitar at the 5th second, and then fade out at the 8th second," significantly improving controllable generation capabilities.
• Three-stage progressive training strategyThe training process consists of three stages: the first stage involves pre-training on plain text-audio data to establish basic acoustic understanding; the second stage introduces video data to expand to multimodal joint generation, learning the audio-visual synchronization relationship; and the third stage performs distillation acceleration, compressing the multi-step model into a 4-step ultra-fast version. This progressive strategy ensures that the model does not lose its multimodal understanding ability while compressing inference costs.

How to use AudioX-Turbo

• Environment configurationClone the GitHub repository and create a Python 3.8 environment, installing FFmpeg, libsndfile, and the dependencies listed in requirements.txt.
• Weight DownloadDownload the pre-trained weights for the AudioX-Turbo 4-step student model, VAE, and Synchformer video encoder from Hugging Face or GitHub Release.
• Initiate reasoning:run python run_gradio.py Launch the local web interface, or directly call the Python API to load the model and tokenizer.
• Parameter settings: Set according to task type video_path,text_prompt,audio_path With input parameters such as text, video, and audio, the model supports any combination of inputs.
• Generate audio: Performing forward inference, 4 sampling steps are sufficient to generate 10 seconds of high-quality audio, taking approximately 0.24 seconds on an RTX 4090.

AudioX-Turbo's core advantages

• Speed ReasoningIt can match the teacher model's sound quality in 100 steps with only 4 sampling steps, reducing NFE by about 25 times and achieving near real-time generation.
• Unified MultimodalA single model supports any combination of text, video, and audio inputs, eliminating the need to train separate expert models for different tasks.
• Precise instruction followingThe 9.2 million structured data entries with timestamps, event counts, and instrument annotations enable the model to accurately understand complex timing instructions such as "cicadas chirping first, then guitar playing".
• High-quality distillationBased on Flow Matching, Distributed Matching Distillation (DMD) combined with a diffusion discriminator, the student model outperformed the teacher model in some metrics.

AudioX-Turbo project address

• GitHub repositoryhttps://github.com/NoizAI/AudioX-Turbo
• HuggingFace model libraryhttps://huggingface.co/HKUSTAudio/AudioX-Turbo
• arXiv technical paper: https://arxiv.org/pdf/2606.12555