AB
AB
AiBoss

All Categories

project

TANGOFLUX - An open-source text-to-audio generation model developed by NVIDIA in collaboration with the Singapore University of Technology and Design.

TANGOFLUX is a high-efficiency text-to-audio generation model jointly developed by the Singapore University of Technology and Design (SUTD) and NVIDIA. The model boasts approximately 515 million parameters and can generate audio files up to 30 seconds long in just 3.7 seconds on a single A40 GPU...

What is TANGOFLUX?

TANGOFLUX is a high-performance text-to-audio generation model jointly developed by the Singapore University of Technology and Design (SUTD) and NVIDIA. With approximately 515 million parameters, the model can generate 30 seconds of 44.1kHz audio in just 3.7 seconds on a single A40 GPU. TANGOFLUX uses the CLAP-Ranked Preference Optimization (CRPO) framework, iteratively generating and optimizing preference data to improve the model's audio alignment capabilities. The model demonstrates excellent performance in both objective and subjective benchmark tests, and its source code and model are open-sourced on platforms such as GitHub, supporting further research.

Main functions of TANGOFLUX

  • High-efficiency audio generationTANGOFLUX can quickly generate high-quality audio content, producing up to 30 seconds of 44.1kHz audio in 3.7 seconds.
  • Text to audio conversionThe model directly converts text descriptions into corresponding audio outputs, achieving direct text-to-audio conversion.
  • Preference optimizationTANGOFLUX optimizes audio output to better match user preferences and the intent of the input text.
  • Training with non-proprietary dataTraining on non-proprietary datasets makes the model more open and accessible.

Technical principles of TANGOFLUX

  • Variational autoencoderThe audio waveform is encoded into a latent representation using VAE, and the original audio is reconstructed from the latent representation.
  • Text and duration embeddingThe model uses text encoding and duration encoding to control the content and duration of generated audio, thus achieving controllable audio generation.
  • FluxTransformer architectureBuilt on FluxTransformer blocks, it combines Diffusion Transformer (DiT) and Multimodal Diffusion Transformer (MMDiT) to handle text prompts and generate audio.
  • Flow Matching (FM)Based on the flow matching framework, it learns the mapping from simple prior distribution to complex target distribution to generate samples.
  • CLAP-Ranked Preference Optimization (CRPO)The CRPO framework optimizes audio alignment by iteratively generating preference data pairs. It uses the CLAP model as a surrogate reward model, evaluating the quality of the audio output based on the joint embeddings of text and audio, and constructing a preference dataset accordingly for preference optimization.
  • Direct preference optimizationTANGOFLUX applies DPO to stream matching, comparing winning and losing audio samples to optimize the model and improve the alignment between audio and text descriptions.

TANGOFLUX project address

Application scenarios of TANGOFLUX

  • Multimedia content creationIn film, game, advertising, and video production, it is used to generate background music, sound effects, and voice-overs, improving production efficiency and reducing costs.
  • Audio production and designMusic producers and sound designers create new musical works or design specific sound effects.
  • Podcasts and audiobooksAutomatically generate background music and sound effects for podcasts or audiobooks to enhance the listener's auditory experience.
  • Education and trainingIn the field of education, audio recordings that simulate real-life scenarios can be created to aid language learning or professional skills training.
  • Virtual assistants and chatbotsIt provides more natural and richer voice responses for virtual assistants and chatbots, enhancing the user interaction experience.
Back