OmniAudio - A spatial audio generation model launched by Alitongyi

What is OmniAudio?

OmniAudio is a technology developed by the Speech Team of Alibaba Tongyi Labs to generate spatial audio (FOA) from 360° videos. It provides a more realistic audio experience for virtual reality and immersive entertainment. By constructing the large-scale dataset Sphere360, containing over 103,000 video clips covering 288 audio events with a total duration of 288 hours, it provides rich resources for model training. OmniAudio training consists of two stages: self-supervised coarse-to-fine flow matching pre-training, based on large-scale non-spatial audio resources; and supervised fine-tuning based on bi-branch video representations, enhancing the model's ability to represent sound source directions.

OmniAudio's main functions

• Generate spatial audioIt can directly generate FOA audio from 360° video. This audio is a standard 3D spatial audio format that captures the directionality of sound, achieving realistic 3D audio reproduction. It uses four channels (W, X, Y, Z) to represent sound, with the W channel capturing the overall sound pressure level, and the X, Y, and Z channels capturing sound information in the front-back, left-right, and vertical directions, respectively. This ensures accurate sound localization even when the head rotates.
• Enhance immersive experienceThis technology opens up entirely new possibilities for virtual reality and immersive entertainment. It can solve the problem that traditional video-to-audio generation technologies mainly generate non-spatial audio, which cannot meet the 3D sound positioning requirements of immersive experiences.

OmniAudio's technical principles

• Self-supervised coarse-to-fine stream matching pre-training
  • Data processingDue to the scarcity of real FOA data, the research team utilized large-scale non-spatial audio resources (such as FreeSound, AudioSet, VGGSound, etc.) to convert stereo sound into a "pseudo-FOA" format. Specifically, the W channel represents the sum of the left and right channels, the X channel represents the difference between the left and right channels, and the Y and Z channels are set to zero.
  • Model trainingThe converted "pseudo-FOA" audio is fed into a four-channel VAE encoder to obtain a latent representation. Then, a random time window mask is applied with a certain probability. The masked latent sequence, along with the complete sequence, is used as conditional input to the stream matching model. The model achieves self-supervised learning of the audio temporal sequence and structure by minimizing the velocity field difference between the latent states before and after masking. This stage enables the model to grasp general audio features and macroscopic temporal patterns, laying the foundation for subsequent refinement of spatial audio.
• Supervised Fine-Tuning Based on Two-Branch Video Representation
  • Data utilizationUsing only real FOA audio data, we continue with the masked stream matching training framework, but this time the model's attention is focused entirely on the spatial characteristics of the four channels.
  • Model reinforcementBy masking the real FOA latent sequence with higher probability, the ability to represent the direction of the sound source (the complementary relationship between the four channels W/X/Y/Z) is enhanced, which improves the reconstruction effect of high-fidelity spatial audio details at the decoding end.
  • Two-branch combinationAfter completing self-supervised pre-training, the model is combined with a dual-branch video encoder. For the input 360° panoramic video, a frozen MetaCLIP-Huge image encoder is used to extract global features; simultaneously, a local FOV viewpoint is cropped from the same video, and the same encoder is used to obtain local detail representations. The global features are max-pooled and used as global conditions for the Transformer, while the local features are temporally upsampled and added element-wise with the audio latent sequence as local conditions in the progressive generation process.
  • Fine-tuning and outputWhile maintaining the general direction of the pre-trained initialization parameters, the conditional flow field is efficiently fine-tuned to accurately "carve" out the FOA potential trajectory that conforms to visual instructions from the noise. After fine-tuning, in the inference stage, only the learned velocity field needs to be sampled, and then the waveform is recovered by the VAE decoder to output a four-channel spatial audio with precise directionality that is highly aligned with the 360° video.

OmniAudio's project address

• Project official website:https://omniaudio-360v2sa.github.io/
• Github repository:https://github.com/liuhuadai/OmniAudio
• arXiv technical paper:https://arxiv.org/pdf/2504.14906

Application scenarios of OmniAudio

• Virtual Reality (VR) and Immersive ExperiencesOmniAudio can generate spatial audio for VR content that highly matches the visual scene, enhancing the user's immersion.
• 360° video soundtrackIt automatically generates immersive sound effects for 360° panoramic videos, giving viewers a more realistic auditory experience when watching videos.
• Intelligent voice assistantIt can be integrated into smart home devices, such as smart speakers and smart appliances, to enable voice control and interaction. Users can use voice commands to control appliances, adjust the temperature, and query information.
• Robotics and autonomous driving fieldsOmniAudio can be applied to robotics and autonomous driving, providing these systems with more accurate sound localization and environmental perception.