PrismAudio - A video-to-audio generation framework launched by Alitongyi - AiBoss

What is PrismAudio?

PrismAudio, developed by Alibaba's Tongyi Lab, is a video-to-audio framework that automatically adds ambient sound effects to silent videos. The model pioneers a "decompositional thinking chain" technique, allowing it to first consider sound content, timing, texture, and spatial location before generating audio. It incorporates four "teachers" (semantic, temporal, aesthetic, and spatial) for multi-dimensional scoring and optimization. With only 518 million parameters, the model generates 9 seconds of audio in just 0.63 seconds, significantly outperforming existing methods. It has been accepted by ICLR 2026.

PrismAudio's main functions

Video to AudioThe model can automatically generate ambient sound effects (such as the sound of horses' hooves, wind and rain) that match the visuals for silent videos.
Semantic alignmentThe model ensures that the generated audio content accurately corresponds to the objects and actions in the video, avoiding audio-visual discrepancies.
Timing synchronizationIt supports precise control over the timing of sound and visual events, achieving seamless synchronization.
Aesthetic optimizationThe model generates natural, layered, and electronic-free high-quality audio, enhancing the listening experience.
Spatial positioningIt supports stereo output and automatically adjusts the left and right channels according to the location of the sound source in the picture to achieve sound positioning.
Mind chain reasoningIt adopts a decompositional thinking chain of "think first, then speak" to make the generation process explainable and controllable.

Key information and usage requirements for PrismAudio

DeveloperAlibaba Tongyi Fun Team
Technology typeVideo-to-Audio (V2A) Framework
Core InnovationDecompositional thinking chain + multi-dimensional reinforcement learning
Model size518 million parameters
Output Specifications44kHz stereo
Reasoning speedGenerating 9 seconds of audio takes only 0.63 seconds.
Input formatSilent video (supports common video formats)
Content restrictionsOnly generates ambient sounds/sound effects; does not support character voice acting.
Optional inputText descriptions can be used to assist in generation (not mandatory).
Hardware requirementsSupports GPU acceleration, but can also run on CPU.

PrismAudio's core advantages

Four-dimensional collaborative optimizationIt pioneered a decompositional thinking chain, independently modeling and collaboratively optimizing the four dimensions of semantics, temporality, aesthetics, and space, avoiding the drawbacks of traditional models that "lose sight of one thing while focusing on another," and achieving a high degree of unity between audio and visuals.
Think before you speakBreaking through the end-to-end black box generation mode, the model first outputs structured reasoning text (sound content, timing, texture, and location), and then generates audio. The process is interpretable and controllable.
High efficiency and lightweightWith only 518 million parameters, it can generate 9 seconds of audio in just 0.63 seconds, which is nearly twice as fast as similar models and is more suitable for real-time application scenarios.
Robust in complex scenariosIt outperforms existing methods on the self-built AudioCanvas complex scene benchmark, and can still maintain stable output in multi-event and multi-sound-source scenarios.

How to use PrismAudio

Try it online (recommended for beginners)Visit the Hugging Face online demo, upload a silent video, optionally input text description to assist in generating audio files, and AI will automatically generate audio files.
Local deploymentDownload the open-source code and model weights from GitHub or Hugging Face, install the dependencies, load the pre-trained model, input the video path, call the inference interface to generate audio, and support custom adjustment of thought chain parameters or reward weights.

PrismAudio's project address

Project official websitehttps://prismaudio-project.github.io/
GitHub repositoryhttps://github.com/FunAudioLLM/ThinkSound/tree/prismaudio
HuggingFace model libraryhttps://huggingface.co/FunAudioLLM/PrismAudio
arXiv technical paper: https://arxiv.org/pdf/2511.18833
Experience the demo onlinehttps://huggingface.co/spaces/FunAudioLLM/PrismAudio

Comparison of PrismAudio's similar products

Comparison Dimensions	PrismAudio	MMAudio	ThinkSound
Developer	Ali Tongyi Lab	Nanyang Technological University, Singapore, etc.	Ali Tongyi Lab
Core Methods	Decompositional thinking chain + multi-dimensional reinforcement learning	Multimodal Transformer	Single-unit thinking chain
Parameters	518 million	Approximately 1 billion	billions
Reasoning speed	0.63 seconds/9 seconds audio	1.30 seconds / 9 seconds audio	1.07 seconds / 9 seconds audio
Output sound quality	44kHz stereo	44kHz mono	44kHz stereo
Semantic Consistency (CLAP)	0.47	0.40	0.43
Timing synchronization (DeSync)	0.41	0.46	0.55
Spatial Accuracy (CRW)	7.72	—	13.47
Aesthetic Quality (MOS-Q)	4.21	3.95	4.05

Application scenarios of PrismAudio

Post-production of film and televisionIt automatically generates ambient sound effects for movies, documentaries, and trailers, replacing traditional sound effects work and reducing post-production costs and time.
Short video creationQuickly add ambient sounds to silent videos such as vlogs, food videos, and travel videos to enhance the immersiveness and reach of ASMR and healing content.
Game developmentGenerate dynamic sound effects for cutscenes and CG promotional videos, and match ambient sounds in real time according to scenes such as forests, cities, and battlefields, reducing repetitive work for sound engineers.
Advertising and MarketingAutomatically add operation sound effects to product demonstration videos, support rapid iteration of multiple audio tracks, and improve advertising testing efficiency and creative flexibility.
Education and TrainingTo supplement instructional videos and demonstrations with prompts and background sounds, enriching the auditory experience of multimedia courseware and improving learning focus and information absorption rate.