ThinkSound - The first CoT audio generation model launched by Alibaba Tongyi

What is ThinkSound?

ThinkSound is the first CoT (Chain Thinking) audio generation model launched by Alibaba's Tongyi Speech Team. Used for video dubbing, it generates custom-matched sound effects for each frame. The model introduces CoT inference to solve the problem of traditional technologies struggling to capture dynamic details and spatial relationships in images, allowing AI to think step-by-step like a professional sound engineer, generating high-fidelity audio with synchronized audio and video. The model is based on a three-order thought chain to drive audio generation, including basic sound effect inference, object-level interaction, and instruction editing. The model is equipped with the AudioCoT dataset, which contains audio data annotated with thought chains. On the VGGSound dataset, ThinkSound outperforms six mainstream methods (Seeing&Hearing, V-AURA, FoleyCrafter, Frieren, V2A-Mapper, and MMAudio), demonstrating superior performance.

ThinkSound's main functions

• Basic sound effect generationGenerate basic sound effects that match the video content semantically and temporally, providing initial audio background for the video.
• Interactive object-level refinementIt allows users to click on specific objects in a video to refine and optimize the sound effects for those objects, making the sound effects more accurately match the specific visual elements.
• Command-driven audio editingIt supports users to edit the generated audio based on natural language commands, such as adding, deleting or modifying specific sound effects to meet different creative needs.

ThinkSound's technical principles

• Chain thinking and reasoningThe audio generation task is broken down into multiple reasoning steps, including analyzing visual dynamics, inferring acoustic properties, and synthesizing sound effects in chronological order, mimicking the creative process of human sound engineers.
• Multimodal Large Language Model (MLLM)Based on models such as VideoLLaMA2, spatiotemporal information and semantic content of videos are extracted, and structured CoT inference chains are generated to provide detailed guidance for audio generation.
• Unified Audio Foundation ModelBased on conditional flow matching technology, this model combines video, text, and audio context information to generate high-fidelity audio. It supports arbitrary combinations of input modalities, flexibly handling different generation and editing tasks.
• Dataset supportBased on the AudioCoT dataset, it provides audio data with structured CoT annotations for training and optimizing models, improving the understanding and generation capabilities of audio-visual relationships.

ThinkSound's project address

• Project official websitehttps://thinksound-project.github.io/
• GitHub repository: https://github.com/liuhuadai/ThinkSound
• HuggingFace model libraryhttps://huggingface.co/liuhuadai/ThinkSound
• arXiv technical paper: https://arxiv.org/pdf/2506.21448

ThinkSound application scenarios

• Film and television productionIt generates realistic background sound effects and scene-specific sound effects for movies, TV series, and short videos, enhancing the audience's immersion and improving the realism of audio-visual synchronization.
• Game developmentIt generates dynamic environmental and interactive sound effects for game scenes, enhancing player immersion and interactivity, and improving the gaming experience.
• Advertising and MarketingGenerate engaging sound effects and background music for advertising videos and social media content to enhance content appeal and reach, and boost brand influence.
• Education and trainingGenerate sound effects that match the content of online educational videos and simulated training environments to help students better understand and remember, thereby improving learning outcomes and training quality.
• Virtual Reality (VR) and Augmented Reality (AR)Generate sound effects that closely match the virtual environment in VR and AR applications, enhancing user immersion and interactivity, and providing a more personalized experience.