AudioStory - An audio generation model launched by Tencent ARC

What is AudioStory?

AudioStory is an audio generation technology released by Tencent ARC Labs, capable of generating high-quality long-form narrative audio based on natural language descriptions. It employs a divide-and-conquer strategy, breaking down complex narrative requests into ordered subtasks, and through a decoupling bridging mechanism, precisely coordinates semantic and sound effect details. End-to-end training enhances model synergy, resulting in audio with temporal logic and emotional depth.

AudioStory's main functions

• Automatic video dubbingUsers can upload silent videos and describe their sound effects style. AudioStory can automatically analyze the video content and generate a background audio track that is synchronized with the video and has a consistent style.
• Audio intelligent continuationGiven an audio clip, AudioStory can intelligently infer subsequent scenes and automatically supplement reasonable audio sequels, such as supplementing player footsteps and basketball slapping sounds based on the coach's voice during basketball training.
• Audiobook creationIt provides high-quality audio content for audiobooks, generating audio with chronological logic and emotional layers based on text descriptions, allowing listeners to better immerse themselves in the story.
• Game sound effects productionIt generates immersive sound effects for games, producing matching audio based on game scene descriptions to enhance the player's gaming experience.
• Smart PodcastIt helps podcast creators quickly generate audio content by generating corresponding audio segments based on topic descriptions, thereby improving creation efficiency.

The technical principles of AudioStory

• Divide and conquer strategyThe complex narrative request is broken down into sequential sub-tasks, each generating a corresponding audio segment, which is then precisely arranged along the timeline to ensure the coherence and logic of the overall audio.
• Decoupling bridging mechanismThe collaboration between the large language model and the audio generator is decomposed into two components: bridge query and residual query. These are used for semantic alignment within events and consistency preservation across events, respectively, to improve the generation effect.
• End-to-end trainingWe adopt a unified training method and optimize both instruction understanding and audio generation to enhance the synergy between different parts of the model and improve overall performance.
• Dual-channel mechanism of semantic tokens and residual tokensBy using dual channels to process macro-narrative and micro-sound effect details respectively, the relationship between the two is precisely coordinated, so that the generated audio not only conforms to the overall narrative logic, but also has rich detail expression.
• Three-stage progressive trainingFrom single-tone generation to audio collaboration, and then to long-form narrative, the performance and adaptability of the model are gradually improved, enabling it to better cope with complex long-form narrative audio generation tasks.

AudioStory project address

• Github repository: https://github.com/TencentARC/AudioStory.
• Paper address: https://arxiv.org/pdf/2508.20088.

Application scenarios of AudioStory

• Video dubbingBased on the silent video and sound effect style description provided by the user, automatically analyze the video content and generate a matching background audio track.
• Audio continuationBased on a given audio segment, infer subsequent scenes and supplement reasonable audio sequels, such as adding player footsteps to basketball training audio.
• Audiobook creationGenerate audio with temporal logic and emotional layers based on text descriptions, enhancing the listening experience of audiobooks.
• Game sound effect generation: Generate immersive sound effects based on game scene descriptions to enhance the player's gaming experience.