Fun-CineForge - An open-source film-grade multimodal dubbing model from Alibaba Tongyi.

What is Fun-CineForge?

Fun-CineForge is the first film-grade multimodal dubbing model open-sourced by Tongyi Lab. Built on CosyVoice3, it innovatively introduces "temporal modality" to achieve accurate audio-visual synchronization. The model supports monologues, narration, dialogues, and multi-person scenes, solving four major challenges: lip-syncing, emotional expression, consistent timbre, and time alignment. Fun-CineForge comes with an open-source CineDub dataset construction process, covering over 350 films and TV series, with a Chinese character error rate as low as 1.49%, maintaining high-quality dubbing even in complex scenes such as facial occlusion and camera transitions.

Fun-CineForge's main functions

• Lip-syncThe model supports high synchronization between synthesized speech and the lip movements of characters in the video, achieving precise audio-visual alignment.
• Emotional expressionBased on the character's facial features and command descriptions, it achieves anthropomorphic presentation and free control of emotional tone.
• Timbre CloningThe model can synthesize highly similar personalized speech by referencing the timbre features of the input audio.
• Time alignmentIt controls the start and end of speech based on timestamps, and can generate speech at the correct time even if the speaker is obscured.
• Multi-scenario adaptationSupports complex film and television dubbing scenarios such as monologues, narration, two-person dialogues, and multi-person dialogues.

The technical principles of Fun-CineForge

• Multimodal fusion architectureThe model can process four types of information simultaneously: the visual modality learns lip movements and facial expressions, the text modality provides dialogue content and emotional cues for the character, the audio modality serves as the prediction target, and the temporal modality controls the timing of speech occurrence and indicates the speaker's identity. The four complement each other to achieve accurate dubbing.
• Temporal Modal InnovationFor the first time, time information is introduced as an independent modality into the dubbing model. Through strong supervision signals such as start time, duration, and speaker identity, the model can understand "when and who is speaking" and accurately locate the speech period when the face is occluded or the camera switches.
• Data-driven trainingThe model is trained on the CineDub dataset, which is automatically constructed. This dataset is extracted from film and television materials through processes such as voice separation, text transcription, and speaker separation. It includes frame-level lip data, millisecond-level timestamps, and sentiment annotations, providing multimodal supervision signals for the model.

Fun-CineForge project address

• Project official websitehttps://funcineforge.github.io/
• GitHub repository: https://github.com/FunAudioLLM/FunCineForge
• HuggingFace model libraryhttps://huggingface.co/FunAudioLLM/Fun-CineForge

Fun-CineForge Application Scenarios

• Film and television post-production: Provide multilingual dubbing for movies and TV series, accurately match lip movements and emotions, and handle complex scenes such as camera transitions and facial obscuration.
• Animation and game developmentGenerates synchronized audio and video voices for animated characters, supports differentiation of multiple character voices, and reduces the cost of voice acting for game storylines.
• Content localizationIt translates and dubs overseas films and television works into other languages, preserving the original film's emotional rhythm and supporting the conversion of long segments such as narration and monologue.
• Short video advertisementThe model can quickly generate voiceovers for spoken videos, adjust the tone of voice according to the emotions in the scene, and clone specific timbres to maintain brand consistency.
• AccessibilityThe model can generate synchronized narration for silent videos, helping visually impaired users understand the visuals and providing accurate audio-subtitle matching.