LatentSync - An end-to-end lip-sync framework open-sourced by ByteDance and Beijing Jiaotong University.

What is LatentSync?

LatentSync is an end-to-end lip-sync framework jointly developed by ByteDance and Beijing Jiaotong University. Based on an audio-conditional latent diffusion model, it requires no intermediate 3D representations or 2D feature points. LatentSync leverages the powerful generative capabilities of Stable Diffusion to capture complex audiovisual relationships and generate dynamically realistic speech videos. To address the temporal consistency issue caused by inconsistencies in the diffusion process between different frames, it introduces the Temporal REPresentation Alignment (TREPA) method. This method uses a large-scale self-supervised video model to extract temporal representations, enhancing the temporal consistency between generated and real frames while maintaining lip-sync accuracy. Based on comprehensive experimental research, LatentSync solves the convergence problem of SyncNet and improves the accuracy of lip-sync.

LatentSync's main functions

• Simultaneous lip shape generationBased on the input audio, it generates matching lip movements, synchronizing the lips of the characters in the video with the audio. It is suitable for scenarios such as dubbing and virtual avatars.
• High-resolution video generationIt generates high-resolution videos, overcoming the limitations of traditional diffusion models that have high hardware requirements when diffusion occurs in pixel space.
• Dynamic and realistic effectsThe generated videos have dynamic and realistic effects, capturing subtle facial expressions related to emotional tone, making the characters' speech more natural and vivid.
• Enhanced time consistencyBased on the Temporal REPresentation Alignment (TREPA) method, it improves the temporal consistency of generated videos, reduces video flickering, and makes video playback smoother.

LatentSync's technical principles

• Audio Conditional Latent Diffusion ModelThis approach uses a latent diffusion model to directly model the latent space, conditioned on audio, without requiring pixel-space diffusion or a two-stage generation process. The latent diffusion model better captures the complex relationship between audio and visual information, generating high-quality lip-synced videos.
• end-to-end frameworkBased on an end-to-end framework, audio feature extraction, latent representation generation, and lip synchronization generation are integrated into a unified model, simplifying intermediate steps and improving generation efficiency and accuracy.
• Temporal REPresentation Alignment (TREPA)Introducing the TREPA method, which uses the large-scale self-supervised video model VideoMAE-v2 to extract temporal representations and calculates the distance between the temporal representations of generated consecutive frames and real consecutive frames as an additional loss to enhance the temporal consistency of generated videos.
• SyncNet supervisionDuring training, the generated videos are supervised using a pre-trained SyncNet to ensure good lip synchronization. A SyncNet loss is added in pixel space to help the model better learn the correspondence between audio and lip movements.

LatentSync project address

• GitHub repository:https://github.com/bytedance/LatentSync
• arXiv technical paper:https://arxiv.org/pdf/2412.09262

Application scenarios of LatentSync

• Film and television post-productionDuring film dubbing, lip animations are automatically generated based on the dubbing audio, improving production efficiency and maintaining the consistency of character image.
• EducationIn online English classes, teachers convert spoken audio into lip-synced video to help students learn pronunciation more accurately.
• Advertising video productionIn car advertisements, lip-synced videos are generated for virtual spokespeople to make the advertising copy more natural and enhance the appeal of the ads.
• Remote conferencingReal-time lip-sync video generation during cross-border remote meetings solves the problem of audio-visual asynchrony caused by network latency, improving communication effectiveness.
• Game developmentIn RPG games, synchronizing NPCs' lip movements with their voices during dialogue enhances game immersion and character interaction.