KeySync - A lip-syncing framework jointly developed by Imperial College London and the University of Wrocław

What is KeySync?

KeySync, developed by Imperial College London and the University of Wrocław, is a high-resolution lip-sync framework that aligns input audio with lip movements in video. Based on a two-stage framework, KeySync first generates keyframes to capture key lip movements in the audio, then uses interpolation to generate smooth transition frames. KeySync introduces a novel masking strategy to effectively reduce facial expression leakage in the input video and automatically handles occlusion issues using a video segmentation model. KeySync outperforms existing methods in visual quality, temporal coherence, and lip-sync accuracy, making it suitable for practical applications such as automatic dubbing.

KeySync's main functions

• High-resolution lip-syncGenerates high-definition (512×512) video that is precisely aligned with the input audio, suitable for practical applications.
• Reduce facial expressionsReduce facial expressions in input videos and improve synchronization.
• ObscuringDuring inference, it automatically identifies and eliminates obstructions (such as hands, objects, etc.) to ensure the naturalness of the generated video.
• Improve visual qualityIt performs well across multiple quantitative metrics and user studies, producing videos with higher clarity and consistency.

KeySync's technical principles

• Two-stage generative framework:
  • Keyframe generationFirst, a set of sparse keyframes is generated to capture the main lip movements in the audio, ensuring that each keyframe accurately reflects the speech content of the audio while preserving the identity information of the person.
  • Interpolation generationInterpolation is performed between keyframes to generate smooth, temporally coherent intermediate frames, enabling fluid lip movement transitions.
• Latent diffusion modelThe latent diffusion model performs denoising operations in a compressed, low-dimensional latent space, improving computational efficiency. Noise is gradually removed, transforming random noise into structured video data.
• Masking strategyBased on calculated facial key points, a mask is designed to cover the lower face region while retaining necessary contextual information to prevent facial expression leakage. During inference, a pre-trained video segmentation model (such as SAM²) is used to automatically identify and eliminate occlusions, ensuring that the generated lip region blends naturally with the occluded objects.
• Audio and video alignmentThe raw audio is converted into a feature representation using the HuBERT audio encoder and then embedded into the video generation model based on an attention mechanism to ensure that the generated lip movements are precisely aligned with the audio.
• loss functionBy combining latent space loss and pixel space loss (L2 loss), the quality of video generation is optimized to ensure that the generated lip region is aligned with the audio.

KeySync project address

• Project official website:https://antonibigata.github.io/KeySync/
• GitHub repository:https://github.com/antonibigata/keysync
• HuggingFace model library:https://huggingface.co/toninio19/keysync
• arXiv technical paper:https://arxiv.org/pdf/2505.00497
• Experience the demo online:https://huggingface.co/spaces/toninio19/keysync-demo

KeySync application scenarios

• Automatic voiceoverUsed for multilingual content production such as film, television, and advertising, improving the alignment between voice-over and lip movements.
• Virtual avatar: Generate synchronized lip movements for virtual characters to enhance the realism of virtual avatars.
• videoconference: Optimize lip-sync in remote communication to improve user experience.
• Accessible content: To help people with hearing impairments better understand video content.
• Content repair: Fix or replace lip movements in the video to improve content quality.