OmniSync - A universal lip-syncing framework jointly launched by Renmin University, Kuaishou, and Tsinghua University.

What is OmniSync?

OmniSync is a general-purpose lip-syncing framework jointly developed by Renmin University of China, Kuaishou Technology, and Tsinghua University. Based on diffusion transformers, it achieves precise synchronization between lip movements and speech in videos. OmniSync directly edits video frames using a maskless training paradigm, eliminating the need for reference frames or explicit masks. It supports unlimited inference duration while maintaining natural facial dynamics and identity consistency. OmniSync introduces progressive noise initialization and Dynamic Spatiotemporal Classifier Free Guidance (DS-CFG) mechanisms based on stream matching to address the issue of weak audio signals, ensuring accurate lip-syncing. OmniSync establishes the AIGC-LipSync benchmark to evaluate the lip-syncing performance in AI-generated videos.

OmniSync's main functions

• Maskless trainingDirectly edit video frames without reference frames or masks, and support unlimited duration inference.
• Identity preservation: Ensure consistency between head posture and identity, while precisely modifying the mouth area.
• Enhance audio conditionsBased on a dynamic spatiotemporal guidance mechanism, the problem of weak audio signals is solved.
• General compatibilitySuitable for stylized characters, non-human entities, and AI-generated content.
• Infinite Duration DeductionMaintain natural facial dynamics and consistency over time.
• occlusion robustnessMaintain high-quality lip-sync under complex conditions such as facial obscuration.

OmniSync's technical principles

• Maskless training paradigmIt enables direct cross-frame editing based on diffusion transformers, eliminating the need for explicit masks or reference frames. It employs an iterative denoising learning mapping function and introduces a time-step-dependent sampling strategy, using different datasets for different denoising stages to ensure stable learning.
• Asymptotic noise initializationBased on flow matching, control noise is injected into the original frame, and only the final denoising step is performed to maintain spatial consistency. It supports precise mouth region modification and effectively solves the problems of pose inconsistency and identity drift.
• Dynamic Spatiotemporal Classifier Free Guidance (DS-CFG)This provides fine-grained control over audio impact, based on spatiotemporally adaptive guidance to balance audio condition intensity. Spatial adaptive guidance uses a Gaussian-weighted spatial guidance matrix to concentrate guidance intensity in the mouth region. Temporally adaptive guidance gradually reduces guidance intensity as the denoising process progresses, ensuring strong guidance in the early and mid-stages of diffusion and reducing interference during later detail refinement.

OmniSync project address

• Project official website:https://ziqiaopeng.github.io/OmniSync/
• arXiv technical paper:https://arxiv.org/pdf/2505.21448

Application scenarios of OmniSync

• Film and television dubbingTo achieve precise matching between the character's lip movements and the voice acting.
• Virtual RealityProvides realistic lip-syncing for virtual characters, enhancing immersion.
• AI content generationImprove the naturalness of lip-sync in AI-generated videos.
• videoconferenceImprove lip-sync performance in remote communication.
• Game developmentEnhance the lip-syncing of game characters and improve interactivity.