ReSyncer - An AI video editing tool jointly launched by Tsinghua University and Baidu

What is ReSyncer?

ReSyncer is an AI video editing tool jointly developed by Tsinghua University and Baidu. It generates high-quality lip-sync videos that are synchronized with the audio, driven by sound. ReSyncer uses Style-SyncFormer to analyze the sound and create a 3D facial model, combining it with the target video to generate a synchronized and expressive virtual character. ReSyncer supports personalized fine-tuning, speech style switching, and face-swapping functions, making it suitable for scenarios such as virtual host and performer creation, and live streaming. It excels in synchronizing audiovisual facial information.

ReSyncer's main functions

• Lip-syncGenerates lip movements synchronized with the given audio.
• Style transfer: Transfer specific speaking styles or facial expressions to target videos.
• Personalized fine-tuning: Quickly adjust the generated facial animation to match the facial features of a specific person.
• Video-driven lip-syncUse facial images from the target video to drive lip-sync animation.
• face-swapping technologyReplace one person's facial features with another person's for identity transformation or special effects.

ReSyncer's technical principles

• 3D facial model generationUsing Style-SyncFormer, a deep learning model, to predict 3D facial dynamics based on vocal features.
• Stylized facial animationIt learns stylized 3D facial dynamics through the Transformer architecture, achieving precise synchronization of facial expressions and lip movements.
• Style-based generatorsThe predicted 3D facial dynamics are combined with facial images in the target video to generate high-fidelity facial images.
• Facial feature fusionDuring the generation process, a simple insertion mechanism is used to fuse 3D facial mesh information with stylized features, thereby improving the quality and stability of lip synchronization.

ReSyncer project address

Application scenarios of ReSyncer

• Film and video productionIn film and video production, ReSyncer can be used to achieve complex special effects, such as face swapping or lip-syncing, to increase visual appeal.
• Advertising industryIn advertising production, style transfer functionality can be used to create unique visual effects and attract viewers' attention.
• Social media and content creationContent creators can use ReSyncer to enhance their video content, such as creating funny parody videos using face-swapping technology.
• Education and trainingIn language learning or professional training, lip-syncing can help learners better understand and imitate pronunciation.