AniPortrait - Tencent's open-source photo-lip-sync video generation framework

What is AniPortrait?

AniPortrait is an open-source AI video generation framework for lip-syncing from photos, similar to Alibaba's EMO, capable of generating high-quality animations from audio and a reference portrait image. AniPortrait works in two stages: first, it extracts 3D facial features from the audio and converts them into 2D facial markers; then, using a diffusion model and motion module, it transforms these markers into coherent and realistic animations. The framework's advantage lies in the high naturalness and diversity of its generated animations, while providing flexibility in editing and reproducing facial movements.

AniPortrait's official website

• GitHub repository:https://github.com/Zejun-Yang/AniPortrait
• arXiv research paper:https://arxiv.org/abs/2403.17694
• Hugging Face Model:https://huggingface.co/ZJYang/AniPortrait/tree/main
• Hugging Face Demo:https://huggingface.co/spaces/ZJYang/AniPortrait_official

AniPortrait Features

• Audio-driven animation generation:AniPortrait can automatically generate facial animations synchronized with speech based on the input audio file, including lip movements, facial expressions, and head posture.
• High-quality visual effects:By using a diffusion model and motion modules, AniPortrait is able to produce high-resolution, visually realistic portrait animations, providing an outstanding visual experience.
• Time Consistency:This framework ensures the continuity of the animation in time, making the characters' movements smooth and natural, without abrupt jumps or inconsistencies.
• Flexibility and controllability:By utilizing 3D facial representations as intermediate features, AniPortrait offers flexibility in animation editing, allowing users to further customize and adjust the generated animations.
• Precise capture of facial expressions and lip movements:With its improved PoseGuider module and multi-scale strategy, AniPortrait is able to accurately capture and reproduce subtle lip movements and complex facial expressions.
• Consistency with reference image:The framework ensures that the generated animation is visually consistent with the original portrait by integrating appearance information from reference images, thus avoiding the problem of identity mismatch.

AniPortrait's working mechanism

AniPortrait mainly consists of two modules: Audio2Lmk and Lmk2Video.

1. Audio2Lmk module (audio to 2D facial markers)

The goal of the Audio2Lmk module is to extract a series of facial expressions and lip movements into a 3D facial mesh and head pose information from audio input. First, a pre-trained wav2vec model is used to extract audio features. This model accurately identifies pronunciation and intonation in the audio, which is crucial for generating realistic facial animation. Then, these audio features are used to transform the image into a 3D facial mesh through two fully connected layers. For head pose prediction, a wav2vec network is also used as the backbone, but weights are not shared, as pose is more related to rhythm and intonation in the audio. Furthermore, a transformer decoder is used to decode the pose sequence, and audio features are integrated into the decoder through a cross-attention mechanism. Finally, perspective projection is used to convert the 3D mesh and pose information into a sequence of 2D facial markers.

2. Lmk2Video module (2D facial markers to video)

The Lmk2Video module is responsible for generating temporally consistent, high-quality portrait videos based on reference portrait images and a series of facial markers. Inspired by the AnimateAnyone network architecture, it uses Stable Diffusion 1.5 as its backbone and incorporates a temporal motion module to convert multi-frame noisy input into a series of video frames. Furthermore, a ReferenceNet with the same structure as SD1.5 is introduced to extract appearance information from the reference image and integrate it into the backbone network, ensuring consistent facial identity across the videos. To improve the accuracy of lip movement capture, the PoseGuider module's design is enhanced by employing ControlNet's multi-scale strategy and using reference image markers as additional input. A cross-attention module facilitates the interaction between reference markers and target markers in each frame, helping the network better understand the relationship between facial markers and appearance.