Follow-Your-Emoji - A portrait animation framework based on a diffusion model, launched by Tencent and others.

What is Follow-Your-Emoji?

Follow-Your-Emoji is a portrait animation framework based on a diffusion model, developed by researchers from the Hong Kong University of Science and Technology, Tencent Hunyuan, and Tsinghua University. It uses a diffusion model to add target expression sequences to reference portraits, achieving dynamic animation effects. This technology precisely aligns expressions with portraits through expression-aware markers, avoiding the leakage of identity information, while using a facial refinement loss function to enhance the model's ability to capture subtle facial changes. Follow-Your-Emoji supports various styles of portrait animation, including live-action, cartoon, sculptural, and even animal animations, demonstrating a high degree of control and expressiveness.

Features of Follow-Your-Emoji

• Synchronized animated emoticonsUsing diffusion model technology, Follow-Your-Emoji can accurately synchronize predefined or real-time captured emoji sequences onto static reference portraits, enabling dynamic changes in complex expressions such as blinking, smiling, and frowning.
• Identity characteristics preservationDuring the animation process, the framework has specially designed a mechanism to ensure that the key identity features of the reference portrait are preserved, and to prevent the distortion or leakage of identity information even when the facial expressions change drastically.
• Exaggerated facial expressionsThrough facial expression recognition marker technology, the frame can capture and reproduce exaggerated facial expressions, such as the large pupil dilation or contraction commonly seen in cartoon or comic styles, enhancing the expressiveness of animation.
• Multi-style adaptationThe Follow-Your-Emoji frame is not limited to realistic portraits; it can also adapt to and animate portraits of different art styles and forms of expression, such as cartoons, sculptures, and animals, demonstrating its wide applicability.
• Temporal coherenceBy using a facial refinement loss function, the framework takes into account the continuity with the preceding and following frames when generating each frame of animation, ensuring a natural transition and smooth performance of the entire animation sequence in time.
• Long-term animation generationUsing a progressive generation strategy, Follow-Your-Emoji can generate animations that are not only coherent in the short term, but also maintain stability and high quality over long-term playback.
• High degree of controlUsers can precisely control the sequence of facial expressions, thereby making precise adjustments to the animation output and allowing users to customize animation effects according to specific needs, thus achieving personalized creation.

Official website entry for Follow-Your-Emoji

• Official project homepage:https://follow-your-emoji.github.io/
• arXiv technical paper:https://arxiv.org/abs/2406.01900

The technical principle behind Follow-Your-Emoji

1. Framework based on diffusion modelIt uses the Stable Diffusion model as its foundation, an advanced deep learning model capable of generating high-quality image and video content.
2. Expression-Aware LandmarkTools like MediaPipe are used to extract 3D keypoints from dynamic videos, which are then projected onto a 2D plane to create facial expression-aware markers to guide the animation process. These markers focus specifically on key areas of facial expression change, such as the eyes (pupil points) and mouth, to achieve more accurate facial expression synchronization.
3. Facial Fine-Grained Loss FunctionA novel loss function is introduced to guide the model to pay more attention to the details of facial expressions during training, using face masking and expression masking. This loss function helps the model learn how to better capture subtle changes in expression by calculating the difference between the predicted and actual results within the masked area.
4. Multi-style adaptabilityThe frame is designed to adapt to portraits of different styles, whether real people, cartoons, sculptures or animals, and can achieve natural animation effects.
5. Progressive generation strategyTo generate long-term animations, a progressive generation strategy from coarse to fine is adopted. First, keyframes are generated, and then intermediate frames are generated through interpolation to maintain the continuity and stability of the animation.
6. Time attention mechanismAdd a temporal attention layer to the UNet network to maintain temporal consistency and dynamic coherence between animation frames.
7. Pre-training and fine-tuningThe model is pre-trained using a large dataset of expressive training data, and then fine-tuned for specific animation tasks to improve its expressiveness and accuracy.
8. Dataset and benchmark constructionThe team built the EmojiBench benchmark, which includes portrait videos with various styles and expressions, to evaluate and validate the model's performance.
9. Reasoning and Animation GenerationDuring the inference phase, the model combines facial expression awareness markers and temporal attention mechanisms to generate dynamic portrait animations while maintaining the identity features of the reference portrait.
10. User control and customizationUsers can control the animation output by providing different input emoji sequences, achieving highly customized animation effects.