FaceShot - A portrait animation generation framework jointly developed by Tongji University and Shanghai AI Lab.
FaceShot is a novel, training-free portrait animation generation framework developed by Tongji University, Shanghai AI Lab, and Nanjing University of Science and Technology. It utilizes an appearance-guided landmark matching module and a coordinate-based landmark relocalization module to generate various character animations...
What is FaceShot?
FaceShot is a novel, training-free portrait animation generation framework developed by Tongji University, Shanghai AI Lab, and Nanjing University of Science and Technology. It uses an appearance-guided landmark matching module and a coordinate-based landmark relocalization module to generate accurate and robust landmark sequences for various characters. Based on semantic correspondences from a latent diffusion model, it generates facial motion sequences across a wide range of character types. The landmark sequences are then input into a pre-trained landmark-driven animation model to generate animated videos. FaceShot overcomes the limitations of relying solely on real-world portrait landmarks, is applicable to any stylized character and driven video, and can be used as a plugin compatible with any landmark-driven animation model, significantly improving overall performance.
FaceShot's main functions
- Character animation generationGenerates smooth and natural facial animations for various types of characters while preserving their original features.
- Cross-disciplinary animationIt supports animation of non-human characters (such as toys, animals, etc.) driven by human videos, expanding the application scope of portrait animation.
- No training requiredHigh-quality animations can be generated directly without additional training or fine-tuning for each character or driving video.
- compatibilityIt integrates seamlessly with any landmark-driven animated model as a plugin.
FaceShot's technical principles
- Appearance-guided landmark matching moduleBased on the semantic correspondence of the latent diffusion model and combined with prior appearance knowledge, accurate facial landmarks are generated for any character. Diffusion features are extracted from reference and target images using the inverse process of DDIM, and appearance differences between different domains are reduced based on image cues. Cosine distance is used for landmark matching to ensure semantic consistency, and an appearance gallery is introduced to further optimize the matching effect.
- Coordinate-based landmark relocation moduleThis module captures subtle facial movements in video based on coordinate system transformations and generates a sequence of landmarks aligned with them. The module is divided into two stages: global motion and local motion. Global motion calculates the overall translation and rotation of the face, while local motion performs relative and point repositioning on parts such as the eyes, mouth, nose, eyebrows, and facial boundaries. Based on a simple coordinate transformation formula, the module can accurately capture both global and local facial movements, generating a stable sequence of landmarks.
- Landmark-driven animated modelsThe module inputs the generated landmark sequence into a pre-trained animation model (such as MOFA-Video) to generate the final animated video. The landmark sequence is input as an additional condition into the U-Net of the animation model to ensure that the model can accurately track the motion within the landmark sequence. Based on this approach, the animation model can generate animation effects consistent with the driving video using the landmark sequence, maintaining the character's visual identity and achieving high-quality portrait animation generation.
FaceShot project address
- Project official website:https://faceshot2024.github.io/faceshot/
- GitHub repository:https://github.com/open-mmlab/FaceShot
- arXiv technical paper:https://arxiv.org/pdf/2503.00740
Application scenarios of FaceShot
- Film and EntertainmentGenerate vivid animations for characters in movies and TV series, enhancing visual effects.
- Game developmentQuickly generate game character animations to enhance expressiveness and fun.
- EducationTo make educational content more engaging and to enhance students' learning interest and teaching interaction.
- Advertising and MarketingGenerate animated brand mascots to enhance brand image and user engagement.
- VR/AR applicationsGenerate virtual character animations to enhance immersion and interactive experience.