StableAnimator - An end-to-end identity consistency video diffusion framework jointly developed by Fudan University, Microsoft, and other institutions.

What is StableAnimator?

StableAnimator is an end-to-end high-quality identity-preserving video diffusion framework jointly developed by Fudan University, Microsoft Research Asia, Huya, and Carnegie Mellon University. Based on a reference image and a series of poses, StableAnimator can directly synthesize high-fidelity videos that maintain consistent person identities without any post-processing tools. The StableAnimator framework is based on computational image and facial embedding, uses a global content-aware facial encoder to optimize facial information, introduces a distributed-aware ID adapter to reduce temporal interference, and employs an optimization method based on the Hamilton-Jacobi-Bellman equation to improve facial quality during the inference stage.

Main functions of StableAnimator

• Identity Preservation Video SynthesisStableAnimator can synthesize video content that maintains the consistency of a person's identity based on provided reference images and pose sequences.
• No post-processing requiredUnlike traditional animation models, StableAnimator can generate high-quality animations without relying on any post-processing tools, such as face swapping tools or face restoration models.
• High fidelityThe videos generated directly by the framework have high fidelity, rich detail, and closely resemble the dynamics and appearance of people in the real world.
• end-to-end frameworkAs an end-to-end video diffusion framework, StableAnimator integrates training and inference modules to ensure identity consistency throughout the animation generation process.

The technical principles of StableAnimator

• Image and face embeddingStableAnimator uses off-the-shelf extractors to compute image and face embeddings separately, providing foundational features for subsequent identity preservation.
• Global Content-Aware Face EncoderBased on the interaction with image embedding, the facial encoder can further optimize facial features and enhance the model's ability to perceive the global layout of the reference image.
• Distributed Aware ID AdapterThis novel component prevents interference caused by the time layer while preserving identity information based on alignment operations, ensuring identity consistency between video frames.
• Hamilton-Jacobi-Bellman (HJB) equation optimizationDuring the inference phase, facial optimization is performed based on the HJB equation to further enhance facial quality. Combined with the diffusion denoising process, an optimized path guides the model towards optimal identity consistency.
• Integration into diffusion denoising processThe solution to the HJB equation is integrated into the diffusion denoising process, which constrains the denoising path and helps preserve identity information.
• Multipath processingThe reference image is processed based on three paths: VAE encoding, CLIP image encoding, and Arcface face encoding, which use the information to modulate the synthetic appearance and ensure identity consistency.

StableAnimator project address

• Project official website:francis-rings.github.io/StableAnimator
• GitHub repository:https://github.com/Francis-Rings/StableAnimator
• HuggingFace model library:https://huggingface.co/FrancisRing/StableAnimator
• arXiv technical paper:https://arxiv.org/pdf/2411.17697

Application scenarios of StableAnimator

• Film and video productionGenerate or enhance character movements in movies and videos, making action scenes smoother and more realistic, while reducing the cost and risk of location shooting.
• Game developmentCreate characters with realistic movements and expressions in the game to enhance player immersion and gaming experience.
• Virtual Reality (VR) and Augmented Reality (AR)Create realistic virtual characters in VR and AR to provide users with a more authentic interactive experience, such as virtual tour guides and virtual assistants.
• Social media and live streamingDevelop virtual anchors and influencers to livestream and interact on social media, attracting more fans and increasing user engagement.
• Advertising and MarketingCustomize personalized advertising content, using virtual characters to attract the target audience and improve the appeal and memorability of the ads.