AB
AB
AiBoss

All Categories

project

FantasyID - An identity-preserving video generation framework jointly launched by Alibaba and Beijing University of Posts and Telecommunications

FantasyID is a novel identity-preserving video generation (IPT2V) framework developed by Alibaba Group and Beijing University of Posts and Telecommunications. It generates high-quality, identity-consistent videos based on enhanced facial knowledge. FantasyID is based on the Diffusion Transformer (D...

What is FantasyID?

FantasyID is a novel identity-preserving video generation (IPT2V) framework developed by Alibaba Group and Beijing University of Posts and Telecommunications. It generates high-quality, identity-consistent videos based on enhanced facial knowledge. FantasyID utilizes diffusion transformers to introduce 3D facial geometry priors, ensuring the stability and plausibility of facial structures in video synthesis. It employs a multi-view face enhancement strategy to avoid simply copying reference faces, increasing the dynamism of facial expressions and head poses. Finally, it uses a learnable hierarchical perceptual injection mechanism to selectively inject fused 2D and 3D features into each layer of the diffusion model, balancing identity preservation and motion dynamism.

Main functions of FantasyID

  • Identity PreservationThe goal is to ensure that the facial features of the people in the generated video are consistent with the input reference image, and to maintain a high degree of identity similarity in complex movements and facial expressions.
  • Dynamic enhancementIncrease the diversity of facial expressions and head postures to avoid the "copy and paste" phenomenon in generated videos.
  • High-quality video generationBy combining 3D facial geometric priors and 2D visual features, videos with stable structures and rich details are generated while maintaining the spatiotemporal coherence of the videos.
  • No fine-tuning requiredIt eliminates the need for additional model fine-tuning for each input image, enabling efficient and flexible identity-preserving video generation, making it suitable for large-scale applications.

The technical principles of FantasyID

  • 3D facial geometric priorBased on the DECA framework, 3D facial structures (such as shape point clouds) are extracted from input face images, providing stable geometric constraints for video generation and ensuring the stability of facial structures during dynamic changes.
  • Multi-view face enhancementConstruct a multi-view face set, sample face images from different angles, enhance the model's understanding of 2D facial appearance features, avoid the uniformity of faces in generated videos, and improve dynamic performance.
  • Feature fusionThe extracted 2D visual features and 3D geometric features are combined through a fusion transformer to generate a comprehensive facial descriptor, which is used to guide video generation.
  • Layered sensing signal injectionTo address the hierarchical characteristics of diffusion transformers, a learnable hierarchical perception mechanism is designed to selectively inject fused features into different levels, achieving a balance between identity preservation and dynamic performance.
  • diffusion modelA diffusion-based generative framework reconstructs video content that conforms to textual descriptions and identity features from noise through a stepwise denoising process.

FantasyID project address

Application scenarios of FantasyID

  • Personalized virtual avatarUsed for virtual social interaction, metaverse, and games to generate virtual avatars that are consistent with the user's identity.
  • Virtual content creationGenerate dynamic video content to assist in the production of films, advertisements, and short videos, reducing creation costs.
  • Virtual Customer Service and Digital HumansCreate natural and realistic digital human avatars for online customer service and intelligent assistants to enhance the interactive experience.
  • Virtual makeup and clothing try-onBy combining e-commerce and the beauty industry, dynamic videos of makeup or clothing try-ons can be generated to optimize the shopping experience.
  • Interactive EducationGenerate dynamic videos of teachers or trainers for use in online courses and simulated scenarios to enhance teaching effectiveness.
Back