ConsisID - A text-to-video generation model jointly developed by Peking University, Pengcheng Laboratory, and other institutions.
ConsisID is a text-to-video (IPT2V) generation model developed by Peking University and Pengcheng Laboratory, among other institutions. It maintains the consistency of identity among individuals in a video based on frequency decomposition technology. The model uses a no-tuning (tuning-free) approach...
What is ConsisID?
ConsisID is a text-to-video (IPT2V) generation model developed by Peking University and Pengcheng Laboratory, among other institutions. It maintains the consistency of identities among individuals in videos based on frequency decomposition technology. The model uses a tuning-free Diffusion Transformer (DiT) architecture, combining low-frequency global features and high-frequency intrinsic features, and employs a hierarchical training strategy to generate high-quality, editable videos with strong identity consistency. ConsisID surpasses existing technologies across multiple evaluation dimensions, driving the development of identity-consistent video generation technology.
The main functions of ConsisID
- Identity preservationMaintain consistency in the identities of individuals during video generation, ensuring that the characteristics of individuals in the video match the provided reference images.
- High-quality video generationGenerate visually realistic and detailed video content.
- No fine-tuning requiredAs a tuning-free model, it does not require fine-tuning for each new case, thus lowering the barrier to entry.
- EditabilityIt supports users controlling video content with text prompts, including character movements, facial expressions, and backgrounds.
- Generalization abilityIt can handle people outside the training data domain, improving the model's generalization ability.
The technical principle of ConsisID
- Frequency decomposition:
- Low frequency controlA global face feature extractor is used to encode reference images and facial key points into low-frequency features, which are then integrated into the shallow layers of the network to alleviate training difficulty.
- High frequency controlWe designed a local facial feature extractor to capture high-frequency details and inject them into the Transformer module, thereby enhancing the model's ability to preserve fine-grained features.
- Hierarchical training strategy:
- coarse to fine trainingFirst, let the model learn global information, and then refine it to local information to maintain the consistency of the video in both spatial and temporal dimensions.
- Dynamic Mask LossThe calculation of the loss function is constrained by the face mask, allowing the model to focus on the face region.
- Dynamic Cross-Face LossIntroducing cross-face reference images improves the model's ability to generalize to unseen identities.
- Feature fusionFeatures are extracted using a face recognition backbone network and CLIP image encoder, and features are fused based on Q-Former to generate intrinsic identity features containing high-frequency semantic information.
- Cross-attention mechanismBased on the cross-attention mechanism, the model can interact with the visual labels generated by the pre-trained model, effectively enhancing the high-frequency information in DiT.
ConsisID's project address
- Project official website:pku-yuangroup.github.io/ConsisID
- GitHub repository:https://github.com/PKU-YuanGroup/ConsisID
- HuggingFace model library:https://huggingface.co/datasets/BestWishYsh/ConsisID
- arXiv technical paper:https://arxiv.org/pdf/2411.17440
- Experience the demo online:https://huggingface.co/spaces/BestWishYsh/ConsisID
Application scenarios of ConsisID
- Personalized entertainmentUsers create virtual avatars that resemble themselves or designated characters for use on social media platforms or for personal entertainment.
- Virtual streamerIn news broadcasts or live online streams, virtual anchors generated using ConsisID work 24/7.
- Film and television productionIn film post-production, it is used to create characters in special effects scenes or to create entirely new virtual characters.
- gaming industry: Provide prototypes for game character design, or generate non-player characters (NPCs) in the game that resemble the player.
- Education and simulation trainingCreate historical figures or simulate specific scenarios for educational purposes or professional training, such as medical simulations or driving training.