ID-Animator - A personalized character video generation framework launched by Tencent and others.

What is ID-Animator?

ID-Animator, developed by researchers from Tencent Photon Studio, the University of Science and Technology of China, and the Hefei Institutes of Physical Science, Chinese Academy of Sciences, is a zero-shot human video generation technology. It can generate personalized videos based on a single reference facial image, preserving the identity features of the person in the image and adjusting the video content based on text prompts. This framework achieves efficient video generation by combining a pre-trained text-to-video diffusion model and a lightweight face adapter, without requiring additional training for specific identities. ID-Animator improves the identity fidelity and generation quality of videos by constructing a specialized dataset and employing a random facial reference training method.

Main functions of ID-Animator

• Modify video character (Recontextualization)ID-Animator can alter the contextual information of characters in a video based on provided reference images and text. For example, it can adjust a character's hairstyle, clothing, background, and even perform specific actions using text prompts, thereby creating entirely new backstory for the character.
• Age and gender alterationThis model can adjust the age and gender of characters in a video as needed to adapt to different video content and style requirements. For example, it can generate videos of young people aging or boys transforming into girls.
• Identity MixingID-Animator can blend features from two different identities to generate videos with combined characteristics in different proportions, which is very useful when creating new characters or blending features from real-world people.
• Integration with ControlNetID-Animator is compatible with existing fine-grained conditional modules such as ControlNet. By providing single or multiple frames of control images, it can generate video sequences that are closely integrated with the control images, which is very useful when generating videos of specific actions or scenes.
• Community Model IntegrationID-Animator can also integrate with community models (such as those on Civitai) and works effectively even without training on these models, maintaining the stability of facial features and dynamic generation.

ID-Animator official website entrance

• Official project homepage:https://id-animator.github.io/
• arXiv research paper:https://arxiv.org/abs/2404.15275
• GitHub source code:https://github.com/ID-Animator/ID-Animator

How ID-Animator works

1. Pre-trained text-to-video diffusion modelID-Animator uses a pre-trained text-to-video (T2V) diffusion model as its foundation, which is able to generate video content based on text prompts.
2. Face AdapterTo generate videos consistent with a specific identity, ID-Animator introduces a lightweight face adapter. This adapter encodes identity-related embedding information by learning latent facial queries.
3. Identity-oriented dataset constructionResearchers constructed an identity-oriented dataset, which includes decoupled human attributes and action captioning techniques, as well as facial features extracted from a constructed pool of facial images.
4. Random facial reference training methodID-Animator is trained using randomly sampled facial images. This approach helps to separate identity-irrelevant image content from identity-related facial features, allowing the adapter to focus on learning identity-related features.
5. Fusion of text and facial featuresID-Animator combines textual and facial features and fuses them through an attention mechanism to generate videos that both match the textual description and retain the identity features.
6. Generation processWhen generating a video, the ID-Animator first receives a reference facial image and a corresponding text cue. The facial adapter encodes the features of the reference image into embeddings, and then feeds these embeddings along with the text features into a diffusion model to ultimately generate the video.
7. Optimization and trainingTo improve model performance, the ID-Animator training process includes using random facial images as references to reduce the influence of identity-irrelevant features in the reference images, and optimizing video generation quality through techniques such as classifier-free guidance.
8. Compatibility and scalabilityID-Animator is designed to be compatible with a variety of pre-trained T2V models, such as AnimateDiff, which makes it easy to integrate into existing systems and extend to different applications.