Ingredients - A framework for generating custom videos from multiple ID photos, based on the combination of multiple ID photos and video diffusion.
Ingredients is a powerful framework based on combining multiple identity (ID) photos with a video diffusion Transformer for custom video creation. Ingredients enables highly customized video creation based on three core modules...
What are Ingredients?
Ingredients is a powerful framework based on combining multiple identity-specific (ID) photos with a video diffusion Transformer for customized video creation. Ingredients achieves highly customized video generation based on three core modules: a face extractor, a multi-scale projector, and an ID router. The face extractor captures facial features for each identity from both global and local perspectives; the multi-scale projector maps these features into the context of the video diffusion model; and the ID router dynamically assigns and combines features from multiple identities to corresponding temporal and spatial regions. Based on a carefully designed multi-stage training protocol, Ingredients can generate personalized videos with high identity fidelity and content flexibility without explicit constraints.
Main functions of Ingredients
- Maintain identity consistencyMaintain consistency in the identity of people across multiple reference images in the generated video.
- Flexible content controlIt allows users to precisely control video content using text prompts.
- High-quality video generationGenerate video content with high visual quality and natural transitions.
- Customization without trainingIt enables customized video generation without requiring model training or fine-tuning for each new identity.
Ingredients' technical principles
- Facial extractorThis function is responsible for extracting facial features for each identity from the input reference image. It employs a combined global and local approach, extracting overall facial information from a global perspective and detailed features from a local perspective, ensuring the diversity and accuracy of facial features in the generated video.
- Multiscale projectorThe extracted facial features are embedded and mapped into the image query context of the video diffusion transformer. Multi-scale feature fusion and cross-attention mechanisms are used to enable effective interaction between facial features and visual tokens in the video diffusion model, accurately reflecting the identity information of the person during the generation process.
- ID RouterThis function is responsible for dynamically allocating and combining multiple identity features within the temporal and spatial regions of video generation. Based on a location-aware routing network, it assigns each potential facial region to a unique identity feature, avoiding the mixing and confusion of identity features and ensuring clear distinction and consistent representation of different individuals in the generated video.
Ingredients project address
- GitHub repository:https://github.com/feizc/Ingredients
- HuggingFace model library:https://huggingface.co/feizhengcong/Ingredients
- arXiv technical paper:https://arxiv.org/pdf/2501.01790
Application scenarios of Ingredients
- Entertainment CreationTo create a multi-scene music video for a virtual idol, maintaining consistency in facial features and style, and enhancing fan interaction.
- Advertising industryCustomize different styles of advertisements for fashion brands, showcasing the fashionable looks of the target audience in scenarios such as campus and streets, and enhance brand appeal.
- Education and teachingIn language learning applications, create videos of international conference scenes so learners can watch exchanges between representatives from different countries and learn business English and cross-cultural communication.
- social mediaUsers create family history videos, combining old photos and oral stories to showcase the lives of family members in different eras, and share them on social media to resonate with others.
- Virtual RealityIn virtual reality tourism applications, videos are generated showing users touring attractions with virtual guides. The guides provide detailed explanations of the attractions' history and culture, enhancing the realism of the tourism experience.