AnchorCrafter - An AI-powered virtual livestreaming e-commerce video production technology jointly launched by the Chinese Academy of Sciences and Tencent.
AnchorCrafter is an intelligent video production system based on a diffusion model, used to automatically generate high-fidelity, anchor-style product promotion videos. It integrates human-object interaction (HOI) with gesture-guided human video generation...
What is AnchorCrafter?
AnchorCrafter is an intelligent video production system based on a diffusion model, used to automatically generate high-fidelity, anchor-style product promotion videos. It integrates Human-Object Interaction (HOI) into pose-guided human video generation, achieving high fidelity in object appearance and motion control, and managing complex human-object interactions. The system employs HOI-appearance perception and HOI-motion injection techniques, along with HOI-region reweighting loss training objectives, to enhance the learning of object details and ensure consistency between human appearance and movement during video generation. AnchorCrafter outperforms existing methods in object appearance preservation, interaction perception, and video quality, offering new possibilities for online advertising and consumer engagement.
AnchorCrafter's main functions
- High-quality video generationAutomatically create high-fidelity product promotion videos in a live streamer style.
- Human-Object Interaction (HOI) IntegrationNaturally integrate the interaction between people and objects into the video to enhance its realism and interactivity.
- Appearance maintainedPreserve the appearance details of objects in the video to ensure that the appearance of objects is accurate when viewed from multiple perspectives.
- Motion controlPrecisely control the trajectory of objects, coordinating it with the movements of the character.
- Mutual occlusion managementTo address occlusion issues during interactions between people and objects, maintaining the continuity and naturalness of the video.
- Detail-enhanced learning: Enhance the learning of object details during training based on HOI-region reweighting loss.
AnchorCrafter's technical principles
- Video diffusion modelBased on the diffusion model architecture, video frames are processed using diffusion UNet and variational autoencoder (VAE) to encode video sequences into the latent space and reconstruct high-quality video frames from noise.
- HOI - Appearance Perception:
- Multi-view feature fusion: Use multi-view object reference images to extract the appearance features of objects, and enhance the model's ability to recognize the shape and texture of objects.
- Human-object dual adapterBased on replacing the cross-attention layer in UNet, we achieve better separation of features between people and objects, avoiding appearance entanglement.
- HOI - Motion Injection:
- Object trajectory controlUsing depth maps as input, this method processes depth information using a lightweight convolutional network to control the motion trajectory of objects in a video.
- Mutual occlusion handlingBy combining 3D hand mesh input, the occlusion problem when the character's hand interacts with objects is handled, ensuring the naturalness and accuracy of the interaction.
- HOI-region reweighting lossDuring training, the weights of the hand-object interaction region are increased, allowing the model to pay more attention to these regions and improve the learning and generation quality of object details.
AnchorCrafter's project address
- Project official website:cangcz.github.io/Anchor-Crafter
- GitHub repository:https://github.com/cangcz/AnchorCrafter(Opening soon)
- arXiv technical paper:https://arxiv.org/pdf/2411.17383
Application scenarios of AnchorCrafter
- Online shopping platformAutomatically generate product introduction videos to enhance the attractiveness of product pages and increase users' willingness to buy.
- Social media marketingIt provides tools for brands and individual creators to produce engaging product promotion content, increase fan engagement, and boost brand exposure.
- TV commercial productionQuickly produce high-quality TV commercials, reducing the costs and time of traditional filming.
- Virtual live-streaming e-commerceUsing virtual anchors in live streaming to showcase and promote products improves live streaming efficiency and viewer experience.
- Education and Training: Create instructional videos that simulate actual processes, such as cooking and handicrafts, to enhance learning effectiveness.