HunyuanCustom - Tencent's open-source multimodal custom video generation framework

What is HunyuanCustom?

HunyuanCustom is a multimodal-driven, customized video generation framework developed by Tencent's Hunyuan team. HunyuanCustom supports various input conditions, including images, audio, video, and text, and can generate high-quality videos with specific subjects and scenes. By introducing an LLaVA-based text-image fusion module and an image ID enhancement module, HunyuanCustom significantly outperforms existing methods in terms of identity consistency, realism, and text-video alignment. The framework supports both audio-driven and video-driven video generation and is widely used in fields such as virtual human advertising, virtual try-on, and video editing, demonstrating strong controllability and flexibility.

Main functions of HunyuanCustom

• Single-subject video customizationGenerate videos based on input images and text descriptions, ensuring consistency in the identity of the subject.
• Multi-subject video customizationSupports interactive generation of multiple entities and handles complex multi-entity scenarios.
• Audio-driven video customizationIt generates videos based on audio and text descriptions, and supports flexible audio-driven animation.
• Video-driven video customizationSupports object replacement or addition based on video input, used in video editing and object replacement.
• Virtual human advertising and virtual try-onGenerate advertising videos featuring virtual people interacting with products, or conduct virtual try-on demonstrations.
• Flexible scene generationIt generates videos in different scenarios based on text descriptions, supporting diverse content creation.

HunyuanCustom's technical principles

• Multimodal fusion module:
  • Text-image fusion moduleBased on LLaVA, identity information in images is fused with text descriptions to enhance multimodal understanding capabilities.
  • Image ID Enhancement ModuleBased on information stitching along the timeline, the temporal modeling capabilities of video models are used to strengthen the identity features of the subject and ensure the consistency of identity in video generation.
• Audio driver mechanismThe AudioNet module is based on a spatial cross-attention mechanism, which injects audio features into video features to achieve hierarchical alignment between audio and video, and supports audio-driven video generation.
• Video-driven mechanismThe video feature alignment module compresses the input video into the latent space using VAE and performs feature alignment based on the patchify module to ensure consistency with the features of the noise latent variables.
• Identity decoupling moduleThe identity-decoupled video condition module efficiently injects video features into the latent space, supporting video-driven video generation.
• Data processing and augmentationBased on a rigorous preprocessing workflow, including video segmentation, text filtering, subject extraction, and data augmentation, we ensure high-quality input data and improve model performance.

HunyuanCustom's project address

• Project official website:https://hunyuancustom.github.io/
• GitHub repository:https://github.com/Tencent/HunyuanCustom
• HuggingFace model library:https://huggingface.co/tencent/HunyuanCustom
• arXiv technical paper:https://arxiv.org/pdf/2505.04512v1

Application Scenarios of HunyuanCustom

• Virtual Human AdvertisingGenerate advertising videos featuring virtual humans interacting with products to enhance appeal.
• Virtual try-onUpload photos to generate videos of yourself trying on different outfits, enhancing the shopping experience.
• Video editingReplace or add objects in the video to enhance editing flexibility.
• Audio-driven animationGenerate synchronized video animations based on audio, used in virtual live streaming or animation production.
• Educational VideosCombine text and images to generate instructional videos to improve learning outcomes.