HOMIE - An open-source digital human video generation framework from the Hong Kong University of Science and Technology

What is HOMIE?

HOMIE is an open-source digital human video generation framework from the Hong Kong University of Science and Technology. Based on the Wan2.1-T2V-14B backbone network and integrating the Qwen3-VL multimodal large model, the framework can uniformly handle four elements: digital human, product, logo, and text. It accurately solves four major challenges: human-object interaction, brand labeling, OCR text fidelity, and multi-view consistency. Model training requires only 5.5K steps/approximately 10,000 A100 hours, and its OCR accuracy is 38.7% higher than the most powerful open-source model. It supports inference from 480P single-card to 720P multi-card, significantly lowering the barrier to entry for video production in e-commerce, virtual anchor, and other scenarios.

HOMIE's main functions

• Personalized human-object interactive videosIt supports multiple people and objects appearing in the same video at the same time, maintaining the consistency of the appearance of each subject, and realizing natural and reasonable interactive actions (such as holding, displaying, and passing products).
• Logo / Abstract Concept Precisely AppliedUsing the semantic reasoning capabilities of Multimodal Large Model (MLLM), abstract concepts such as brand logos are automatically attached to the most semantically relevant objects without explicitly describing the positional relationships in the prompts.
• OCR high-fidelity text generationIn conjunction with OCR reference images, it ensures that the text on product packaging and labels is clear, legible, and unblurred in the generated video, solving the pain point of "text becoming a blurry mess" in AI video generation.
• Multi-view consistency generationGiven a multi-view reference image of the same object, it can maintain a consistent appearance and detail when the object rotates or moves in the video, which is suitable for display scenarios such as figurines, 3D models, and dolls.

HOMIE's technical principles

• Overall architecture: The Alibaba Wan2.1-T2V-14B uses DiT as its backbone and introduces Qwen3-VL-2B-Thinking as a visual understanding module. Together with the UmT5 text encoder and VAE visual encoder, it forms a three-way parallel generation architecture of "text + multimodal semantics + vision".
• MLLM integration strategy: Without disrupting the original text-based video alignment relationships or incurring expensive realignment costs, the semantic reasoning capabilities of a multimodal large model are injected into DiT, enabling the model to automatically understand the potential relationships between reference images.
• Global Multimodal Self-Attention Guidance (GMG): In the self-attention layer of DiT, the semantic features extracted by MLLM are globally aligned with the visual tokens of VAE. Through projection, pooling and affine transformation, each video token in the generation process can "see" the global semantic information of the reference graph.
• Modal-reference embedding (MRE): Modal embeddings are assigned to MLLM feature tokens and VAE visual tokens respectively, independent reference embeddings are assigned to different reference images, and multi-view/OCR reference image tokens of the same subject are linked together to avoid information confusion.
• Three-stage progressive training: The training process, from single-subject 480P to multi-subject 480P to high-definition 720P, took only 5.5K steps, far fewer than the tens of thousands of steps required by similar methods.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use HOMIE

• Environmental preparationClone the HOMIE code repository from GitHub to your local machine and install the necessary environment dependencies.
• Weight DownloadDownload the pre-trained weights released by HuggingFace to the specified model directory.
• Material preparationPrepare reference images, including multiple main materials such as photos of people, product images, logo images, or OCR text images.
• Prompt word writingWrite text prompts to accurately describe the interactions and scenes between people and objects in the target video.
• Running inferenceBy running the inference script, HOMIE will automatically fuse multimodal reference information and generate personalized videos.
• Resolution selectionA single GPU can directly generate 832×480 resolution video, while multiple GPUs with FSDP can be expanded to 720P high-definition output.

HOMIE's core advantages

• Unified frameworkA single framework handles four major tasks simultaneously: human-object interaction, logo labeling, OCR text fidelity, and multi-view consistency, without the need to switch between different models for different scenarios.
• High-efficiency trainingTraining can be completed in just 5.5K steps / approximately 10,000 A100 hours, at a cost far lower than the 30,000–40,000 steps typically required by similar methods.
• MLLM lossless integration: Inject the semantic reasoning capabilities of multimodal large models into the video generation pipeline without sacrificing the controllability of the text encoder or incurring expensive realignment costs.
• Quality LeadingThe OCR accuracy improved by 38.7%, the multi-view consistency improved by 8.2%, and in a study of 40 users, 64.7% voted it as the best in terms of overall quality.
• Flexible reasoningIt supports 480P single-card to 720P multi-card inference and can output 1280×720 vertical screen video, directly adapting to live streaming e-commerce and short video production.

HOMIE's project address

• Project official website：https://yiyangcai.github.io/homie-page.github.io
• GitHub repository：https://github.com/YIYANGCAI/HOMIE
• HuggingFace model library：https://huggingface.co/yychai/homie-r2v-wan2.1
• arXiv technical paper：https://arxiv.org/pdf/2607.18217

Comparison of HOMIE's similar products

HOMIE's application scenarios

• E-commerce live streaming salesGenerates promotional videos featuring a digital human holding a specified product and performing specific actions, supporting 1280×720 vertical screen output and directly adapting to short video platforms.
• Brand advertising productionAutomatically attaches the brand logo to semantically relevant products without requiring explicit specification of the positional relationship in the prompt.
• Virtual anchor / Digital humanMaintaining a consistent digital identity while interacting naturally with a variety of products.
• Product Multi-Perspective DisplayGiven a multi-view reference image of a figurine, 3D model, or doll, generate a rotating display video that maintains a consistent appearance from all angles.
• Packaging text authenticity guaranteed: Used in conjunction with OCR reference images to ensure that the text on product packaging and labels is clearly readable in videos, suitable for scenarios such as food, cosmetics, and health products where ingredients/instructions need to be displayed.