OmniWeaving - A video generation framework jointly developed by Tencent Hunyuan and Zhejiang University, among others.

What is OmniWeaving?

OmniWeaving is a unified video generation framework jointly developed by Zhejiang University, Tencent Hunyuan, and Nanyang Technological University. Breaking through the single-task limitations of traditional open-source models, the framework enables free combination of multimodal data and enhanced inference generation. It can bind interlaced text and image video sequences to generate coherent content, acting like an "intelligent director" to proactively infer complex intentions. The framework employs a three-component architecture of MLLM semantic understanding, MMDit generation, and VAE encoding, and is accompanied by the IntelligentVBench evaluation benchmark. Achieving SoTA performance in open-source unified models, it provides a powerful open-source reference for bridging the gap between open-source and commercial video generation.

OmniWeaving's main functions

• Unified Multimodal GenerationIt seamlessly integrates text, multiple images, and video inputs within a single framework, supports interleaved and free combinations, enables diverse video generation tasks, and eliminates the fragmented limitations of traditional single-task models.
• Timing binding generationIt aligns and binds content from different modalities in time to generate coherent dynamic videos across modalities.
• Reasoning-enhanced creationBy using MLLM's "thinking mode," it can proactively infer complex and ambiguous user intentions, and autonomously plan camera language and narrative logic like an experienced director, upgrading from passive rendering to active creation.
• Advanced semantic understandingWe use a multimodal large language model to map free-form inputs to a high-level semantic space, and combine it with a diffusion Transformer to generate fine-grained video content.
• End-to-end video generationThe integrated workflow from concept understanding to video output supports complex creative needs such as maintaining consistency of roles and style transfer, and achieves the level of open source model SoTA in IntelligentVBench benchmark tests.

How to use OmniWeaving

• Environmental preparationAfter cloning the OmniWeaving repository from GitHub, install the project dependencies in requirements.txt and choose to install either Flash Attention or SageAttention acceleration libraries as needed to optimize inference performance.
• Model DownloadDownload the HY-OmniWeaving model weight file released by Tencent Hunyuan from the HuggingFace platform to a specified local directory.
• Wensheng VideoRun the generation script and specify the task type as t2v. Input text description, set the screen ratio and output path. Optionally, add a thinking mode to let the model infer the intent before generating the video.
• Image and videoUsing the i2v task type, provide the path to the first frame image and action description text, and the model will generate continuous dynamic video based on the static image and prompts.
• First and last frame interpolationSelect the interpolation task, pass in two images (starting frame and ending frame) and a transition description, and the model will automatically generate a coherent video to fill in the intermediate process.
• Multi-image combination generationUpload one to four reference images (such as people, props, and backgrounds) through the reference2v task, and use them with text prompts to create videos that freely combine multiple elements.
• Video editingUpload the source video using the editing task and input editing instructions (such as style transfer or object replacement). The model will intelligently modify the video content based on the text guidance.
• Joint editing of text, images, and videosThis tool uses the tiv2v task to simultaneously input source video and reference images, enabling advanced editing by integrating visual elements from the reference images into the dynamic video scene.

OmniWeaving's project address

• Project official websitehttps://omniweaving.github.io/
• GitHub repositoryhttps://github.com/Tencent-Hunyuan/OmniWeaving
• HuggingFace model libraryhttps://huggingface.co/tencent/HY-OmniWeaving
• arXiv technical paper: https://arxiv.org/pdf/2603.24458

Key information and usage requirements for OmniWeaving

• Project PositioningA unified video generation framework open-sourced by Tencent Hunyuan in collaboration with Zhejiang University and Nanyang Technological University on April 3, 2026.
• Core technologies:
  • MLLM+MMDiT+VAE ArchitectureMultimodal large language model parses interleaved image and video input, diffusion Transformer generates video, and variational autoencoder compresses visual signals.
  • Free Form CombinationSupports timing binding and flexible combination of text, multiple images (1-4), and video inputs.
  • Reasoning-enhanced generationActivate MLLM's "thinking mode" to proactively infer complex user intentions and generate semantically accurate video content.
• Usage Requirements:
  • hardwareMulti-GPU environment (official example uses 8 cards), supports memory optimization options (Flash Attention/SageAttention).
  • software: Python environment, based on PyTorch, requires HunyuanVideo-1.5 dependency to be installed.

OmniWeaving's core advantages

• Unified All-rounderOmniWeaving integrates six major task categories—text-to-video, image-to-video, multi-image combination, and video editing—through a single framework, replacing the fragmented solutions that require multiple dedicated models and achieving unified generation throughout the entire process.
• Free combinationBreaking through the limitations of fixed input formats in traditional models, it supports interleaved multimodal input of 1-4 images, video clips and text, and can perform temporal binding to understand the spatiotemporal relationships between elements, achieving true fusion generation of complex scenes rather than simple splicing.
• Reasoning EnhancementAfter activating the MLLM thinking mode, the model is upgraded from a "renderer" that passively executes instructions to an "intelligent director" that actively infers user intent. It can automatically complete the shot language and narrative logic, significantly reducing the engineering threshold of prompt words for complex creative descriptions.
• Deep Semantic InjectionThe DeepStacking mechanism is used to extract multi-granular semantic features from the multi-layer hidden states of MLLM and directly inject them into the bottom layer of the generative network. This achieves dual protection of pixel-level detail control and high-level semantic alignment, solving the problem of detail loss in multi-subject generation.

Comparison of OmniWeaving's similar products

Application scenarios of OmniWeaving

• Film and television advertising creative production: Generate concept preview videos directly from text to verify storyboards, or freely combine reference images of characters, scenes, and props to generate complete commercials.
• E-commerce product dynamic displayUpload a product background image and scene reference image to automatically generate a product usage scene video, and combine user photos and product videos to generate a personalized virtual try-on effect.
• Social media content creationThis feature transforms still photos into dynamic videos with natural facial expressions and movements, bringing old photos to life. It quickly generates looping animations or fun emoji packs based on frame interpolation.
• Game animation asset creationUpload character design sketches and action descriptions to directly generate character animation clips, accelerating the production of cutscenes. Provide keyframes so that AI can automatically complete intermediate animations for scene transitions.