UnityVideo - A video generation framework open-sourced by Kuaishou Keling in collaboration with HKUST

What is UnityVideo?

UnityVideo is a novel multimodal, multi-task video generation framework developed by the Hong Kong University of Science and Technology in collaboration with Kuaishou's Keling team, Tsinghua University, and others. It enhances the video generation model's understanding of the physical world by integrating multiple visual modalities (such as segmentation, skeletonization, depth, and optical flow) and training paradigms. The framework achieves bidirectional learning between RGB video and auxiliary modalities through dynamic noise injection and modality adaptive learning mechanisms, accelerating model convergence and significantly improving zero-shot generalization ability.

Main functions of UnityVideo

• Multimodal video generationThe framework can generate high-quality RGB videos from text descriptions and combine multiple auxiliary modalities (such as depth maps, optical flow, segmentation masks, human skeletons, DensePose, etc.) for joint generation to improve the physical realism and consistency of the videos.
• Controllable video generationIt supports controllable video generation based on multiple modal conditions (such as depth maps, optical flow, etc.), and generates video content that meets specific requirements based on given modal information.
• Modal estimationIt estimates other auxiliary modal information (such as depth map, optical flow, segmentation mask, etc.) from RGB video to achieve multimodal parsing of video content.
• Zero-shot generalizationIt has strong zero-shot generalization ability, and can generate high-quality videos on scenes and objects outside the training data, accurately estimating multiple modal information.
• Multi-task joint trainingIt supports multiple tasks such as video generation, controllable generation, and modality estimation within a single framework, and improves the overall performance and generalization ability of the model through joint training of multiple tasks.

The technical principles of UnityVideo

• Unified multimodal frameworkA unified framework based on a diffusion model (such as the DiT diffusion transformer) is used to integrate RGB video and multiple auxiliary modal information into a shared feature space. Through a dynamic noise injection strategy, different tasks (such as conditional generation, modality estimation, and joint generation) are randomly selected during training, enabling the model to learn the joint distribution of multiple tasks and modalities simultaneously.
• Modal adaptive learningTo address this, a modality-adaptive switcher is introduced, assigning an independent parameter table (such as AdaLN parameters) to each modality, enabling the model to dynamically adjust network parameters based on different modalities. An in-context learner is used to inject textual cues of modality types (such as "depth map" or "optical flow"), allowing the model to semantically distinguish between different modalities and enhancing its modality awareness.
• Dynamic noise schedulingDuring training, the noise injection strategy is dynamically adjusted, applying different noise to the RGB and auxiliary modalities based on the task type (conditional generation, modality estimation, joint generation) to promote cross-task collaborative learning. A probabilistic task selection mechanism balances the learning difficulty of different tasks, preventing the model from favoring one task during joint training.
• Course learning strategiesThe learning approach is to first train pixel-aligned modalities (such as depth and optical flow) on single-person data to establish the foundation of spatial correspondence, and then expand to multi-person data and more modalities to gradually improve the model's ability to understand complex scenes.
• Large-scale multimodal datasetsThe project constructs the OpenUni dataset, containing 1.3 million pairs of multimodal video samples, covering various modalities such as RGB, depth, optical flow, segmentation masks, and skeletons, providing rich data support for model training. It also provides the UniBench benchmark set to evaluate the model's performance on multimodal video tasks, ensuring the model's generalization ability and accuracy.

UnityVideo project address

• Project official website: https://jackailab.github.io/Projects/UnityVideo/
• GitHub repositoryhttps://github.com/dvlab-research/UnityVideo
• HuggingFace model libraryhttps://huggingface.co/JackAILab/UnityVideo
• arXiv technical paper: https://arxiv.org/pdf/2512.07831

Application scenarios of UnityVideo

• Film and television special effects productionThe framework can quickly generate realistic special effects videos, such as waterfalls, aurora borealis, and liquid flow, providing efficient preview and creative verification for film and television post-production.
• Virtual Reality (VR) and Augmented Reality (AR)The framework can generate highly realistic virtual scenes and dynamic backgrounds, enhancing the user's immersion and interactive experience.
• Education and TrainingBased on the teaching content, simulated videos of scientific phenomena, such as physics experiments and biological animations, are generated to help students better understand complex concepts.
• Game developmentThe framework can generate dynamic scenes, character actions, and cutscenes in games, improving the visual effects and development efficiency of games.
• Advertising and MarketingQuickly generate creative ad videos based on brand descriptions for use on social media, television commercials, and more, meeting diverse marketing needs.