Motion Dreamer - A motion-based video generation framework developed by the Hong Kong University of Science and Technology

What is Motion Dreamer?

Motion Dreamer, a video generation framework proposed by researchers at the Hong Kong University of Science and Technology (Guangzhou), generates videos with believable motion. Based on a two-stage generation approach, it first generates an intermediate motion representation based on the input image and motion conditions, and then uses this representation to generate a high-detail video. It introduces a novel motion modality, instance flow, enabling motion control from sparse to dense. Users provide sparse motion cues, and the model generates temporally coherent videos. During training, a random mask instance flow strategy is employed to enhance the model's inference and generalization abilities. Experiments on the Physion dataset and an autonomous driving dataset demonstrate that Motion Dreamer significantly outperforms other models in terms of motion coherence and physical plausibility, while also generating high-quality videos.

Main functions of Motion Dreamer

• Generate physically coherent videoAs a two-stage video generation framework, Motion Dreamer can generate videos that conform to physical laws. In the first stage, the model generates intermediate motion representations, such as segmentation maps or depth maps, based on the input image and motion conditions, focusing on the motion itself; in the second stage, the intermediate motion representations are used to generate high-detail videos.
• Achieving sparse to dense motion controlThe introduction of instance flow, a new motion modality, allows users to provide sparse motion cues, such as directional arrows representing average optical flow. The model can then generate pixel-aligned dense motion representations, enabling temporally coherent video generation.
• Enhance model reasoning abilityThe training strategy employs a random masked instance stream, where a portion of the instance stream is randomly masked during training. This requires the model to reconstruct a complete dense motion representation, prompting the model to infer missing motion cues and improving the model's generalization ability and inference-based motion generation performance.

The technical principle of Motion Dreamer

• Two-stage generative framework
  • Phase 1: Motion ReasoningBased on the input image and motion conditions, intermediate motion representations, such as segmentation maps or depth maps, are generated, focusing on the motion itself. This stage employs a diffusion-based video generation model, emphasizing low-frequency motion representations to improve temporal coherence. Specifically, the model predicts intermediate motion representations such as optical flow, instance segmentation maps, and depth maps, which together constitute a comprehensive description of the scene's dynamics.
  • Phase Two: High-Fidelity Video SynthesisThe model generates high-detail videos by using intermediate motion representations generated in the first stage as conditions. By decoupling motion inference from video synthesis, the model can more accurately generate physically consistent motion while maintaining high-quality video detail.
• Instance Stream:Instance streams are a novel sparse-to-dense motion modality used to connect human input with dense motion representations. To effectively integrate instance streams into the model, multi-scale versions of the instance streams are prepared to match feature maps of different scales in the network. Then, the Softmax Splatting function is used to deform the feature maps according to the flow field, distributing the features to new locations, seamlessly integrating motion information while maintaining differentiability to support end-to-end training.
• Random Mask Instance Stream Training StrategyDuring training, a randomized mask is applied to a portion of the instance stream, requiring the model to reconstruct a complete dense motion representation. This prompts the model to infer missing motion cues, enhancing its generalization ability and the effectiveness of inference-based motion generation. By training the model to handle incomplete motion information, it can better understand and predict interactions between objects and reasonable motion trajectories, generating plausible motion even with sparse input.

Motion Dreamer's project address

• Project official website:https://envision-research.github.io/MotionDreamer
• Github repository:https://github.com/EnVision-Research/MotionDreamer
• arXiv technical paper:https://arxiv.org/pdf/2412.00547

Application scenarios of Motion Dreamer

• Video content creationIn the production of video content such as movies, TV series, and advertisements, Motion Dreamer can generate high-quality and physically coherent video clips, providing creators with more creative space and material choices.
• Animation ProductionFor the production of animated films and game animations, Motion Dreamer can generate realistic character animations, reducing the time and cost of manual animation production and improving the efficiency of animation production.
• Immersive experienceIn VR and AR applications, Motion Dreamer can generate realistic virtual scenes and dynamic effects, providing users with a more immersive experience.
• Interactive applicationsBy providing sparse motion cues through user input, Motion Dreamer can generate dynamic content that interacts with the user, resulting in a more natural and fluid interactive experience.
• Driving scenario simulationMotion Dreamer can be used in the field of autonomous driving to generate various complex driving scenarios, helping to test and optimize autonomous driving algorithms.
• Traffic flow analysisThe large amount of driving scenario data generated by Motion Dreamer can be used for traffic flow analysis and prediction, providing a reference for urban traffic planning and management.