MotionCanvas - A controllable image-to-video generation method developed by CUHK, Adobe, and other organizations.

What is MotionCanvas?

MotionCanvas, developed by the Chinese University of Hong Kong, Adobe Research, and Monash University, is an image-to-video (I2V) generation method that transforms still images into videos with rich dynamic effects. Based on a motion design module, MotionCanvas allows users to intuitively plan camera and object movements on images, enabling complex shot designs. With a motion signal conversion module, MotionCanvas accurately translates the user's motion intentions in the 3D scene space into control signals in the 2D screen space, driving a video diffusion model to generate high-quality videos. MotionCanvas supports joint control of camera and object movements, generating long videos with complex trajectories and achieving precise control of local object movements.

Main functions of MotionCanvas

• Joint control of camera and object movementUsers can intuitively design camera motion paths (such as translation, rotation, zoom, etc.) and global motion of objects (such as movement, scaling, etc.) and local motion of objects (such as arm swinging, etc.) on the input image.
• 3D sensing motion controlThe system is based on depth estimation and motion signal conversion modules to realize motion design in 3D scene space, convert it into motion signals in 2D screen space, and generate videos with 3D perception effects.
• Long video generationIt supports generating videos of any length and supports complex motion trajectories and continuous shot designs.
• Diverse application scenariosIt supports applications for simple video editing tasks, such as motion transfer and video generation based on initial frame editing.

The technical principle of MotionCanvas

• Motion Design ModuleThe user-input image serves as the "canvas," and camera and object motions are designed within the 3D scene space. Camera motion is achieved based on the camera pose of specified keyframes, while object motion is defined using scene-anchored bounding boxes and sparse point trajectories.
• Motion signal conversion moduleThis module converts the user's motion intentions designed in the 3D scene space into motion signals in the 2D screen space. Based on depth estimation and camera parameter estimation, it converts the 3D camera path into 2D point trajectories and the scene-anchored bounding boxes and point trajectories into signals in the screen space.
• Motion Conditional Video Generation ModelThis method is based on a pre-trained DiT (Diffusion Transformer) model, fine-tuned to generate videos based on motion conditions in screen space. The model uses encoded point trajectories and bounding box sequences as conditional signals input into the video generation model to generate videos that match the user's intent.
• Autoregressive generationThe method uses an autoregressive approach, using the generated video segments as conditions for subsequent generation to gradually generate continuous video content, thus avoiding motion discontinuity issues that may result from directly generating long videos.

MotionCanvas project address

• Project official website:https://motion-canvas25.github.io/
• arXiv technical paper:https://arxiv.org/pdf/2502.04299

Application scenarios of MotionCanvas

• Film and Video ProductionQuickly design complex shots and transform still images into dynamic videos for use in advertising, promotion, and other applications to enhance visual appeal.
• Animation CreationGenerate animation previews and character movements, extract dynamic effects from static storyboards or images, and improve creative efficiency.
• VR/AR ExperienceIt generates dynamic scenes for virtual and augmented reality applications, enhancing user immersion and supporting real-time interaction.
• Game developmentTo create game cutscenes and dynamic scenes, enhancing the game's narrative and visual effects, and enriching the player experience.
• Education and TrainingTransforming static teaching images into dynamic videos for use in classroom lectures and professional simulation training enhances the interest and effectiveness of teaching.