MotionCtrl - A motion controller for video generation models developed by Tencent and others.

What is MotionCtrl?

MotionCtrl is a unified and flexible motion controller designed for video generation models, developed by researchers from Tencent, the University of Hong Kong, Shanghai AI Lab, Tsinghua University, Guangdong University of Technology, and other institutions. It can independently control camera motion and object motion perspectives within a video. The system consists of two main modules: a camera motion control module and an object motion control module. These modules work in conjunction with a latent video diffusion model to achieve precise control over the motion perspective in the generated video.

MotionCtrl official website entrance

• Official project homepage:https://wzhouxiff.github.io/projects/MotionCtrl/
• Arxiv research paper:https://arxiv.org/pdf/2312.03641.pdf
• GitHub repository:https://github.com/TencentARC/MotionCtrl
• Hugging Face (MotionCtrl + VideoCrafter):https://huggingface.co/spaces/TencentARC/MotionCtrl
• Hugging Face (MotionCtrl + SVD):https://huggingface.co/spaces/TencentARC/MotionCtrl_SVD

Features of MotionCtrl

• Video generation by matching modelsMotionCtrl can be used with video generation models (such as VideoCrafter and Stable Video Diffusion) to generate videos based on text prompts. These videos not only contain still images, but also include coherent and smooth motion effects.
• Controlling camera movementUsers can specify the camera's movement in the video, such as panning (moving left and right, up and down), scaling (zooming in or out), and rotating (rotating clockwise or counterclockwise). MotionCtrl can then generate the corresponding video based on these instructions.
• Controlling the movement of objectsMotionCtrl can control the motion trajectory of objects in a video, such as the object's movement path, speed, and direction in the scene, so that the object's motion matches the text description or the trajectory specified by the user.
• Combined motion controlMotionCtrl can not only control the movement of the camera and the object independently, but also control both simultaneously to achieve complex motion effects, such as the camera translating or rotating while tracking the object.
• Adaptable to various camera poses and trajectoriesAfter training, MotionCtrl can adapt to various camera poses and object motion trajectories without needing to retrain the model for each new situation.

How MotionCtrl works

The MotionCtrl works primarily based on two core modules: the Camera Motion Control Module (CMCM) and the Object Motion Control Module (OMCM), and their collaboration with the Latent Video Diffusion Model (LVDM).

Here are the detailed steps on how MotionCtrl works:

1. Camera Motion Control (CMCM):
   • CMCM receives a series of camera poses (including rotation and translation matrices) that describe the camera's motion path in the video.
   • CMCM integrates camera pose information into the video generation process by interacting with temporal transformers in LVDM.
   • In the second self-attention module of the time transformer, the camera pose sequence is expanded and combined with time step information, and then processed through a fully connected layer for use in the subsequent time transformer.
   • In this way, the generated video will follow the specified camera movements, such as translation, scaling, and rotation, thereby enabling control over the global scene motion.
2. Object Motion Control (OMCM):
   • OMCM is responsible for processing the motion trajectories of objects in a video, which are typically represented as the spatial position of the object in each frame.
   • OMCM uses convolutional layers and downsampling operations to extract multi-scale features from the trajectory and spatially fuses these features into the convolutional layers of LVDM.
   • In this way, OMCM can indicate the position of objects in each frame of the video, enabling control over the local motion of objects.
3. Training strategy:
   • The training of MotionCtrl is divided into two stages. First, CMCM is trained using the Realestate10K dataset, which contains video clips and camera pose annotations but lacks object motion trajectories.
   • OMCM was then trained using the WebVid dataset and object motion trajectories synthesized using the ParticleSfM algorithm. This dataset contains video clips, titles, and object motion trajectories.
   • During training, CMCM and OMCM are added to the pre-trained LVDM model as adapter-like modules, allowing them to train independently without affecting other parts of the LVDM.
4. Video generation:
   • During the video generation stage, MotionCtrl generates a video that matches the text description based on the text prompts, combined with the camera pose and the object's motion trajectory.
   • LVDM first generates a latent representation of the video from the noise, and then CMCM and OMCM adjust the global and local motion of the video according to the given motion conditions, finally generating a video with the specified motion effects.