Motion Prompting - A motion trajectory control video generation model developed by Google in collaboration with Michigan and Brown University

What is Motion Prompting?

Motion Prompting is a video generation technology jointly developed by Google DeepMind, the University of Michigan, and Brown University. It controls and guides the generation of video content based on motion trajectories. Motion Prompting uses point trajectories as a flexible motion representation, capable of encoding motions of arbitrary complexity, ranging from single points to the entire scene. Users can design "motion prompts," similar to text prompts, to stimulate different capabilities of the video model, including object control, camera control, and simulation of physical phenomena. Motion Prompting improves the flexibility and accuracy of video generation, providing new possibilities for future interactive video generation and world model queries.

Main functions of Motion Prompting

• Object controlBased on motion cues, it allows for fine-grained control of the movement of specific objects in a video, such as rotation or movement.
• Camera controlIt enables motion control of the camera in video, including translation, rotation, and zoom.
• Physical phenomenon simulationShowcase physical phenomena in the video, such as fluid dynamics (water, smoke) and rigid body dynamics.
• Simultaneous object and camera controlCombines object control and camera control motion cues to enable complex scene interactions.
• Drag-and-drop image editingIt supports users to edit images based on drag-and-drop operations, enabling dynamic image editing.
• motion transferThis involves transferring motion from one video to the first frame of another video, thus reusing the motion.
• Motion amplification: Zoom in on subtle movements in the video to make them more noticeable.

The technical principle of Motion Prompting

• Point locus representationIt uses point trajectories as a method of representing motion to capture the motion of any number of points in a video, including object-specific motion or global scene motion.
• Conditional video generation modelBased on the pre-trained video diffusion model, a control network (ControlNet) is trained to accept motion cues as conditional input.
• Motion cues buildingIt converts user input (such as mouse dragging) into point trajectories, or generates detailed motion trajectories from advanced user requests based on computer vision technology.
• Trajectory codingEncode point trajectories as space-time volumes, with each trajectory having a unique embedding vector placed at each visited location.
• Model trainingThe model is trained using trajectory data extracted from videos, and videos are generated based on trajectory prompts.
• Trajectory sparsity adjustmentUsers can adjust the sparsity of the trajectory, balancing the fineness of control and the freedom of the video model.
• MultitaskingBased on combining different motion cues, multiple complex video generation tasks can be achieved in a single model.

The project address for Motion Prompting

• Project official website:motion-prompting.github.io
• arXiv technical paper:https://arxiv.org/pdf/2412.02700

Application scenarios of Motion Prompting

• Film and video productionDirectors and video producers can create complex dynamic scenes, such as special effects shots or action sequences, without the need for complex manual animation.
• Game developmentGame developers generate dynamic background videos for games or use them for dynamic interactions between game characters and the environment.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, create more natural and realistic virtual environments and interactive effects.
• Interactive Media ArtArtists create new interactive artworks, allowing viewers to interact with the artworks based on their body movements.
• Education and trainingBased on simulating real-world physical phenomena and dynamic scenarios, it can be used in the field of education, such as physics, engineering and medical training.