Perception-as-Control - An image and animation framework developed by Alibaba Tongyi Labs
Perception-as-Control is an image and animation framework developed by Alibaba Tongyi Labs, capable of achieving fine-grained motion control based on user intent. Perception-as-Control is based on building a 3D perceptual motion representation, and...
What is Perception-as-Control?
Perception-as-Control is an image animation framework developed by Alibaba Tongyi Labs, capable of achieving fine-grained motion control based on user intent. Based on building 3D perceptual motion representations, Perception-as-Control transforms camera and object motion into intuitive and consistent visual changes, using the perceptual results as motion control signals to support various motion-related video synthesis tasks. The Perception-as-Control framework is based on a diffusion model of the U-Net architecture, combining appearance information from reference images with motion information from motion control signals to generate controllable image animations. Perception-as-Control introduces a three-stage training strategy to balance camera and object motion control, promoting fine-grained motion control and demonstrating superior performance in various application scenarios.
The main functions of Perception-as-Control
- Fine-grained cooperative motion controlIt enables coordinated control of camera and object movement, allowing users to control individual elements in the animation in a fine-grained manner. This allows users to precisely adjust the movement of objects in the scene and changes in the camera's perspective.
- Multi-motion related video synthesis taskIt supports a variety of motion-related video synthesis tasks, including image-based motion generation, video-based motion cloning, motion transfer, and motion editing.
- Motion generation: Generates animation based on reference images and 2D trajectories input by the user.
- sports cloning: Imitate the movement of the camera and objects in the source video.
- motion transfer: Transfer local motion from the source video to objects at different locations and scales in the reference image.
- Sports EditorUsers provide a segmentation mask and edit the motion within the segmentation mask.
- 3D motion perception representationBased on the construction of 3D perception motion representation, complex 3D scenes are simplified into key object parts (represented by unit spheres) and world envelope, achieving intuitive and consistent visual changes.
The technical principle of Perception-as-Control
- 3D motion perception representationThe 3D scene is simplified into key object parts (represented by unit spheres) and world envelope, and local object motion and global camera motion are captured based on 3D point tracking and visual odometry technology.
- Network architectureBased on the denoising U-Net architecture, two lightweight encoders are used to encode camera and object control signals respectively, avoiding RGB level interference, and the fusion module merges the encoded signals.
- Reference image injectionReferenceNet injects the appearance information of the reference image into the generation process, ensuring that the generated video retains the specific appearance of the reference image.
- Three-stage training strategy:
- Phase 1: Train only the camera encoder to process video clips containing only camera motion.
- Phase TwoAdded object encoder and fusion module to process video clips containing camera and object motion.
- Phase ThreeFine-grained object motion control is achieved based on sparse unit spheres, adaptively determining the control range of each rendering point.
- diffusion modelUsing the basic principles of the image diffusion probability model, a denoising U-Net architecture is used to generate videos. The optimization objective is to minimize the difference between the predicted noise and the actual noise. The conditional inputs include a reference image and motion control signals.
The project address for Perception-as-Control
- Project official website:https://chen-yingjie.github.io/projects/Perception-as-Control/
- GitHub repository:https://github.com/chen-yingjie/Perception-as-Control
- arXiv technical paper:https://arxiv.org/pdf/2501.05020
Application scenarios of Perception-as-Control
- Movie and video special effectsGenerate animated scenes with specific motions, mimic complex movements in existing videos, or transfer the motion of one character to another.
- Game developmentGenerate natural and controllable animations for game characters and objects, enhancing the game's immersion and dynamic effects.
- Virtual Reality (VR) and Augmented Reality (AR)Generate real-time animated feedback in VR environments to enhance user interaction; integrate virtual object animations with real-world scenes in AR applications.
- Advertising and MarketingGenerate engaging dynamic ads and brand promotion animations to enhance brand image and memorability.
- Education and trainingGenerate scientific experiment animations and skills training simulation animations to help students and trainees better understand and master knowledge and skills.