MIMO - A controllable role AI video synthesis framework launched by Alibaba Research Institute.
MIMO is a novel AI framework for controllable character video synthesis developed by Alibaba Group's Intelligent Computing Research Institute. Based on spatial decomposition modeling technology, it converts 2D video into 3D spatial code, enabling precise control over characters, actions, and scenes...
What is MIMO?
MIMO is a novel AI framework for controllable character video synthesis developed by Alibaba Group's Intelligent Computing Research Institute. Based on spatial decomposition modeling technology, it converts 2D video into 3D spatial code, enabling precise control over characters, actions, and scenes. MIMO can process the synthesis of arbitrary characters, adapt to novel 3D actions, and interact with real-world scenes. The core of MIMO lies in decomposing the video into three parts: the main character, the underlying scene, and floating occlusions. These are then encoded into identity code, motion code, and scene code, respectively, as control signals for the synthesis process. This not only improves the realism of the synthesized video but also enhances the user's control over the video content.
The main functions of MIMO
- Controllable character synthesisUsers can control the appearance of characters in videos by providing simple input.
- motion controlMIMO can synthesize character movements based on a provided sequence of poses, including complex 3D movements.
- Scene Interaction: To naturally integrate characters into real-world scenes, including handling occlusion and object interaction.
- Spatial decomposition modelingThe video is broken down into different spatial components, including the main characters, the underlying scene, and floating occlusions.
- 3D perception synthesis: Improve the realism and depth perception of synthesized videos based on 3D representation.
- Flexible user controlUsers can freely combine different potential codes to control various aspects of video synthesis.
- Scalability of any roleMIMO can synthesize any character, not just those in the training dataset.
MIMO technical principles
- 3D Depth Estimation: Use a monocular depth estimator to convert 2D video frames into representations in 3D space.
- Spatial decompositionBased on the estimated 3D depth information, the video is decomposed into three main spatial components:
- Main charactersThe core object in the video.
- underlying sceneThe background of the video.
- Floating occlusionForeground objects in the video, such as objects that might obscure the main subject.
- Component codingThe above spatial components are further encoded into different codes:
- Identity Code: Indicates the character's visual identity.
- motion code: Indicates the character's actions and posture.
- Scene code: Indicates background scene information of the video.
- Structured motion representationUsing deformable human models (such as SMPL) to represent and encode human movements allows for more precise capture of complex 3D motion.
- Standardized identity representationTransforming a character into a standardized pose (such as an A-pose) decouples identity from action, making identity representation independent of action.
- Scene and Occlusion CodingThe scene and occlusion components are encoded using a shared variational autoencoder (VAE) and then reorganized into complete scene code.
MIMO project address
- Project official website:menyifang.github.io/projects/MIMO/index.html
- GitHub repository:https://github.com/menyifang/MIMO
- arXiv technical paper:https://arxiv.org/pdf/2409.16160
Application scenarios of MIMO
- Film and video productionIt can quickly generate animated character performances, reducing the cost and time of special effects and animation production.
- Game developmentCreate realistic movements and interactions for game characters to enhance the gaming experience.
- Virtual Reality (VR)Create dynamic characters that interact with users in a virtual environment to enhance immersion.
- Augmented Reality (AR)Overlaying virtual characters and objects onto the real world for educational, entertainment, or navigation purposes.
- Social media and entertainmentIt allows users to create and share videos of virtual characters with personalized actions and scenes.
- Advertising and MarketingCreate engaging dynamic ads where characters are customized based on the target audience.
- Education and trainingCreate simulated scenarios and roles for educational purposes, such as historical reenactment or language learning.