MimicMotion - An AI video generation framework jointly developed by Tencent and Shanghai Jiao Tong University

What is MimicMotion?

MimicMotion, developed by Tencent researchers, is a high-quality human motion video generation framework. It utilizes confidence-aware pose guidance technology to ensure high-quality video frames and smooth temporal transitions. Furthermore, MimicMotion significantly reduces image distortion and enhances the detail of human hand movements through region loss amplification and hand region enhancement. The framework can also generate long videos using a progressive latent fusion strategy, producing high-quality, long-duration human motion videos that conform to specific motion guidelines, while significantly improving the controllability and detail richness of video generation.

Features of MimicMotion

• Generate diverse videosMimicMotion can generate video content of various movements based on posture guidance provided by the user. Whether it's dance, sports, or daily activities, as long as the corresponding posture sequence is provided, MimicMotion can create corresponding dynamic videos.
• Control video lengthUsers can specify the duration of the video according to their needs. MimicMotion can generate short clips ranging from a few seconds to full videos of several minutes or even longer, providing flexibility to adapt to different application scenarios.
• Attitude guidance and controlThe framework uses a reference pose as a condition to ensure that the generated video content maintains consistency with the specified pose in terms of motion. MimicMotion allows users to precisely control the motion of the video, enabling highly customized video generation.
• Quality assurance in detailMimicMotion pays special attention to details in videos, especially in areas prone to distortion such as hands. Through a confidence-aware strategy, the system can provide clearer visuals in these areas.
• Time smoothnessTo provide a more natural viewing experience, MimicMotion ensures smooth transitions between video frames, avoiding stuttering or discontinuity, making the video look smoother and more natural.
• Reduce image distortionThrough confidence-aware pose guidance, MimicMotion can identify and reduce image distortion caused by inaccurate pose estimation, especially in the hand area of a person.
• Long video generationMimicMotion employs a progressive latent fusion technique, allowing the system to maintain high temporal coherence when generating long videos. This technique effectively avoids flickering and inconsistencies by fusing latent features between video segments.
• Resource consumption controlWhen generating videos, the MimicMotion optimization algorithm ensures that resource consumption remains within a reasonable range. Even when generating longer videos, it effectively manages computational resources, avoiding excessive costs.

MimicMotion's official website

• Official project homepage:https://tencent.github.io/MimicMotion/
• GitHub source code repository:https://github.com/Tencent/MimicMotion
• arXiv technical paper:https://arxiv.org/abs/2406.19680

The technical principle of MimicMotion

• Posture-guided video generationMimicMotion uses user-provided pose sequences as input to guide the generation of video content, allowing the model to synthesize corresponding actions based on changes in pose.
• Confidence-aware posture guidanceThe framework introduces the concept of confidence, weighting each keypoint in the pose sequence by analyzing the confidence score provided by the pose estimation model. This allows the model to place greater trust in keypoints with high confidence, reducing the impact of inaccurate pose estimation on the generated results.
• Regional loss amplificationSpecifically targeting areas prone to distortion, such as hands, MimicMotion enhances the model's training of these areas by increasing their weights in the loss function, thereby improving the quality of hand details in the generated video.
• Potential diffusion modelMimicMotion uses a latent diffusion model to improve generation efficiency and quality. The model reduces computational costs by performing the diffusion process in a low-dimensional latent space instead of operating directly in the pixel space.
• Progressive potential fusionTo generate long videos, MimicMotion employs a progressive latent fusion strategy. By gradually fusing the latent features of overlapping frames between video segments, a smooth transition between segments is achieved, avoiding flickering and discontinuity that may occur when generating long videos.
• Utilization of pre-trained modelsMimicMotion is based on a pre-trained video generation model (such as Stable Video Diffusion, SVD), which reduces the amount of data and computational resources required to train a model from scratch.
• Structure of U-Net and PoseNetMimicMotion's model architecture consists of a U-Net for spatial-temporal interaction and a PoseNet for extracting pose sequence features. These network structures work together to achieve high-quality video generation.
• Cross-frame smoothnessMimicMotion takes into account the temporal relationship between frames during the generation process, ensuring the continuity and smoothness between video frames.