Champ - A 3D-based model for converting images of people into video animations.

What is Champ?

Champ is a 3D-based model proposed by researchers from Alibaba, Fudan University, and Nanjing University to convert human images into video animations. This method combines 3D parametric models (especially the SMPL model) and latent diffusion models to accurately capture and reproduce the 3D shape and dynamics of the human body while maintaining the temporal consistency and visual realism of the animation, in order to generate high-quality human animated videos.

Champ's official website

• Official project homepage:https://fudan-generative-vision.github.io/champ/#/
• GitHub source code repository:https://github.com/fudan-generative-vision/champ
• arXiv research paper:https://arxiv.org/abs/2403.14781

Champ's features

• Converting images of people into videos and animationsChamp can convert static images of people into dynamic video animations, creating realistic and controllable dynamic visual content by accurately capturing and reproducing the shape and movement of the human body.
• 3D shape and pose representationChamp can accurately represent and control the shape and posture of the human body, and can extract human geometry and motion features from source videos more accurately.
• Cross-identity animation generationChamp can apply motion sequences from one video to a reference image of a different identity, enabling cross-identity animation generation.
• High-quality video generationChamp maintains consistency between characters and backgrounds when generating videos, while ensuring smooth transitions between frames through a time alignment module, resulting in high-quality video output.
• Combined with T2I text-based graph modelBy combining the T2I text-to-image model, which generates images based on text descriptions, users can specify the appearance and actions of characters in animations through text descriptions, and then Champ generates animated videos based on these descriptions.

How Champ works

1. 3D Parametric Human Model (SMPL):
   • The SMPL model is used to represent the shape and posture of the human body. The SMPL model is a parametric 3D human body model that can capture changes in the shape and posture of the human body.
   • By fitting the SMPL model onto a reference image, the shape and pose parameters of the human body can be obtained.
2. Extracting motion from source video:
   • Motion sequences of the human body are extracted from source videos using existing frameworks such as 4D-Humans. These motion sequences include continuous SMPL model parameters to describe the movements of the people in the video.
3. Generate depth, normal, and semantic graphs.:
   • The SMPL model is rendered into a depth map, normal map, and semantic map, which contain detailed information about the 3D structure, surface orientation, and human body parts.
4. Motion alignment and guidance:
   • The extracted SMPL model parameters are used to align the shape and pose of the characters in the reference image, ensuring that the characters in the animation move in accordance with the characters in the source video.
   • Introduce skeleton-based motion guidance to enhance the representation of complex movements such as facial expressions and finger movements.
5. Multi-level motion fusion:
   • By using a self-attention mechanism, feature maps containing depth, normal, semantic, and skeleton information are fused to generate a comprehensive motion guidance signal.
6. Potential diffusion model:
   • Using a latent diffusion model (such as the Latent Diffusion Model) as a generation framework, the motion guidance signal and the encoded features of the reference image are combined to generate animation frames.
   • A denoising process is applied in the latent space to gradually recover clear animation frames from a noisy representation.
7. Training and reasoning:
   • During the training phase, the model learns how to generate coherent animation sequences based on given reference images and motion instructions.
   • During the inference phase, the model generates animations based on new reference images and motion sequences, demonstrating its generalization ability.
8. Video generation:
   • The generated frame sequence is combined into a video, while ensuring that the people in the video are visually consistent with the reference images and that their movements are smooth and natural.