DisPose - A technology developed by Peking University and other universities to enhance the quality of character image control animation.

What is DisPose?

DisPose, a joint research team from Peking University, University of Science and Technology of China, Tsinghua University, and Hong Kong University of Science and Technology, is a control technology for improving the quality of human image animation. It is based on extracting effective control signals from skeletal pose and reference images, requiring no additional dense input. DisPose decomposes pose control into motion field guidance and keypoint mapping, generating dense motion fields to provide region-level guidance while maintaining generalization ability across different body types. DisPose includes a plug-and-play hybrid ControlNet that improves the quality and consistency of videos generated by existing models.

DisPose's main functions

• Sports field guidanceIt generates dense motion fields from skeletal poses, providing regional-level dense guidance and enhancing motion consistency in video generation.
• Key point correspondenceExtract the diffusion features corresponding to the pose key points in the reference image, transfer the diffusion features to the target pose, and maintain the consistency of identity information.
• Plug and play moduleAs a plugin module, it supports seamless integration into existing character image animation models without requiring modification of existing model parameters.
• Quality and consistency improvementHybrid ControlNet improves the quality and appearance consistency of generated videos.
• No additional intensive input requiredIt works without relying on additional dense inputs (such as depth maps) and reduces sensitivity to differences in body shape between the reference character and the driving video.

The technical principle of DisPose

• Sports field estimation:
  • Sparse sports fieldsDWpose estimates the skeletal pose, tracks motion displacement based on keypoints, and represents it as a trajectory graph.
  • Dense sports fieldsConditional motion propagation (CMP) predicts dense motion fields based on sparse motion fields and reference images, providing more detailed motion signals.
• Key point feature extraction: Extract DIFT features from the reference image using a pre-trained image diffusion model, and map these features to key points to form a key point feature map.
• Hybrid ControlNetA hybrid ControlNet was designed that updates during training without freezing other parts of the existing model, making it easy to seamlessly integrate motion field guidance and keypoint mapping into existing animation models.
• Feature fusion:
  • The feature fusion layer combines sparse and dense motion features to generate the final motion field guidance signal.
  • The multi-scale point encoder combines keypoint features with intermediate features from the U-Net encoder to enhance the semantic correspondence of features.
• Control signal integrationThe motion field guidance and key point correspondence are injected as additional control signals into the underlying video diffusion model to generate accurate character image animation.

DisPose project address

• Project official website:lihxxx.github.io/DisPose
• GitHub repository:https://github.com/lihxxx/DisPose
• arXiv technical paper:https://arxiv.org/pdf/2412.09349

Application scenarios of DisPose

• Artistic CreationArtists create dynamic artworks with specific movements and expressions, such as motion paintings and digital sculptures.
• social mediaOn social media platforms, generate personalized animated avatars or animated emojis to increase the fun of interaction.
• Digital people and virtual idolsCreate and control the movements and expressions of virtual characters for use in live streaming, video conferencing, or performances as virtual idols.
• FilmmakingIn film post-production, generating or modifying character movements improves production efficiency.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, virtual characters that interact with users are generated, providing a more natural and realistic interactive experience.