HumanDiT - A posture-guided human video generation framework jointly developed by Zhejiang University and ByteDance.

What is HumanDiT?

HumanDiT is a pose-guided high-fidelity human video generation framework jointly proposed by Zhejiang University and ByteDance. Based on the Diffusion Transformer (DiT), it can be trained on large-scale datasets to generate long sequences of human motion videos with detailed body rendering. HumanDiT's core advantage lies in its pose guidance mechanism, which generates subsequent pose sequences through a Keypoint DiT, ensuring the coherence and naturalness of human movements in the video. It introduces a prefix latent reference strategy to preserve personalized features in long sequences. HumanDiT supports various video resolutions and variable sequence lengths, making it suitable for long-sequence video generation. A pose adapter is used to achieve pose transfer for a given sequence, further improving the accuracy of pose alignment.

Main functions of HumanDiT

• Posture-guided video generationHumanDiT generates subsequent pose sequences based on the Keypoint Diffusion Transformer (Keypoint-DiT), ensuring the continuity and naturalness of human movements in the video.
• Long sequence video generationIt supports multiple video resolutions and variable sequence lengths, making it suitable for generating high-quality videos with long sequences.
• Personalized features maintainedPreserve personalized features in long sequences through a prefix latent reference strategy.
• Flexible input and outputIt can generate videos from still images or existing videos, making it suitable for a variety of application scenarios.
• Posture adaptation and refinementThe pose transfer of a given sequence is achieved based on a pose adapter, and the alignment of facial and hand features with the reference image is enhanced by a pose refinement module.

HumanDiT's technical principles

• Attitude guidance and diffusion converterHumanDiT generates video through pose guidance, using a Keypoint DiT converter to generate subsequent pose sequences during the inference phase. This ensures the continuity and naturalness of human movements in the video, while the DiT converter can handle videos of different resolutions and sequence lengths.
• Prefix Potential Reference StrategyTo preserve personalized features in the generation of long video sequences, HumanDiT introduces a prefix latent reference strategy. This strategy maintains visual consistency by using the first frame of the video as a noise-free prefix latent vector for the model to reference during generation.
• Attitude adapter and attitude refinement moduleHumanDiT uses a pose adapter to achieve pose transfer for a given sequence, and further optimizes the generated pose sequence through a pose refinement module. This helps improve the generation quality of details such as faces and hands, and ensures alignment with the reference image.
• Training on large datasetsHumanDiT is trained on a large dataset containing 14,000 hours of high-quality video, collected through a structured data processing pipeline, covering a variety of human action scenarios. This enables the model to learn rich motion priors and exhibits strong generalization ability during inference.

HumanDiT project address

• Project official website:https://agnjason.github.io/HumanDiT-page/
• arXiv technical paper:https://arxiv.org/pdf/2502.04847

Application scenarios of HumanDiT

• Virtual HumanHumanDiT can be used to generate dynamic videos of virtual humans, resulting in more natural and fluid movements. In virtual customer service scenarios, virtual humans can generate corresponding actions and expressions in real time based on user questions, providing a more vivid and realistic interactive experience.
• Animated filmHumanDiT can generate high-quality, long sequences of human motion videos, which can be used to generate motion sequences for animated characters, helping animators quickly generate high-quality motion clips and improve production efficiency.
• Immersive experienceHumanDiT can be used to create virtual characters and actions in immersive experiences. In immersive VR experiences, HumanDiT can generate character movements and expressions that match the scene, allowing participants to experience a more realistic and vivid experience.
• Video generation and continuationIt can generate videos from a single image or continue generating subsequent content from an existing video.