HumanVid - A high-quality dataset designed specifically for generating animations of human images.

What is HumanVid?

HumanVid is a high-quality dataset jointly developed by the Chinese University of Hong Kong and the Shanghai Artificial Intelligence Laboratory, specifically designed for training human image animation. It combines real-world video and synthetic data, filtering high-quality videos through carefully designed rules and annotating them using 2D pose estimation and SLAM techniques. HumanVid aims to improve the controllability and stability of video generation, and its effectiveness has been validated using the baseline model CamAnimate, achieving state-of-the-art levels in controlling human poses and camera movement. The project plans to release the code and dataset publicly by the end of September 2024.

HumanVid's main functions

• High-quality data integrationIt combines real-world and synthetic data to ensure the richness and diversity of the dataset.
• Copyright FreedomAll video and 3D avatar assets are copyright-free, facilitating research and use.
• Rule FilteringThe rule-based filtering mechanism ensures that the videos in the dataset are of high quality.
• Human and camera motion annotationsAccurate annotation of human and camera motion in videos is achieved using 2D pose estimation and SLAM techniques.

HumanVid's technical principles

• Dataset ConstructionHumanVid constructs its dataset by collecting a vast amount of copyrighted, free real-world video from the internet and combining it with synthetic data. The videos are filtered using carefully designed rules to ensure the high quality of the dataset.
• annotation techniquesThe video uses a 2D pose estimator to annotate human motion, while employing a SLAM (Simultaneous Localization and Mapping)-based method to annotate camera motion.
• Synthetic data generationTo increase the diversity of the dataset, HumanVid collected copyrighted, free 3D avatar assets and introduced a rule-based camera trajectory generation method to simulate different camera movements.
• Model trainingHumanVid has built a baseline model, CamAnimate, which takes human and camera motion as conditions. By training on the HumanVid dataset, it is able to generate videos with control over human poses and camera motion.

HumanVid project address

Application scenarios of HumanVid

• Video productionIt provides high-quality animation generation for film, television and other video content production, enabling directors and producers to create more vivid and realistic scenes by controlling character poses and camera movements.
• Game developmentIn video games, HumanVid can generate realistic NPC (non-player character) animations, enhancing the game's immersion and interactivity.
• VR and ARIn VR and AR applications, HumanVid can generate virtual characters that interact with users, providing a more natural and fluid experience.
• Education and trainingHumanVid can create instructional videos that simulate character actions and scenarios, helping students better understand and learn complex concepts.