AvatarGO - A 4D human-object interaction generation framework jointly developed by Nanyang Technological University and Shanghai AI Lab, among others.
AvatarGO is a novel framework jointly developed by Nanyang Technological University's S-Lab, Shanghai AI Lab, and the University of Hong Kong. It's used to directly generate animable 4D human-object interaction scenes from text input. This is achieved through zero-shot technology...
What is AvatarGO?
AvatarGO is a novel framework jointly developed by Nanyang Technological University's S-Lab, Shanghai AI Lab, and the University of Hong Kong. It's used to directly generate animable 4D human-object interaction scenes from text input. By employing a zero-shot method based on a pre-trained diffusion model, it addresses the limitations of traditional methods in generating everyday HOI scenes due to a lack of large-scale interactive data. AvatarGO's core features include: LLM-guided contact relocalization, which identifies contact points from text prompts based on Lang-SAM, ensuring accurate representation of the spatial relationship between the human and objects; and correspondence-aware motion optimization, using SMPL-X's linear mixture skinning function to construct motion fields, optimizing human and object animations and reducing penetration issues. The AvatarGO framework demonstrates superior generation and animation capabilities across various human-object combinations and diverse poses.
AvatarGO's main functions
- Generating 4D interactive scenes from textIt can directly generate 4D animations that include human and object interactions based on simple text descriptions.
- Precise representation of human-object contactAccurately identify the contact points between the human body and objects (such as hands, feet, etc.) to ensure that the spatial relationship between the human body and objects is reasonable in the generated 3D and 4D scenes.
- Solving the penetration problem in animationDuring the animation generation process, the phenomenon of human bodies penetrating objects is effectively avoided.
- Diverse 4D animation generationGenerates dynamic 4D animations, supporting various character actions and object interactions.
- Supports combinations of various characters and objectsIt handles combinations of various characters and objects, including virtual characters (such as anime characters and superheroes) and real people, as well as various everyday items (such as weapons, tools, musical instruments, etc.).
AvatarGO's technical principles
- LLM-guided contact relocationLang-SAM (Language Segment Anything Model) extracts contact points (such as "hand") from text descriptions. It generates segmentation masks of the human body's contact points by rendering a 3D human model into a 2D image and combining this with text prompts. The masks are then back-projected onto the 3D model to initialize the object's position, ensuring accurate contact points between the object and the human body.
- Fractional distillation sampling with spatial awarenessIntroducing SSDS enhances the attention weights of text tags related to human-object interactions (such as "holding"), helping the diffusion model understand the spatial relationship between the human body and objects.
- Correspondence-aware motion optimizationThe SMPL-X model acts as an intermediary, constructing motion fields for both the human body and objects. Based on a linear hybrid skinning (LBS) function, it optimizes the synchronization of object and human motion. A new training objective—correspondence-aware loss—is introduced to minimize the spatial deviation between the human body and objects, ensuring that they do not pass through each other during animation.
- 3D and 4D generation based on diffusion model:
- 3D generationHigh-quality 3D human and object models are generated using the DreamGaussian method. The generated results are optimized using fractional distillation sampling (SDS) based on the 3D Gaussian point cloud representation of the scene.
- 4D animation generationBased on 3D models, dynamic 4D animations are generated using HexPlane features and the SMPL-X model. Global parameters of objects (such as rotation and translation) and the motion sequence of the human body are optimized to generate coherent and realistic 4D animations.
AvatarGO project address
- Project official website:https://yukangcao.github.io/AvatarGO
- GitHub repository:https://github.com/yukangcao/AvatarGO
- arXiv technical paper:https://arxiv.org/pdf/2410.07164
Application scenarios of AvatarGO
- Virtual shopping guideProvide customers with product information and shopping advice in the store.
- Exhibition Hall GuideIntroducing exhibits and product information in museums or exhibition halls.
- Digital Lobby ManagerProvide consultation and guidance services at banks or branch offices.
- In-vehicle virtual assistantIn cars, it acts as a smart assistant, providing companionship and interactive experiences.
- VR/AR content creationGenerate 4D animations for character and interaction design in virtual reality and augmented reality.