GAS - A single-image 3D human body frame generation product launched by Carnegie Mellon University in collaboration with Shanghai AI Lab and others.

What is GAS?

GAS (Generative Avatar Synthesis from a Single Image) is a framework proposed by researchers from Carnegie Mellon University, Shanghai Artificial Intelligence Laboratory, and Stanford University for generating high-quality, viewpoint-consistent, and temporally coherent virtual avatars from a single image. The core of GAS lies in combining the advantages of regressive 3D human reconstruction models and diffusion models. Based on the 3D human reconstruction model, intermediate viewpoints or poses are generated from a single image and used as conditional input to the video diffusion model, achieving high-quality viewpoint consistency and temporal coherence. The framework introduces a "mode switcher" to distinguish between viewpoint synthesis and pose synthesis tasks, further improving the generation results.

The main functions of GAS

• Consistent multi-view synthesisGenerate high-quality multi-view renderings from a single image to ensure consistency in appearance and structure across different viewpoints.
• Continuous dynamic pose animation: Generate smooth and realistic non-rigid deformation animations from a given sequence of poses, achieving continuity of dynamic poses.
• Unified framework and generalization abilityThe goal is to unify the tasks of viewpoint synthesis and pose synthesis, improve the model's generalization ability to real-world scenes by sharing model parameters and training with large-scale real data (such as online videos).
• Dense appearance cuesThe dense information generated based on the 3D reconstruction model is used as a conditional input to ensure high fidelity in appearance and structure of the generated results.

GAS Technical Principles

• 3D Human Reconstruction and Dense Conditional SignalsGAS first generates intermediate viewpoints or poses from the input image based on a regressive 3D human reconstruction model (such as single-view general human NeRF). By mapping the input image to a canonical space and repositioning it, dense appearance cues are generated. This dense information serves as conditional input, providing rich detail and structural information for the subsequent diffusion model, ensuring high quality and consistency of the generated results.
• Video diffusion model and unified frameworkThe generated intermediate viewpoints or poses are used as conditional inputs to a video diffusion model, which generates high-quality viewpoint consistency and temporal coherence animations. GAS proposes a unified framework that merges viewpoint synthesis and pose synthesis tasks, shares model parameters, and achieves natural generalization from pose synthesis to viewpoint synthesis.
• mode switcherTo differentiate between viewpoint synthesis and pose synthesis tasks, GAS introduces a mode switcher module. This module enables the network to focus on consistency when generating viewpoints and on realistic deformation when generating poses.
• Generalization ability of real-world dataGAS significantly improves its generalization ability to real-world scenes by incorporating large-scale real-world video (such as online video) for training. The diversity of data sources enables the model to adapt to various lighting, clothing, and motion conditions.
• Training and ReasoningThe training of GAS is divided into two phases: first, a 3D human reconstruction model is trained, and then the model is frozen and a video diffusion model is trained. During inference, different classifier free-guided (CFG) strategies are used depending on the nature of the task (viewpoint synthesis or pose synthesis).

GAS project address

• Project official website:https://humansensinglab.github.io/GAS/
• arXiv technical paper:https://arxiv.org/pdf/2502.06957

Application scenarios of GAS

• Gaming and Virtual Reality (VR)GAS can generate high-quality virtual avatars from a single image, supporting coherent synthesis of multiple perspectives and dynamic poses.
• Film and television productionIn film and television special effects and animation production, GAS can quickly generate realistic virtual characters, reducing the time and cost of traditional modeling and animation production.
• Sports and FitnessBy generating dynamic virtual avatars from single images, GAS can be used to create personalized motion animations, helping athletes analyze movements or for use in fitness applications.
• Fashion and Clothing DesignGAS can generate virtual avatars with different poses and perspectives, helping designers quickly preview clothing effects and improve design efficiency.