MagicMan - Tencent, in collaboration with several universities, launched an AI project to generate 3D human models from 2D images.

What is MagicMan?

MagicMan is an AI project jointly launched by research teams from Tsinghua University Shenzhen International Graduate School, Tencent AI Lab, Hong Kong University of Science and Technology, Stanford University, and the Chinese University of Hong Kong. It focuses on generating high-quality 3D human models from single 2D images using deep learning technology. Combining a pre-trained 2D diffusion model and a parameterized SMPL-X model, it achieves accurate 3D perception and image generation through a hybrid multi-view attention mechanism and iterative refinement strategy. It has broad application potential in multiple fields such as games, movies, and virtual reality.

MagicMan's main functions

• Generate 3D models from a single imageGenerate high-quality 3D human models from a single 2D human image.
• Multi-view image synthesisGenerates images of characters from different perspectives, providing a comprehensive visual experience.
• Normal map generationSimultaneously, it generates normal maps corresponding to the RGB images, enhancing the texture and realism of the 3D model.
• 3D perception capabilityBy combining the SMPL-X model, MagicMan can understand and generate character models with accurate 3D structures.
• Hybrid multi-view attention mechanismImages generated from different angles maintain visual coherence and consistency.

MagicMan's technical principles

• Pre-trained 2D diffusion model:Pre-training on a large amount of image data allows for the learning of rich texture and appearance features.
• Parameterized SMPL-X model:SMPL-X is a parametric 3D human body model that can accurately describe the geometry and posture changes of the human body.
• Hybrid multi-view attention mechanism:By combining 1D and 3D attention mechanisms, effective information exchange between different perspectives can be achieved.Ensure that images generated from different angles remain visually consistent and coherent.
• Geometrically-aware bi-branch generation:at the same timeGenerate RGB and normal images, and improve the geometric consistency of the image using geometric cues.MagicMan can generate highly realistic 3D images in terms of both visual appearance and geometry.

MagicMan's project address

• Project official websitethuhcsi.github.io/MagicMan
• GitHub repository:https://github.com/thuhcsi/MagicMan
• arXiv technical paper:https://arxiv.org/pdf/2408.14211

Application scenarios of MagicMan

• Game developmentIn game design, MagicMan quickly generates realistic game characters and dynamic environments, enhancing the diversity and realism of character design.
• Film and Animation ProductionThe film industry uses MagicMan to generate 3D character models from existing 2D images or photos of real actors for motion capture or direct use in animation, saving time and costs associated with traditional modeling.
• Virtual Reality (VR) and Augmented Reality (AR)In VR and AR applications, MagicMan creates realistic virtual characters and environments, enhancing user immersion and interactive experience.
• Fashion and RetailThe fashion industry is using MagicMan technology to create virtual fitting rooms, where consumers can upload their own images and preview how different clothes look on them, providing a personalized shopping experience.
• Education and Training SimulationIn the field of education, MagicMan is used to generate various roles and scenarios for simulation training, such as medical simulations and historical reenactments, to improve learning outcomes and training quality.