Make-A-Character: Alibaba's open-source AI 3D digital human generation framework

What is Make-A-Character?

Make-A-Character (Mach for short) is an AI-powered 3D digital human generation framework developed by the Alibaba Group's Intelligent Computing Research Institute. It aims to rapidly create realistic 3D characters from text descriptions. The system is particularly well-suited to meet the needs of AI agents and the metaverse for personalized and expressive 3D characters. Mach's core functionality involves using large-scale language and visual models to understand the intent within text and generate intermediate images. These images are then transformed into complete 3D character models through a series of modules targeting human visual perception and 3D generation.

Make-A-Character official website entrance

• Official project homepage:https://human3daigc.github.io/MACH/
• Arxiv research paper:https://arxiv.org/abs/2312.15430
• GitHub repository:https://github.com/Human3DAIGC/Make-A-Character(Code to be open source)
• ModelScope Demo:https://www.modelscope.cn/studios/XR-3D/InstructDynamicAvatar/summary

Features of Make-A-Character

• Text-to-3D Character GenerationUsers can specify the character's appearance features, such as facial features, hairstyle, and clothing, by entering text descriptions. The Mach system will then generate a corresponding 3D character model based on these descriptions.
• Flexible and customizableThe system allows users to customize their characters in detail, including facial features, eye shape, iris color, hairstyle and color, eyebrows, mouth and nose, and even add wrinkles and freckles to create characters that meet the user's individual needs.
• Highly realistic renderingMach uses physically based rendering (PBR) technology, combined with real human scan data, to generate highly realistic characters. The characters' hair is constructed as individual strands of hair, rather than traditional meshes, to enhance realism.
• Complete character modelThe generated character model includes all the details such as eyes, tongue, teeth, full body and clothing, ensuring that the character can be used immediately in various application scenarios.
• Animation supportThe character is equipped with an advanced skeletal rigid body that supports standard animation, enabling the character to perform various dynamic expressions, such as changes in facial expressions.
• Industry compatibilityMach-generated character models use a clear 3D representation that can be seamlessly integrated into standard CG workflows in the game and film industries, facilitating subsequent animation production and rendering.

How Make-A-Character works

1. Text parsing and visual cue generation:
   • The user enters text prompts to describe the character's characteristics.
   • Large Language Models (LLMs) are used to understand semantic information in text and extract key facial features and attributes.
   • These features are mapped to visual cues, such as pose and edge maps, to guide subsequent image generation.
2. Reference portrait image generation:
   • By combining the Stable Diffusion model and ControlNet, reference portrait images are generated based on extracted visual cues. ControlNet ensures that the generated images have a frontal pose and neutral expression, facilitating subsequent 3D modeling.
   • Openpose and Canny edge detection technologies are used to ensure a reasonable distribution of facial features.
3. Dense facial coordinate detection:
   • The geometry of the face and head is reconstructed using dense facial coordinates (431 coordinate points), which are more detailed than the traditional 68 or 98 coordinate points and cover the entire head.
   • These coordinate points are generated using synthetic images as training data through a multi-view capture and processing process.
4. Geometric generation:
   • The head geometry is reconstructed based on a reference portrait image and dense facial coordinates. Mesh optimization is achieved by mapping a 3D mesh to a 2D plane.
   • Coordinate projection loss and local smoothing constraints are used to ensure the accuracy of the geometry.
5. Texture generation:
   • Differentiable rendering techniques are used to extract textures from reference images, and high-resolution textures are generated incrementally using a multi-resolution strategy.
   • A neural de-lighting method is introduced to remove unnecessary lighting effects from the texture image, resulting in a diffuse map suitable for rendering.
6. Texture correction and completion:
   • The generated diffuse map is corrected to resolve imperfections in areas such as the eyes, mouth, and nostrils.
   • The mask for the erroneous area is extracted using a facial analysis algorithm and then merged with the template diffuse map using Poisson blending technology to improve the visual effect.
7. Hair growth:
   • Various hairstyles are synthesized from 2D images, and then 3D hair strands are reconstructed based on these images.
   • Using advanced hair generation technologies, such as NeuralHDHair, we train models and generate high-quality 3D hair strands.
8. Asset matching:
   • The generated head is matched with pre-made assets such as hair, body, clothing, and accessories.
   • The CLIP text encoder is used to calculate the similarity between input prompts and asset tags, and the best-matching asset is selected.
9. Character Assembly:
   • Assemble all generated and matched parts into a complete 3D character model.
   • The character model supports animation and can be dynamically represented through rigid bodies.