HRAvatar - A monocular video-to-3D avatar generation technology jointly developed by Tsinghua University and IDEA
HRAvatar is a monocular video reconstruction technology developed by Tsinghua University in collaboration with the IDEA team. It supports the generation of high-quality, relit 3D avatars from ordinary monocular videos. HRAvatar uses learnable deformation bases and linear skinning techniques...
What is HRAvatar?
HRAvatar is a monocular video reconstruction technology developed by Tsinghua University in collaboration with the IDEA team. It supports the generation of high-quality, re-illuminated 3D avatars from ordinary monocular videos. HRAvatar uses learnable deformation bases and linear skinning techniques, based on a precise facial encoder to reduce tracking errors and improve reconstruction quality. HRAvatar decomposes the avatar appearance into attributes such as albedo, roughness, and Fresnel reflection, and combines them with a physically based rendering model to achieve realistic re-illuminated effects. HRAvatar outperforms existing methods in several metrics, supports real-time rendering (approximately 155 FPS), and provides a new technical solution for fields such as digital humans, virtual anchors, and AR/VR.
HRAvatar's main functions
- High-quality reconstructionGenerate detailed and expressive 3D avatars from ordinary monocular videos.
- Real-timeIt supports real-time rendering at speeds up to approximately 155 FPS, making it suitable for real-time applications.
- Re-lightingThe generated avatars are relit in real time to adapt to different lighting conditions.
- AnimationSupports animation of facial expressions and actions for avatars.
- Material EditingIt supports editing the material properties of avatars (such as albedo, roughness, and reflectivity) to achieve different visual effects.
- Cross-view renderingSupports rendering avatars from different perspectives, providing 3D consistency and multi-view support.
HRAvatar's technical principles
- Precise facial expression trackingAn end-to-end facial expression encoder is used to jointly optimize and extract more accurate facial expression parameters, reducing the impact of pre-tracking parameter errors on reconstruction. The facial expression encoder and 3D head reconstruction are jointly trained, with Gaussian reconstruction loss used for supervision to improve reconstruction accuracy.
- Geometric deformation modelBased on learnable linear blendshapes, a shape basis, expression basis, and pose basis are introduced for each Gaussian point, enabling flexible deformation from standard space to pose space. Linear blend skinning is then used to transform the Gaussian points to pose space, adapting to individual pose deformations.
- Appearance modelingThe avatar's appearance is decomposed into material properties such as albedo, roughness, and Fresnel reflectivity. A BRDF physically based rendering model is used for shading, combined with a simplified SplitSum approximation technique, to achieve high-quality, relit real-time rendering. A pseudo-albedo prior is introduced to better decouple material information and avoid misinterpreting local lighting into the albedo.
- Normal estimation and material priorUsing the shortest axis of each Gaussian point as its normal, the normal map of the rendered image is supervised based on the normal map obtained from the depth derivative to ensure geometric consistency. Pseudo-realistic albedo is extracted from the existing model, and the albedo is rendered under supervision, limiting roughness and base reflectivity within predefined ranges to obtain more realistic materials.
HRAvatar's project address
- Project official website:https://eastbeanzhang.github.io/HRAvatar/
- GitHub repository:https://github.com/Pixel-Talk/HRAvatar
- arXiv technical paper:https://arxiv.org/pdf/2503.08224
Application scenarios of HRAvatar
- Digital Humans and Virtual BroadcastersCreate realistic digital humans and virtual anchors, supporting real-time expressions and movements to enhance interactivity.
- AR/VRGenerate 3D avatars that can be relit in real time, enhancing the immersive experience of the virtual environment.
- Immersive MeetingsProvides high-quality 3D avatars to make remote communication more natural and realistic.
- Game developmentQuickly generate high-quality 3D character avatars to enhance the visual effects of games.
- Film and television productionUsed in special effects production to quickly generate high-quality avatars and improve production efficiency.