TaoAvatar - Alibaba's real-time high-definition 3D full-body dialogue digital human technology

What is TaoAvatar?

TaoAvatar is a high-fidelity, lightweight 3D full-body dialogue virtual human technology developed by Alibaba Group's research team. Based on 3D Gaussian sputtering technology, it can generate photorealistic 3D full-body virtual avatars, supporting high-resolution rendering with low storage requirements. TaoAvatar can run in real time at a high frame rate of 90FPS on various mobile and AR devices, achieving natural synchronization of lip movements, facial expressions, gestures, and body postures through multiple signals such as voice, facial expressions, gestures, and body postures.

Main functions of TaoAvatar

• High-fidelity full-body dynamic virtual avatar generationIt can generate realistic, topologically consistent 3D full-body virtual avatars from multi-view image sequences, supporting fine control over posture, gestures, and expressions.
• Real-time rendering and low storage requirementsIt can run in real time at a high frame rate of 90FPS on a variety of mobile and AR devices, supports high-resolution rendering, and maintains low storage requirements.
• Multi-signal driveIt can be driven by various signals such as voice, facial expressions, gestures and body postures to achieve natural synchronization of lip movements, facial expressions and actions.
• Lightweight architectureBy "baking" complex non-rigid deformations into a lightweight MLP network and combining hybrid shape compensation details, the operating efficiency is significantly improved.

The technical principles of TaoAvatar

• 3D Gaussian Sputtering (3DGS) Technology3DGS represents points in a scene using 3D Gaussian functions, which are then projected onto a 2D image plane for rendering. Each 3D Gaussian function is described by parameters such as position, covariance, color, and transparency. 3D point clouds are estimated from multi-view images using Structure from Motion (SfM) technology, and each point is then converted into a Gaussian function, trained using stochastic gradient descent.
• Attitude-dependent non-rigid deformation processingTaoAvatar decomposes complex non-rigid deformations into rigid deformations and shape deformations. It then uses knowledge distillation to "bake" the shape deformations into a lightweight MLP network. This allows for efficient handling of complex pose-dependent non-rigid deformations while maintaining the realism and controllability of the virtual avatar.
• Learnable Gaussian Mixture ShapesTo further enhance the visual detail of virtual avatars, TaoAvatar introduces learnable Gaussian mixture shapes. By training a neural network to learn Gaussian mixture shape parameters for different poses and expressions, these parameters are then applied to the virtual avatar. This allows the virtual avatar to maintain extremely high fidelity across various poses and expressions.
• Real-time rendering and optimizationTaoAvatar employs various optimization techniques, such as GPU acceleration, reducing unnecessary computation, and optimizing model structure and parameters, to achieve high-quality real-time rendering. It maintains a smooth 90 frames per second on high-definition stereoscopic display devices such as the Apple Vision Pro.

TaoAvatar's project address

• Project official website:https://pixelai-team.github.io/TaoAvatar/
• arXiv technical paper:https://arxiv.org/pdf/2503.17032

Application scenarios of TaoAvatar

• e-commerce live streamingCreate realistic virtual anchors to improve user experience and reduce labor costs.
• Holographic communicationGenerate realistic virtual avatars in remote communication to enhance immersion.
• Virtual MeetingParticipants can use personalized virtual avatars to communicate, enhancing interactivity.
• Online EducationUsing virtual humans for online courses increases engagement and fun.
• Virtual EntertainmentCreate personalized virtual characters in games and virtual reality applications.