Live Avatar - A real-time digital human model open-sourced by Alibaba in collaboration with universities

What is Live Avatar?

Live Avatar, developed by Alibaba in collaboration with several universities, is a real-time digital human model capable of generating digital human videos of unlimited length through audio and video-driven processes. Leveraging a 14 billion-parameter diffusion model, Live Avatar achieves real-time streaming generation at 20FPS, supporting stable output for over 10,000 seconds while avoiding facial drift and color distortion during long-duration generation. It can be used in various scenarios such as e-commerce live streaming, news broadcasting, and virtual assistants, providing a high-fidelity, low-latency interactive experience.

The main functions of Live Avatar

• Real-time audio and video driverIt captures the user's voice and facial expressions through microphones and cameras, driving the digital human to respond synchronously with lip movements and facial expressions, achieving low-latency face-to-face interaction.
• Stable generation for unlimited durationIt supports continuous video generation of up to 10,000 seconds, ensuring that the digital human's facial features, skin tone, style, and other characteristics remain consistent throughout the process, thus avoiding identity drift and image quality degradation during long-term generation.
• High-fidelity image qualityBased on a diffusion model with 14 billion parameters, it generates digital human images that are rich in detail, clear, and natural.
• Streaming generationIt adopts streaming processing technology, supports continuous expansion of real-time video streams, and is suitable for real-time application scenarios such as live streaming.

The technical principles of Live Avatar

• Diffusion model optimization:Using a diffusion model with 14 billion parameters, the multi-step bidirectional diffusion model is transformed into a high-efficiency 4-step flow diffusion model through the Distribution Matching Distillation technique, which greatly improves the generation speed.The Timestep-forcing Pipeline Parallelism technique is used to decouple the denoising stage and distribute it across multiple devices, achieving linear acceleration and supporting large-scale parallel computing.
• Unlimited duration generation mechanism:
  • Rolling RoPE (Relative Positional Encoding): Dynamically update the relative position encoding of the reference frame to maintain the relative position relationship between the generated frame and the reference frame and avoid identity drift.
  • Adaptive Attention SinkThe initial reference frame is replaced with the generated frame to eliminate persistent factors that cause distribution drift and maintain the distribution consistency of the generated frames.
  • Historical interference mechanismNoise is injected into the KV buffer to simulate errors in inference, guide the model to extract motion information from historical frames and stable details from reference frames, and avoid error accumulation.
• Real-time interactive technology:By combining microphone and camera inputs, the digital human model is driven in real time through audio and video signals, achieving low-latency interactive responses.Optimize generation speed and latency to achieve a real-time generation speed of 20FPS, with a first frame latency of only 2.89 seconds, ensuring a smooth interactive experience.

Live Avatar project address

• Project official websitehttps://liveavatar.github.io/
• GitHub repositoryhttps://github.com/Alibaba-Quark/LiveAvatar
• HuggingFace model libraryhttps://huggingface.co/Quark-Vision/Live-Avatar

Application scenarios of Live Avatar

• e-commerce live streamingLive Avatar enables 24/7 uninterrupted product demonstrations and explanations, reducing labor costs and improving live streaming efficiency.
• News BroadcastThe model supports the automatic generation of virtual anchors to broadcast news, quickly generating multilingual videos and improving the timeliness of news.
• Entertainment Live StreamingVirtual idols enhance audience engagement by performing live interactive shows and holding online concerts.
• Intelligent Customer ServiceDigital human customer service can answer user questions in real time, providing 24/7 uninterrupted service and improving user experience.
• bank tellerVirtual financial advisors provide standardized business consulting services, supplementing human services and improving bank operational efficiency.