LLIA - Meituan's audio-driven portrait video generation framework
LLIA (Low-Latency Interactive Avatars) is a real-time audio-driven portrait video generation framework developed by Meituan, based on a diffusion model. The framework generates virtual avatars based on audio input and supports low latency...
What is LLIA?
LLIA (Low-Latency Interactive Avatars) is a real-time audio-driven portrait video generation framework developed by Meituan, based on a diffusion model. The framework generates virtual avatars based on audio input, supporting low-latency, high-fidelity real-time interaction. LLIA uses variable-length video generation technology to reduce initial video generation latency, and combines a consistency model training strategy and model quantization technology to significantly improve inference speed. LLIA supports fine-grained control of the virtual avatar's state (such as speaking, listening, idle) and facial expressions using category labels, providing users with a smooth and natural interactive experience.
LLIA's main functions
- Real-time audio-driven portrait video generationIt generates corresponding portrait videos in real time based on the input audio signal, achieving synchronization of voice, facial expressions, and movements.
- Low-latency interactionIt can achieve high frame rate (e.g., 78 FPS at 384×384 resolution) and low latency (e.g., 140 ms) video generation on high-performance GPUs, making it suitable for real-time interactive scenarios.
- Multiple state switchingIt supports controlling the state of virtual avatars based on category tags, such as speaking, listening, and idle states, allowing virtual avatars to react naturally according to the scene.
- Facial expression controlBased on portrait animation technology, the facial expressions of reference images are modified to achieve fine control over facial expressions in the generated video, thereby enhancing the expressiveness of the virtual avatar.
LLIA Technical Principles
- Diffusion Model FrameworkLLIA uses a diffusion model as its basic architecture, based on powerful generation capabilities and high-fidelity output. The diffusion model generates images and videos by progressively removing noise.
- Variable length video generationLLIA introduces a dynamic training strategy that allows the model to generate video clips of different lengths during inference, reducing latency while maintaining video quality.
- Consistency ModelsBy introducing a consistency model and discriminator, LLIA achieves high-quality video generation with fewer sampling steps, significantly accelerating inference speed.
- Model quantization and parallelization: Use model quantization (such as INT8 quantization) and pipelined parallelism techniques to further optimize the inference performance of the model and reduce the computational resource requirements.
- Conditional Input and ControlBased on category tags and portrait animation technology, LLIA dynamically adjusts the state and expression of the virtual avatar according to the characteristics of the input audio, achieving a natural interactive effect.
- High-quality datasetsLLIA is trained using over 100 hours of high-quality datasets, including open-source data, network-collected data, and synthetic data, to improve the model's performance in different scenarios.
LLIA project address
- Project official website:https://meigen-ai.github.io/llia/
- GitHub repository:https://github.com/MeiGen-AI/llia
- arXiv technical paper:https://arxiv.org/pdf/2506.05806
Application scenarios of LLIA
- Virtual interviewGenerate virtual interviewers or job applicants, enhancing the realism and interactivity of the interview based on real-time facial expressions and gestures.
- Mobile chatbotIt provides chatbots with vivid virtual avatars, generating expressions and actions in real time based on voice input, thereby enhancing the user interaction experience.
- Virtual Customer ServiceGenerate virtual customer service representatives that respond to customer voice messages in real time and improve customer satisfaction with natural expressions and gestures.
- Online EducationIt generates virtual teachers or teaching assistants, and adjusts their expressions and actions in real time based on the teaching content and student feedback to enhance teaching interactivity.
- Virtual socialIt generates virtual avatars for users, enabling them to control facial expressions and actions via voice, thus achieving a more realistic and natural social experience.