MirrorMe - An audio-driven portrait animation framework launched by Alitongyi

What is MirrorMe?

MirrorMe is a real-time, high-fidelity audio-driven portrait animation framework developed by Alibaba's Tongyi Lab. Based on the LTX video model, the framework incorporates three key innovations: an identity injection mechanism, an audio-driven control module, and a progressive training strategy, addressing the challenge of generating high-fidelity, time-consistent animated videos in real time. MirrorMe topped the EMTD benchmark, demonstrating superior image fidelity, lip-sync accuracy, and temporal stability. It also boasts high inference speed, meeting real-time generation requirements and providing robust technical support for applications such as e-commerce live streaming.

MirrorMe's main functions

• Real-time high-fidelity animation generationMirrorMe can generate high-quality half-body animated videos in real time, supporting smooth playback at 24FPS to meet the needs of real-time interaction.
• Audio-driven lip-syncMirrorMe can accurately convert audio signals into corresponding lip movements, achieving highly realistic lip-sync.
• Identity preservationMirrorMe uses a reference identity injection mechanism to ensure that the generated animated video is highly consistent in appearance with the input reference image.
• Precise facial expression and gesture controlMirrorMe can control facial expressions and achieve precise control of hand movements based on gesture signals.

MirrorMe's technical principles

• InfrastructureThe core architecture is based on the LTX video model. The LTX model is a video generation model based on the Diffusion Transformer, which uses spatiotemporal tokenization (each token corresponds to 32×32×8 pixels) to achieve an extremely high compression ratio (1:8192).
• Identity Injection MechanismTo ensure the generated video closely resembles the input reference image in appearance, MirrorMe employs a reference identity injection mechanism. The reference image is encoded using a 3D variational autoencoder (VAE) from the LTX model, yielding reference latent space variables. These encoded latent space variables are then concatenated with the noisy latent space variables over time, injecting identity information based on a self-attention mechanism. This ensures the generated video maintains visual consistency with the reference image.
• Audio driver control moduleThis system achieves precise synchronization between audio signals and video generation based on an audio driver control module. The specific implementation is as follows:
  • Causal audio encoderThe pre-trained wav2vec2 model is used to extract frame-level audio embeddings from the original audio signal, and the audio sequence is progressively compressed based on a causal audio encoder to match the temporal resolution with the video latent space variables.
  • audio adapterThe extracted audio features are injected into the video generation process via an audio adapter. The audio adapter fuses audio and video features based on a cross-attention mechanism, ensuring that the audio signal can accurately drive facial expressions and lip movements.
• Progressive training strategyTo improve the training effectiveness and generation quality of the model, MirrorMe employs a progressive training strategy. It first trains on close-up facial images, focusing on learning the mapping relationship between audio and facial expressions. Building upon this close-up training, it gradually expands to half-body synthesis, while introducing facial masking to maintain the dynamic response of the facial region. A pose encoder module incorporates key hand signals to achieve precise control of hand movements.
• Efficient ReasoningMirrorMe utilizes efficient compression and denoising technology based on the LTX model, significantly improving inference speed. Temporally, it compresses the input video to 1/8 of its original length, and spatially, it compresses it to 1/32 of its original resolution in both height and width, greatly reducing the number of latent space variables that need to be processed. On consumer-grade NVIDIA GPUs, MirrorMe can achieve real-time generation at 24 FPS, meeting the stringent latency requirements of real-time applications.

MirrorMe project address

• arXiv technical paperhttps://arxiv.org/pdf/2506.22065v1

Application scenarios of MirrorMe

• e-commerce live streamingIt generates lifelike virtual anchors, producing natural expressions and movements in real time based on the input audio, thereby enhancing the interactivity and appeal of live streams.
• Virtual Customer ServiceIn online customer service scenarios, the generated virtual customer service avatar interacts with users in real time based on audio, providing a natural and friendly service experience. It also supports multiple languages and can serve users from different language backgrounds.
• Online EducationMirrorMe generates virtual teachers who display expressions and actions in real time based on the teaching content, making teaching more vivid and interesting. It also generates virtual avatars similar to students for personalized learning, providing teaching content that closely matches students' needs.
• Virtual MeetingMirrorMe generates virtual attendee avatars that display facial expressions and gestures in real time based on attendees' voices, enhancing meeting interactivity and engagement. It is suitable for remote collaboration and strengthens the sense of face-to-face communication among team members.
• social mediaUsers can generate their own virtual avatars, interact on social media via audio, share interesting video content, and enhance the fun and engagement of live streams.