EMO - Alibaba's AI portrait video generation framework

What is EMO?

EMO (Emote Portrait Alive) is a framework developed by researchers at Alibaba Group's Intelligent Computing Research Institute. It's an audio-driven AI portrait video generation system capable of generating videos with expressive facial expressions and various head poses from a single input reference image and audio recording. The system captures subtle differences in human expressions and the diversity of individual facial styles, resulting in highly realistic and expressive animations.

EMO's official website entrance

• Official project homepage:https://humanaigc.github.io/emote-portrait-alive/
• arXiv research paper:https://arxiv.org/abs/2402.17485
• GitHub:https://github.com/HumanAIGC/EMO(The model and source code will be open-sourced later)

Key characteristics of EMO

• Audio-driven video generationEMO can generate video directly from input audio (such as speaking or singing) without relying on pre-recorded video clips or 3D facial models.
• High expressiveness and realismEMO-generated videos are highly expressive, capable of capturing and reproducing subtle differences in human facial expressions, including delicate micro-expressions and head movements that match the rhythm of the audio.
• Seamless frame transitionEMO ensures smooth and natural transitions between video frames, avoiding issues such as facial distortion or frame jitter, thereby improving the overall quality of the video.
• Identity preservationThrough the FrameEncoding module, EMO is able to maintain the consistency of character identity during video generation, ensuring that the character's appearance remains consistent with the input reference image.
• Stable control mechanismEMO employs stabilization control mechanisms such as a speed controller and a face region controller to enhance stability during video generation and avoid issues such as video crashes.
• Flexible video lengthEMO can generate videos of any length based on the length of the input audio, providing users with flexible creative space.
• Cross-language and cross-styleEMO's training dataset covers a variety of languages and styles, including Chinese and English, as well as realism, animation, and 3D styles, which allows EMO to adapt to different cultures and art styles.

How EMO works

1. Input preparationThe user provides a reference image (usually a static portrait of the target character) and corresponding audio input (such as a voice speaking or singing). These inputs will serve as the basis for generating the video.
2. Feature extractionReferenceNet is used to extract features from reference images. ReferenceNet is a network with a similar structure to the backbone network, but it focuses on extracting detailed features from the input image.
3. Audio processingThe audio input is processed by a pre-trained audio encoder to extract audio features. These features capture information such as the rhythm, pitch, and pronunciation of the speech, which will be used to drive the facial expressions and head movements of the characters in the video.
4. diffusion processDuring the diffusion process, the main network receives multiple frames of noise as input and attempts to denoise this noise into consecutive video frames at each time step. This process involves two main attention mechanisms: Reference-Attention and Audio-Attention. Reference-Attention is used to maintain the consistency of character identity, while Audio-Attention is used to modulate the character's actions.
5. Time moduleTo handle the temporal dimension and adjust the speed of motion, EMO uses temporal modules. These modules operate on intra-frame features through a self-attention mechanism to capture the dynamic content of the video and ensure coherence and consistency between consecutive frames.
6. Facial positioning and speed controlTo ensure the stability and controllability of the generated character movements, EMO uses a Face Locator and Speed Layers. The Face Locator encodes the facial bounding box region through lightweight convolutional layers, while the Speed Layers control the speed and frequency of the movements by embedding the head rotation speed into the generation process.
7. Training strategyThe training of EMO is divided into three stages: image pre-training, video training, and velocity layer ensemble. In the image pre-training stage, the main network and ReferenceNet learn from single-frame images. In the video training stage, a temporal module and an audio layer are introduced to handle consecutive frames. Finally, in the velocity layer ensemble stage, only the temporal module and the velocity layer are trained to ensure the audio's ability to drive the character's movements.
8. Generate videoDuring the inference phase, EMO uses the DDIM sampling algorithm to generate video segments. Through an iterative denoising process, a portrait video synchronized with the input audio is finally generated.