VASA-1 - A framework for generating lip-sync videos from still photos, introduced by Microsoft.
VASA-1 is a framework proposed by Microsoft Research Asia that converts still photos into lip-syncing videos. It can generate realistic 3D speaking facial animations in real time based on a single still face photo and a piece of audio.
What is VASA-1?
VASA-1, proposed by Microsoft Research Asia, is a framework for converting still photos into lip-synced animated videos. It can generate realistic 3D speaking facial animations in real time from a single still facial photo and a short audio clip. This technology creates highly realistic and dynamic virtual characters through precise lip-sync, rich facial expression details, and natural head movements. VASA-1's core innovation lies in its full-view facial dynamics and head motion generation model, which operates in the facial latent space, efficiently generating high-resolution videos while supporting online generation and low latency.
VASA-1 Features
- Realistic lip-syncVASA-1 can generate lip movements that are precisely synchronized with the input speech audio, providing a highly realistic speaking effect.
- Rich facial expressionsIn addition to lip movements, VASA-1 can capture and reproduce a range of complex facial expressions and subtle emotional details, enhancing the realism of animation.
- Natural head movementsThe model can simulate natural head movements, such as turning and tilting, making the generated speaking facial videos more lifelike.
- High-efficiency video generationVASA-1 supports online generation of 512×512 resolution video at up to 40 FPS with extremely low initial latency, making it suitable for real-time applications.
- Flexible and controllable generationBy accepting optional signals as conditions, such as primary gaze direction, head distance, and emotional shift, VASA-1 can control the generation process, improving the diversity and adaptability of the output.
- Processing different inputsVASA-1 is able to process photo and audio inputs outside of the training distribution, such as artistic photos, singing audio, and non-English speech.
VASA-1 official website entrance
- Official project homepage:https://www.microsoft.com/en-us/research/project/vasa-1/
- arXiv research paper:https://arxiv.org/abs/2404.10667
How VASA-1 works
- Input preparationVASA-1 accepts two main inputs: a static facial image of any individual and a clip of speech audio from any individual.
- Facial feature extractionThe face encoder is used to extract features such as 3D appearance volume, identity code, head pose, and facial dynamic code from the input static face image.
- Facial latent space modeling: Construct a facial latent space that is highly decoupled from facial dynamics and other factors (such as identity and appearance), and has the ability to express rich facial details and dynamic nuances.
- Diffusion model trainingTrain a diffusion-based model (Diffusion Transformer) that can generate comprehensive facial dynamics and head movements in the facial latent space, given audio and optional control signals.
- Conditional signal integrationThe main gaze direction, head distance, and emotional shift are used as control signals and input into the diffusion model to guide the generation of facial dynamics.
- Facial dynamics and head motion generationUsing a trained diffusion model, latent code sequences for facial dynamics and head movements are generated based on the input audio features and conditional signals.
- Video frame generationUsing a face decoder and appearance and identity features extracted from the encoder, the final video frames are generated based on the generated facial dynamics and potential head movement codes.