Hallo - An open-source AI lip-syncing portrait video generation framework from Fudan University, Baidu, and others.

What is Hallo?

Hallo, a collaborative research project by researchers from Fudan University, Baidu, ETH Zurich, and Nanjing University, is an AI-powered lip-sync portrait animation technology that generates realistic and dynamic portrait images and videos based on speech audio input. The framework employs a diffusion-based generative model and a hierarchical audio-driven visual synthesis module, improving the synchronization accuracy between audio and visual output. Hallo's network architecture integrates a UNet denoiser, temporal alignment techniques, and a reference network to enhance the quality and realism of the animation, significantly improving not only image and video quality but also enhancing lip-sync accuracy and increasing the diversity of movements.

Hallo's features

• Audio-synchronized animationHallo utilizes advanced audio analysis technology to combine input speech audio with portrait images to generate dynamic facial animations. Through a precise lip-sync algorithm, it ensures that the lip movements in the video animation are synchronized with the sound in the audio, thus creating a realistic speaking effect.
• Facial expression generationBased on the emotions and tone changes in the audio signal, Hallo can automatically recognize and generate corresponding facial expressions, including smiles, frowns, and surprise, making the "performance" of the animated characters in the video more natural and emotional.
• Head posture controlHallo allows for fine-tuning of head posture in video animations, such as head tilting and turning, enabling video animations to better reflect the intent and emotion of audio content and enhance visual and auditory coordination.
• Personalized animation customizationUsers can customize the style, expressions, and movements of animations based on different application scenarios and personal characteristics. Hallo's personalization feature allows users to create unique characters that meet specific visual and emotional expression needs.
• Time consistency maintenanceHallo uses time alignment technology to ensure smooth transitions between actions and expressions in animation, avoiding abrupt and unnatural changes.
• Movement diversityIn addition to basic audio synchronization, Hallo also supports generating a variety of movements and styles. Users can select different motion libraries as needed to add more dynamic elements to animated characters, such as gestures and blinks, thereby enriching the expressiveness of the video.

Hallo's official website entrance

• Official project homepage:https://fudan-generative-vision.github.io/hallo/#/
• GitHub repository:https://github.com/fudan-generative-vision/hallo
• Hugging Face model library:https://huggingface.co/fudan-generative-ai/hallo
• arXiv technical paper:https://arxiv.org/abs/2406.08801

Hallo's technical principles

• Layered audio-driven visual synthesisHallo employs a hierarchical approach to process audio and visual information. This hierarchical structure allows the model to process lip movements, facial expressions, and head poses separately, and then fuse these elements together using adaptive weights.
• End-to-end diffusion modelHallo uses a diffusion-based generative model, a method for generating data from a latent space. During the training phase, noise is gradually added to the data, and then the noise is removed in the reverse process to reconstruct a sharp image.
• Cross-attention mechanismThrough a cross-attention mechanism, Hallo is able to establish a connection between audio and visual features. This mechanism allows the model to focus its attention on the facial regions most relevant to the current audio input.
• UNet DenoisingHallo utilizes a UNet-based denoiser to progressively remove noise from images, generating clear animation frames. The UNet architecture is renowned for its effectiveness in image segmentation tasks, using low-level feature maps through skip connections to improve generation quality.
• Time alignment technologyTo maintain the continuity of the animation in time, Hallo uses time alignment technology, which helps ensure smooth transitions and consistency between consecutive frames.
• ReferenceNetReferenceNet is used to encode global visual texture information to achieve consistent and controllable character animation. It helps the model reference existing images during the generation process to enhance the visual quality of the output.
• Face and audio encoderHallo uses a pre-trained facial encoder to extract identity features from portraits, while using an audio feature encoder (such as wav2vec) to convert audio signals into information that can drive animated motion.
• Adaptive weight adjustmentHallo allows you to adjust the weights of different visual components (such as lips, facial expressions, and poses) to control the diversity and detail of animation.
• Training and ReasoningDuring training, Hallo improves single-frame generation capability by optimizing the parameters of the facial image encoder and the spatial cross-attention module. During inference, the model combines reference images and driving audio to generate animated video sequences.