Hallo3 - A high dynamic range and realistic portrait animation generation framework jointly open-sourced by Fudan University and Baidu.

What is Hallo3?

Hallo3, jointly developed by Fudan University and Baidu, is a portrait image animation technology based on Diffusion Transformer Networks (DTNs) that generates highly dynamic and realistic videos. Based on a pre-trained transformer video generation model, Hallo3 effectively addresses the challenges of existing methods in handling non-frontal viewpoints, rendering dynamic objects, and generating immersive backgrounds. Hallo3 utilizes a novel video backbone network and an identity reference network to ensure facial identity consistency across video sequences. It also investigates speech audio conditions and motion frame mechanisms to achieve continuous video generation driven by speech audio. Experiments demonstrate that Hallo3 excels in generating realistic portraits with multiple orientations, adapting to complex poses and dynamic scenes, and producing realistic and smooth animation effects.

Hallo3's main functions

• Multi-view animation generationIt generates animated portraits from different perspectives (such as front, side, overhead, or low angle), breaking through the limitations of traditional methods that mainly rely on the frontal perspective.
• Dynamic object renderingIt processes the animation of dynamic objects around a portrait, such as a person holding a smartphone, microphone, or wearing close-fitting items, generating realistic motion of objects in a video sequence.
• Immersive background generationGenerate backgrounds with dynamic effects, such as a campfire in the foreground or a crowded street scene in the background, to enhance the realism and immersion of the video.
• Maintaining identity consistencyIt maintains the consistency of portrait identity throughout the entire video sequence and accurately identifies and preserves facial features even in long animations.
• Voice-driven animationBased on voice audio, it drives facial expressions and lip movements of portraits, achieving highly synchronized voice and facial animation, making the animation more natural and realistic.

The technical principles of Hallo3

• Pre-trained transformer video generation model:
  • InfrastructureThe CogVideoX model serves as the underlying architecture, using a 3D variational autoencoder (VAE) to compress video data. The model combines latent variables with text embeddings and processes them based on an expert transformer network.
  • Conditional MechanismThree conditional mechanisms are introduced: text prompts (ctext), audio conditions (caudio), and identity appearance conditions (cid). These conditional information are mainly integrated using cross-attention and adaptive layer normalization (adaLN).
• Identity Reference Network:
  • 3D VAE and Transformer LayerWe use a causal 3D VAE combined with a 42-layer identity reference network to extract identity features from reference images and embed them into the denoising latent code. Based on the self-attention mechanism, we enhance the model's representation and long-term retention of identity information.
  • Feature fusionThe visual features generated by the reference network are fused with the features of the denoising network to ensure that the generated facial animations maintain consistency and coherence over long time sequences.
• Voice audio conditions:
  • Audio embeddingThe wav2vec framework extracts audio features, generates frame-specific audio embeddings, and transforms the audio embeddings into a representation suitable for the model based on a linear transformation layer.
  • Cross-attention mechanismIn denoising networks, the cross-attention mechanism interacts with the audio embedding and the latent encoding, enhancing the coherence and relevance of the generated output and ensuring that the model effectively captures the audio signals that drive character generation.
• Video extrapolationIntroducing motion frames as conditional information: The last few frames of the generated video are used as input for the generation of subsequent segments. The motion frames are processed using 3D VAE to generate new latent codes, enabling time-consistent long video inference.
• Training and reasoning:
  • Training processThe process is divided into two phases. The first phase trains the model to generate videos with consistent identity; the second phase extends to audio-driven video generation by integrating an audio attention module.
  • Reasoning processThe model takes a reference image, driving audio, text prompts, and motion frames as input to generate a video with identity consistency and lip-sync.

Hallo3's project address

• Project official website:https://fudan-generative-vision.github.io/hallo3
• GitHub repository:https://github.com/fudan-generative-vision/hallo3
• HuggingFace model library:https://huggingface.co/fudan-generative-ai/hallo3
• arXiv technical paper:https://arxiv.org/pdf/2412.00733

Application scenarios of Hallo3

• Game developmentGenerate dynamic portrait animations for characters in the game, making their performance more natural and realistic, and enhancing the player's gaming experience.
• FilmmakingGenerate realistic character animations to enhance the visual effects and immersion of movies and animations.
• social mediaGenerate dynamic avatars for social media users, making their profiles more engaging and improving their personalized experience on social media.
• Online EducationAnimated virtual instructors can be generated to make online courses more vivid and interesting, thereby increasing students' learning interest and participation.
• Virtual Reality and Augmented RealityGenerate virtual characters in VR and AR applications to provide a more realistic interactive experience and enhance user immersion and engagement.