OmniHuman - A framework launched by ByteDance for generating full-body dynamic videos from a single photo.

What is OmniHuman?

OmniHuman is an end-to-end multimodal conditional human video generation framework launched by ByteDance. It can generate realistic human videos based on a single human image and motion signals (such as audio, video, or a combination of both). OmniHuman overcomes the performance bottleneck caused by the scarcity of high-quality data in previous methods by employing a multimodal motion conditional hybrid training strategy. It supports image input with arbitrary aspect ratios (including portrait, half-body, and full-body images) and can adapt to various scenarios. OmniHuman excels in singing, dialogue, and gesture processing, supports multiple visual and audio styles, and is compatible with audio, video, and combined-drive technologies to generate high-quality video content.

OmniHuman's main functions

• Multimodal driven video generation:
  • It supports audio-driven (such as speaking and singing) and gesture-driven (such as gestures and movements) approaches, and can combine the two to generate natural and smooth human motion videos.
  • It supports multiple input formats, including close-up faces, half-body portraits, and full-body portraits, and is compatible with images of different proportions and styles.
• High realism and diverse movements:
  • The generated videos are visually highly realistic, featuring natural facial expressions, body movements, and smooth dynamic effects.
  • It can handle complex actions and object interactions, such as playing an instrument while singing, and natural interactions between gestures and objects.
• Flexible video generation:
  • It supports video generation with arbitrary aspect ratios and durations, and generates video clips of different lengths based on the input signal.
  • It is compatible with a variety of image styles, including realistic, cartoon, and stylized characters.
• Multi-scenario adaptabilityGenerate high-quality videos in various scenarios, including different backgrounds, lighting conditions, and camera angles.

OmniHuman's technical principles

• Mixed Conditional Training Strategy:
  • Multi-condition fusionThis approach integrates various motion-related conditions, such as text, audio, and posture, into the training process, reducing waste caused by data filtering and leveraging the complementarity between different conditions.
  • Phased trainingBased on a three-stage training strategy, different conditions (text, audio, and pose) are gradually introduced, and the training ratio is adjusted according to the strength of the conditions to optimize the model's generalization ability.
  • Training principlesFor tasks with stronger conditions, use tasks with weaker conditions and their corresponding data to expand the data scale. The stronger the conditions, the lower the training ratio should be to avoid the model becoming overly dependent on the strong conditions.
• Diffusion converter architecture:
  • DiT-based modelsOmniHuman is based on the advanced video generation model architecture DiT, which uses causal 3DVAE to project videos into a latent space and uses flow matching as the training objective.
  • Conditional Injection:
    • Audio conditionsThe audio features are extracted using the wav2vec model, combined with video frame features to generate audio tokens, which are then injected into the model based on a cross-attention mechanism.
    • Postural conditionsThe pose condition is processed using a pose guider, which combines pose heatmap features with video frame features to generate pose tokens, which are then fed into the model along with a noisy latent representation.
    • Text conditions: Retain the text branches in the DiT architecture for use in describing the content of the generated video.
  • Reference condition processingIt adopts an innovative reference condition strategy, based on Modified Rotation Position Embedding (RoPE), to fuse reference image features with video features without the need for additional network modules.
  • Reasoning strategies:
    • Free Classifier Guidance (CFG)During the inference process, a CFG strategy is applied to the audio and text conditions, based on gradually reducing the CFG intensity to balance expressiveness and computational efficiency, and to reduce imperfections (such as wrinkles) in the generated video.
    • Long video generationUse the last few frames of the previous video clip as motion frames to ensure temporal continuity and identity consistency in the generation of long videos.

OmniHuman's project address

• Project official website:https://omnihuman-lab.github.io/
• arXiv technical paper:https://arxiv.org/pdf/2502.01061

Application scenarios of OmniHuman

• Film and EntertainmentGenerate virtual character animations, virtual anchors, music videos, etc., improving content production efficiency and visual effects.
• Game developmentGenerate natural movements for game characters and NPCs, enhancing game immersion and interactivity.
• Education and TrainingCreate virtual teachers and simulated training videos to assist language learning and vocational skills training.
• Advertising and MarketingGenerate personalized ads and brand promotion videos to enhance user engagement and content appeal.
• Social media and content creationHelps creators quickly generate high-quality short videos, supports interactive video creation, and increases the fun of content.