Open-source audio and video generation models such as daVinci, MagiHuman, and Sand.ai

What is daVinci-MagiHuman?

daVinci-MagiHuman is an open-source audio and video co-generation foundation model jointly developed by the GAIR Lab at Shanghai Institute of Innovation and Technology and Sand.ai. The model employs a single-stream Transformer architecture with 15 billion parameters, uniformly modeling text, video, and audio modalities without requiring cross-attention mechanisms. It excels at person-centric generation, supports multiple languages including Chinese, English, Japanese, Korean, German, and French, and can generate a 5-second 256p video in just 2 seconds from a single H100 image. Compared to Ovi 1.1 and LTX 2.3, it achieves win rates of 80% and 60.9% respectively. The code, model weights, and online demo are fully open-source.

daVinci-MagiHuman's main functions

• Audio and video co-generationIt supports the simultaneous generation of videos featuring natural speech and lip-sync, achieving true audio-visual integration.
• Multilingual supportIt supports voice generation for multiple languages, including Chinese (Mandarin and Cantonese), English, Japanese, Korean, German, and French.
• Human Image Derivation GenerationFocusing on the central scene of the character, it generates expressive facial expressions, body movements, and emotional communication.
• Speed ReasoningSupports generating a 5-second 256p video in 2 seconds on a single H100 GPU, meeting real-time interactive needs.
• High resolution outputThrough latent space super-resolution technology, it can be expanded to 540p or 1080p high-definition video.

The technical principles of daVinci-MagiHuman

• Single-stream unified architecturedaVinci-MagiHuman employs a single-stream Transformer architecture, unifying text, video, and audio into a single 15 billion parameter, 40-layer denoising network. It uses a pure self-attention mechanism for joint modeling, completely eliminating cross-attention or modality-specific branches. Architecturally, it adopts a "sandwich" design, with a few layers at the beginning and end retaining modality-related parameters, while the backbone network shares parameters, achieving a balance between modality specialization and deep fusion. Simultaneously, it introduces mechanisms such as explicit timestep conditional injection and Attention-Head gating to improve training stability and expressive power.
• Latent space super-resolutionThe model employs a two-stage pipeline: the bottom model first generates low-resolution audio and video latent variables, and then performs high-resolution refinement directly in the latent space through latent space super-resolution, avoiding additional VAE encoding and decoding overhead. The audio latent variables will continue to be used as input into the super-resolution model to maintain lip-sync effect.
• Inference acceleration optimizationThe inference stage uses a lightweight Turbo VAE decoder to reduce latency, integrates the self-developed MagiCompiler for full-graph compilation optimization, and brings about 1.2 times speedup through cross-layer operator fusion; combined with DMD-2 distillation technology, it achieves high-quality generation with only 8 steps of denoising.

Key information and usage requirements for daVinci-MagiHuman

• Model size15 billion parameters, 40 layers of Transformer
• Architectural featuresSingle-stream unified architecture, purely self-attention, no cross-attention.
• Generation capabilitySupports text/image-driven joint generation of human portrait audio and video.
• Supported languagesChinese (Mandarin, Cantonese), English, Japanese, Korean, German, French
• Reasoning speedA single H100 image can generate a 5-second 256p video in 2 seconds and a 1080p video in 38 seconds.
• PerformanceCompared to Ovi 1.1, the win rate is 80.0%; compared to LTX 2.3, the win rate is 60.9%.
• hardwareNVIDIA GPU (H100 recommended), CUDA support required.
• Software environment: Python 3.12, PyTorch 2.9.0, CUDA 12.x
• Dependency Components: Flash Attention (Hopper architecture), MagiCompiler (self-developed compiler), Turbo VAE