Wan2.1 - Alibaba's open-source AI video generation model

What is Wan2.1?

Wan2.1 is an open-source AI video generation model from Alibaba Cloud, boasting powerful visual generation capabilities. Wan2.1 supports both text-based and image-based video generation tasks, and includes two model sizes: a professional version with 14B parameters excels in complex motion generation and physical modeling, offering superior performance; and a high-speed version with 1.3B parameters, capable of running on consumer-grade graphics cards with low memory requirements, suitable for secondary development and academic research. The Wan2.1 model is based on a causal 3D VAE and video Diffusion Transformer architecture, achieving efficient spatiotemporal compression and long-term dependency modeling. The 14B version significantly outperformed domestic and international models such as Sora, Luma, and Pika in the authoritative Vbench benchmark with a total score of 86.22%, firmly holding the top position. Wan2.1 is open-source under the Apache 2.0 license, supports multiple mainstream frameworks, and is available on GitHub, HuggingFace, and the Moda community, facilitating developer use and deployment.

Main functions of Wan2.1

• Wensheng VideoIt generates corresponding video content based on the input text description, supports long text commands in Chinese and English, and accurately reproduces scene transitions and character interactions.
• Image and videoIt generates videos based on images, enabling more controllable creation and is suitable for the need to expand from static images to dynamic videos.
• Complex motion generationIt can stably display complex movements of people or objects, such as rotation, jumping, and turning, and supports advanced camera movement control.
• Physical law simulationIt accurately recreates real-world physical scenarios such as collisions, bounces, and cuts, generating video content that conforms to the laws of physics.
• Multi-style generationIt supports a variety of video styles and textures to suit different creative needs, and also supports video output with different aspect ratios.
• Text effects generationIt has the ability to generate Chinese text and supports Chinese and English text effects, enhancing the visual appeal of videos.

Technical Principles of Wan2.1

• Causal 3D VAE (Variational Autoencoder) ArchitectureWanxiang's self-developed causal 3D VAE architecture is specifically designed for video generation. It works by compressing input data into a representation of the latent space using an encoder, and then reconstructing the output using a decoder. In video generation, 3D VAE can handle spatiotemporal information in the video, while incorporating causal constraints to ensure the coherence and logic of the generated video.
• Video Diffusion Transformer ArchitectureIt is based on the mainstream video diffusion model and Transformer architecture. The diffusion model gradually removes noise to generate data, while the Transformer captures long-term dependencies based on the self-attention mechanism.
• Model training and inference optimization:
  • Training phaseA distributed strategy combining DP (data parallelism) and FSDP (fully sharded data parallelism) is used to accelerate the training of text and video encoding modules. For the Diffusion module, a parallel strategy based on a mixture of DP, FSDP, RingAttention, and Ulysses is used to further improve training efficiency.
  • Reasoning stageDistributed acceleration is achieved using CP (channel parallelism) to reduce the latency of generating individual videos. For large models, inference efficiency is further optimized based on model segmentation techniques.

Wan2.1 performance advantages

• Excellent production qualityIn the Vbench evaluation, the professional version with 14 parameters achieved a total score of 86.22%, significantly outperforming other domestic and international models (such as Sora, Luma, Pika, etc.) and firmly holding the top spot.
• Supports consumer-grade GPUsThe 1.3B version, with its ultra-fast parameters, requires only 8.2GB of video memory to generate 480P video. It is compatible with almost all consumer-grade GPUs and can generate a 5-second 480P video in about 4 minutes on an RTX 4090.
• Multifunctional supportIt supports various tasks such as text-to-video, image-to-video, video editing, text-to-image, and video-to-audio conversion, while also possessing visual effects and text rendering capabilities to meet the creative needs of multiple scenarios.
• High-efficiency data processing and architecture optimizationBased on a self-developed causal 3D VAE and optimized training strategies, it supports efficient encoding and decoding of videos of arbitrary length, significantly reduces inference memory usage, and improves training and inference efficiency.

Wan2.1 project address

• Project official website:https://wanxai.com
• GitHub repository:https://github.com/Wan-Video/Wan2.1
• HuggingFace model library:https://huggingface.co/Wan-AI

Wan2.1 Effect Demonstration

• Complex motionIt excels at generating realistic videos that include a wide range of body movements, complex rotations, dynamic scene transitions, and smooth camera movements.

• Physics simulationIt can generate videos that accurately simulate the physical laws of the real world and realistically depict the interactions between objects.

• Cinema-quality pictureIt offers cinematic visual effects with rich textures and diverse stylized effects.

• Controllable editingIt features a universal editing model, allowing for precise editing using image or video references.

Application scenarios of Wan2.1

• Film production and special effectsGenerate complex action scenes, special effects shots, or virtual character animations, reducing shooting costs and time.
• Advertising and MarketingQuickly generate creative advertising videos, producing personalized video content based on product characteristics or brand tone.
• Education and TrainingGenerate educational videos, such as science experiment demonstrations, historical scene reenactments, or language learning videos, to enhance the learning experience.
• Game developmentUsed to generate in-game animations, cutscenes, or virtual character movements to enhance the game's visual effects and immersion.
• Personal creation and social mediaIt helps creators quickly generate creative videos for social media sharing, vlog production, or personal project showcases.