Open-Sora 2.0 - An open-source AI video generation model from Luchen Technology.

What is Open-Sora 2.0?

Open-Sora 2.0 is a brand-new open-source, state-of-the-art (SOTA) video generation model launched by Luchen Technology. Open-Sora 2.0 successfully trained a commercial-grade model with 11 parameters using $200,000 (224 GPUs), significantly reducing training costs compared to traditional high-performance video generation models. The model performs exceptionally well in VBench and user preference tests, with performance comparable to or even surpassing mainstream closed-source models such as HunyuanVideo and the 30-parameter Step-Video. Open-Sora 2.0 is based on a 3D autoencoder, a 3D full attention mechanism, and an MMDiT architecture, combined with an efficient parallel training scheme and a high-compression autoencoder, significantly improving training efficiency and inference speed.

Main features of Open-Sora 2.0

• High-quality video generationIt generates smooth 720p resolution, 24 FPS video, supports a variety of scenes and styles, and performs excellently from natural scenery to complex dynamic scenes.
• Controllable range of motionAdjust the range of motion of people or objects in the video according to user needs to achieve a more delicate and precise dynamic performance.
• Text-to-Video (T2V) GenerationIt supports generating corresponding video content directly from text descriptions, meeting the needs of creative video production and content generation.
• Image to Video (I2V) GenerationBy combining open-source image models, videos can be generated based on images, further improving the generation effect and diversity.

Technical Principles of Open-Sora 2.0

• Model ArchitectureThis system efficiently processes video data based on a 3D autoencoder, capturing dynamic information in the temporal dimension. It introduces a full attention mechanism to improve the spatiotemporal consistency of video generation. Combined with a multimodal diffusion (MMDiT) architecture, it more accurately captures the correlation between text and video content.
• High compression ratio self-encoderBased on a 4×32×32 high compression ratio autoencoder, inference costs are significantly reduced.
• High-efficiency training methodsBased on a multi-stage, multi-level data filtering mechanism, high-quality data input is ensured, improving training efficiency. Training is prioritized at low resolution to learn key dynamic features, gradually increasing resolution to significantly reduce computational overhead. Prioritizing training on image-to-video tasks, image features guide video generation, accelerating model convergence.
• Parallel Training and OptimizationCombining ColossalAI and system-level optimizations improves the utilization of computing resources. Key technology optimizations include sequence parallelism, ZeroDP, fine-grained Gradient Checkpointing, automatic recovery mechanisms, efficient data loading and memory management, etc., to ensure maximum training efficiency.
• Model initialization and distillationWe utilize the open-source graph-based video model FLUX for initialization, reducing training costs. A distillation-based optimization strategy enhances the expressive power of the autoencoder's feature space, reducing the amount of data and training time required.

Performance of Open-Sora 2.0

• Comparable to HunyuanVideo and 30B Step-VideoOpen-Sora 2.0, with only 11 parameters, achieves comparable performance in VBench and human preference tests to high-cost, mainstream closed-source large models such as HunyuanVideo and Step-Video with 30 parameters.
• User Preference AssessmentIn the three key dimensions of visual performance, text consistency, and motion performance, Open-Sora 2.0 outperforms the open-source state-of-the-art model HunyuanVideo and the commercial model Runway Gen-3 Alpha in at least two metrics, achieving better performance with fewer resources.
• VBench indicator shows strong performanceAccording to VBench, a leading benchmark for video generation, Open-Sora 2.0 demonstrates outstanding performance. After upgrading from Open-Sora 1.2 to 2.0, the performance gap with the industry-leading closed-source OpenAI Sora model has narrowed from 4.52% to just 0.69%, achieving near-complete performance alignment. Open-Sora 2.0's benchmark score surpasses Tencent's HunyuanVideo, achieving higher performance levels with lower training costs and setting a new milestone in the open-source video generation field.

Open-Sora 2.0 project address

• GitHub repository:https://github.com/hpcaitech/Open-Sora
• Technical Papers:https://github.com/hpcaitech/Open-Sora-Demo/blob/main/paper/Open_Sora_2

The generated result of Open-Sora 2.0

• Prompt wordsA tomato surfing on a piece of lettuce down a waterfall of ranch dressing, with exaggerated surfing moves and creamy wave effects to highlight the 3D animated fun.

• Prompt wordsA drone camera circles a historic church on a rocky outcrop along the Amalfi Coast, highlighting its stunning architecture, tiered patios, and the dramatic coastal views with waves crashing below and people enjoying the scene in the warm afternoon light.

• Prompt wordsA scene from a disaster movie.

• Prompt wordsA realistic depiction of ancient Chinese style. A young woman, dressed in an embroidered red qipao, walks along the ancient streets of a bustling Chinese town. The red lanterns hanging above her sway gently in the evening breeze, and her calm, confident stride contrasts sharply with the lively atmosphere of merchants and performers around her.

How to use Open-Sora 2.0

• Install from source code:
  • Create a virtual environment (Conda is recommended).:

conda create -n opensora python=3.9
conda activate opensora

git clone https://github.com/hpcaitech/Open-Sora
cd Open-Sora

pip install
-r requirements/requirements-cu121.txt

• • • Installation Project:

pip install
-v
.
# 或使用开发模式：pip install -v -e .

pip install git+https://github.com/hpcaitech/TensorNVMe.git
pip install git+https://github.com/hpcaitech/ColossalAI.git
pip install packaging ninja
pip install flash-attn --no-build-isolation
pip install
-v --disable-pip-version-check --no-cache-dir --no-build-isolation --config-settings "--build-option=--cpp_ext" --config-settings "--build-option=--cuda_ext" git+https://github.com/NVIDIA/apex.git

docker build -t opensora .

docker run -ti
--gpus all -v .:/workspace/Open-Sora opensora

Application scenarios of Open-Sora 2.0

• Video productionQuickly generate creative videos such as advertisements and animations, reducing production costs.
• Post-production of film and televisionIt assists in generating special effects shots and virtual scenes, improving production efficiency.
• EducationGenerate educational videos to enhance the fun and effectiveness of teaching.
• Game developmentUsed to generate game animations and virtual scenes, enriching game content.
• VR/AR Applications: To create immersive virtual scenes and enhance the user experience.