Step-Video-TI2V - Step-Star's open-source graph-based video model

What is Step-Video-TI2V?

Step-Video-TI2V is an open-source image-to-video generation model from StepFun. It boasts 30 billion parameters and can generate videos up to 102 frames long based on text descriptions and image input. The model is based on a deep compression variational autoencoder (Video-VAE), achieving 16×16 spatial compression and 8× temporal compression, significantly improving training and inference efficiency. Users can balance the dynamism and stability of the video by setting a motion score. It supports various camera movements such as push, pull, pan, tilt, rotate, and follow.

Main functions of Step-Video-TI2V

• Image-generated videoUsers can provide an image and a related text description, and the model will generate a coherent video based on these inputs.
• High-quality video outputIt supports generating videos with up to 102 frames, 5 seconds, and 540P resolution, which can meet a variety of creative needs.
• Dynamic adjustmentUsers can control the dynamism of a video by setting a motion score. For example, a motion score of 2 results in a more stable but less dynamic video, while a motion score of 10 or 20 results in a more dynamic video.
• Balancing dynamics and stabilityBy adjusting the motion score, users can find the optimal balance between dynamic effects and stability.
• Camera motion controlIt supports multiple camera movement modes, including fixed lens, up/down/left/right movement, up/down/left/right panning, zooming in/out, rotation, surround, and focus shifting.
• Cinematic camera workIt can generate complex camera movements similar to those in movies, meeting the needs of professional creation.
• Anime effect optimizationIt excels in generating anime-style videos, producing videos with effects such as blurred backgrounds and dynamic movements. Suitable for applications such as animation creation and short video production.
• Flexible video sizesIt supports video generation in multiple sizes, including landscape, portrait, and square, allowing users to choose the appropriate video size based on their creative needs and platform characteristics.
• Multilingual supportEquipped with a bilingual text encoder, it supports input prompts in both Chinese and English, making it convenient for users with different language backgrounds.
• Special effects generation capabilityIt has initially acquired the ability to generate special effects, and will further improve the quality of special effects generation through technical optimization in the future.

The technical principle of Step-Video-TI2V

• Deeply compressed variational autoencoder (Video-VAE)Step-Video-TI2V utilizes a deep compression variational autoencoder (Video-VAE) to achieve 16×16 spatial compression and 8× temporal compression. This significantly reduces the computational complexity of video generation tasks while maintaining excellent video reconstruction quality. The Video-VAE employs a dual-path architecture, effectively separating high- and low-frequency information to further optimize video generation results.
• Diffusion-based Transformer (DiT) architectureThe model is based on a diffusion-based Transformer (DiT) architecture, incorporating a 3D full attention mechanism. Through a Flow Matching training method, input noise is progressively denoised into latent frames, with text embeddings and time steps used as conditional factors. This architecture excels in generating videos with strong motion dynamics and high aesthetic quality.
• Bilingual text encoderStep-Video-TI2V is equipped with a bilingual text encoder, capable of handling both Chinese and English prompts. This allows the model to directly understand Chinese or English input and generate videos that match the text descriptions.
• Direct Preference Optimization (DPO)To further improve the quality of generated videos, Step-Video-TI2V introduces the Video Direct Preference Optimization (Video-DPO) method. DPO fine-tunes the model using human preference data, reducing artifacts and enhancing visual effects, resulting in smoother and more realistic generated videos.
• Cascaded training strategyThe model employs a cascaded training process, including text-to-image (T2I) pre-training, text-to-video/image (T2VI) pre-training, text-to-video (T2V) fine-tuning, and direct preference optimization (DPO) training. This accelerates model convergence and makes full use of video data of varying qualities.
• System optimizationStep-Video-TI2V has undergone system-level optimizations, including tensor parallelism, sequence parallelism, and Zero1 optimization, to achieve efficient distributed training. It introduces the high-performance communication framework StepRPC and the two-layer monitoring system StepTelemetry to optimize data transmission efficiency and identify performance bottlenecks.

Step-Video-TI2V project address

• Github repository:https://github.com/stepfun-ai/Step-Video-TI2V
• HuggingFace model library:https://huggingface.co/stepfun-ai/stepvideo-ti2v
• arXiv technical paper:https://arxiv.org/pdf/2503.11251

How to use Step-Video-TI2V

• Visit Yuewen VideoVisit the official website or app of Yuewen Video.
• Operating stepsClick to upload an image and enter a text description. Adjust parameters (such as exercise score). Click the generate button to download or share the video.

Application scenarios of Step-Video-TI2V

• Animation ProductionStep-Video-TI2V excels at generating anime-style videos, producing smooth animations based on input images and text descriptions.
• Short video productionThe model supports various camera movement methods, such as push, pull, pan, tilt, rotate, and surround, and can generate short videos with cinematic effects.
• Movement teachingStep-Video-TI2V can generate complex dynamic scenes, such as sports movement instruction and dance instruction.
• Special effects productionThe model can generate aesthetically pleasing and realistic videos, suitable for special effects production in movies, TV series, and games.
• Product ShowcaseStep-Video-TI2V can generate engaging advertising videos that showcase product features or brand stories.