CogVideoX-2 - A text-to-video generation model launched by Zhipu AI.

What is CogVideoX-2?

CogVideoX-2 is a text-to-video generation model launched by Zhipu AI. Based on an advanced 3D variational autoencoder (VAE), it compresses video data to 2% of its original size, reducing resource usage while ensuring smooth continuity between video frames. Through unique 3D rotational position encoding technology, the video can flow naturally along the timeline, giving the image a sense of life. The model structure, training methods, and data engineering have been completely updated, significantly improving the basic capabilities of the image-to-video model by 38%. Generation is more controllable, supporting large movements of the main subject while maintaining image stability. Its instruction compliance capability is industry-leading, able to understand and implement various complex prompts. It can handle various artistic styles, greatly enhancing the aesthetics of the image. It supports multiple inference accuracies such as FP16, BF16, FP32, FP8, and INT8.

Main functions of CogVideoX-2

• Text to video generationCogVideoX-2 can generate high-quality video content based on user-input text descriptions, supporting video output up to 6 seconds long, 8 frames per second, and a resolution of 720×480.
• Image and videoThis feature can convert user-provided static images into dynamic videos. For best results, it is recommended to upload images with a 3:2 aspect ratio.
• High-efficiency video memory utilizationThe model requires only 18GB of video memory for inference at FP16 precision, making it suitable for running on resource-constrained devices.
• Multi-inference precision supportIt supports multiple inference precisions such as FP16, BF16, and INT8, allowing users to select the appropriate precision based on their hardware conditions to optimize performance.
• Flexible secondary developmentThe model has a simple design, is easy to develop and customize, and is suitable for developers of different levels.
• High-quality video generationCogVideoX-2 is able to generate coherent and high-quality video through a 3D variational autoencoder (3D VAE) and an expert Transformer architecture.
• Low-threshold promptsUsers can use simple text descriptions as input, and the model can understand and generate corresponding video content.

Technical Principles of CogVideoX-2

• 3D Variational Autoencoder (3D VAE)CogVideoX-2 employs 3D VAE technology, which compresses both the spatial and temporal dimensions of video through three-dimensional convolution, reducing video data to 2% of its original size and significantly reducing the consumption of computing resources.
• Expert Transformer ArchitectureThe model incorporates an expert Transformer architecture, capable of deeply analyzing encoded video data and combining it with text input to generate high-quality, story-driven video content. The architecture utilizes 3D Full Attention to achieve spatiotemporal attention modeling, optimizing the alignment between text and video.
• 3D Rotational Position Encoding (3D RoPE)To better capture the spatiotemporal relationships between video frames, CogVideoX-2 uses 3D RoPE technology to encode the rotational positions of the temporal and spatial coordinates, thereby improving the model's ability to model in the temporal dimension.
• High-quality data-drivenZhipu AI has developed an efficient video data filtering method that eliminates low-quality videos, ensuring high standards and purity of training data. It has also constructed a pipeline for generating image captions to video captions, addressing the common problem of insufficient detailed text descriptions in video data.
• Hybrid training strategyCogVideoX-2 employs strategies such as mixed image and video training, progressive resolution training, and fine-tuning with high-quality data to further enhance the model's generative capabilities and coherence.

CogVideoX-2 project address

• Project official websiteBigModel

Application scenarios of CogVideoX-2

• Film and television creationFilmmakers can use CogVideoX-2 to quickly transform script concepts into visual presentations, intuitively assessing the rationality of plot developments and scene settings.
• Advertising and MarketingBrands and advertising agencies can use CogVideoX-2 to directly generate advertising videos in various styles based on the copy, saving production costs while increasing creative flexibility.
• Education and TrainingEducators can use models to create vivid teaching videos in batches, helping students better understand and master knowledge.
• Social media and short video productionSocial media bloggers and short video creators can quickly transform written ideas into engaging video content that attracts followers.