project
ExVideo - AI-powered video length extension and optimization technology launched by Alibaba and East China Normal University
ExVideo, developed by researchers from Alibaba and East China Normal University, is a post-tuning technique for video synthesis models. It extends the temporal scale of existing video synthesis models to generate longer videos. The team is based on Stable V...
What is ExVideo?
ExVideo, developed by researchers from Alibaba and East China Normal University, is a post-tuning technique for video synthesis models. It extends the temporal scale of existing video synthesis models to generate longer videos with more frames. Based on the Stable Video Diffusion model, the team trained an extended model capable of generating coherent videos up to 128 frames long, while retaining the original model's generative capabilities. ExVideo optimizes temporal modules such as 3D convolution, temporal attention, and positional embedding, enabling the model to handle content with longer time spans. It significantly increases the number of video frames while maintaining the original model's generative capabilities, and its training cost is relatively low, making it particularly suitable for scenarios with limited computational resources.
ExVideo's features
- Time Scale ExpansionOne of ExVideo's core functionalities is its ability to extend the temporal scale of video synthesis models, enabling it to process and generate video sequences that are longer than originally designed. This extension allows ExVideo to generate videos with more frames, telling more complete stories or showcasing longer dynamic scenes.
- Post-tuning strategyExVideo's post-tuning strategy is a key part of its technology. By retraining specific parts of models such as Stable Video Diffusion, ExVideo enables these models to generate longer videos, reaching 128 frames or more. This not only increases the video length but also maintains the model's generalization ability to various inputs, resulting in diverse and highly adaptable generated videos.
- High efficiency of parametersCompared to traditional training methods, ExVideo employs a post-tuning strategy, eliminating the need to train a completely new model from scratch. Instead, it optimizes existing models, significantly reducing the number of parameters and computational resources required, making model expansion more efficient and practical.
- Maintain generation capabilityWhile extending the video length, ExVideo focuses on maintaining video quality, resulting in videos that not only have increased length but also meet high standards in visual coherence, clarity, and overall quality.
- Compatibility and universalityExVideo is designed with compatibility with various video synthesis models in mind, enabling its wide application in different video generation tasks. Whether it's 3D convolution, temporal attention, or positional embedding, ExVideo provides corresponding extension strategies to adapt to different model architectures.
ExVideo's official website entrance
- Official project homepage:https://ecnu-cilab.github.io/ExVideoProjectPage/
- GitHub repository:https://github.com/modelscope/DiffSynth-Studio
- Hugging Face model download:https://huggingface.co/ECNU-CILab/ExVideo-SVD-128f-v1
- ModelScope model download:https://www.modelscope.cn/models/ECNU-CILab/ExVideo-SVD-128f-v1/summary
- arXiv technical paper:https://arxiv.org/abs/2406.14130
ExVideo's technical principles
- Post-tuning of parametersExVideo employs a post-parameter tuning approach to improve existing video synthesis models. This involves retraining specific parts of the model rather than retraining the entire model, thereby improving efficiency.
- Time module extensionFor the temporal module in video synthesis models, ExVideo proposes extension strategies. These strategies include optimizations to 3D convolutional layers, temporal attention mechanisms, and location embedding layers to accommodate longer video sequences.
- 3D convolutional layers3D convolutional layers are used in video synthesis to capture features along the temporal dimension. ExVideo preserves the 3D convolutional layers in the original model because they are able to adapt to different time scales without additional fine-tuning.
- Time attention mechanismTo improve the model's ability to handle long-term sequences, ExVideo fine-tuned its temporal attention module. This helps the model better understand the temporal coherence of video content.
- Position embeddingTraditional video synthesis models may use static or trainable positional embeddings to represent the frame order in a video. ExVideo adapts to longer video sequences by introducing trainable positional embeddings and initializing them with a loop pattern.
- Identity 3D ConvolutionFollowing the location embedding layer, ExVideo introduces an additional identity 3D convolutional layer to learn long-term video features. This layer is initialized to an identity matrix before training to ensure that the video representation is not altered and to maintain consistency with the original model.
- Engineering optimizationTo enable effective training with limited computing resources, ExVideo employs various engineering optimization techniques, such as parameter freezing, mixed-precision training, gradient checkpointing, and Flash Attention, as well as using the DeepSpeed library to slice optimizer states and gradients.
- Training processExVideo was trained using the publicly available dataset OpenSoraPlan2, which contains a large number of videos, to enhance the model's ability to generate diverse videos.
- Loss function and noise schedulingDuring training, ExVideo maintained the same loss function and noise scheduling strategy as the original model, ensuring the stability and efficiency of model training.