StreamingT2V - A model from PicsArt that can generate videos up to 2 minutes long.

What is StreamingT2V?

StreamingT2V, developed by the PicsArt AI research team, is a text-to-video generation model designed to address the challenges of existing models, which can only generate high-quality short videos of 16 or 24 frames. When generating longer videos, they encounter problems such as degraded video quality, inconsistent scene transitions, and video lag. StreamingT2V achieves smooth generation of long videos (up to 1200 frames and 2 minutes in length) by introducing a Conditional Attention Module (CAM), an Appearance Preservation Module (APM), and a stochastic mixing method, ensuring temporal coherence and close alignment with the text description. This method not only improves video quality but also enriches and enhances the video content, representing a significant advancement in the field of long video generation.

StreamingT2V official website entry

• Official project homepage:https://streamingt2v.github.io/
• GitHub repository:https://github.com/Picsart-AI-Research/StreamingT2V(Model and source code are pending release)
• arXiv research paper:https://arxiv.org/abs/2403.14773

Features of StreamingT2V

• Long video generationStreamingT2V can generate long videos (80, 240, 600, 1200 frames or more) based on text descriptions, far exceeding the length of short videos typically generated by traditional models.
• Temporal coherenceThe generated video frames have smooth transitions and consistency, avoiding the hard switching or discontinuity that is common when generating long videos.
• High-quality image framesThis model prioritizes frame-level image quality, ensuring that each frame remains clear and detailed, even in long videos.
• Text alignmentStreamingT2V generates videos that are closely aligned with the input text prompts, ensuring that the video content is consistent with the user's text instructions.
• Video EnhancementBy using a random mixing method, StreamingT2V can enhance the quality of generated videos without introducing inter-block inconsistencies, thereby improving video resolution and visual effects.

StreamingT2V Workflow

The StreamingT2V workflow can be divided into the following main stages:

1. Initialization Stage:
   • In this stage, a pre-trained text-to-video model (such as Modelscope) is first used to synthesize an initial video chunk, typically a short 16-frame video sequence.
2. Streaming T2V Stage:
   • Next, the model enters the autoregressive long video generation process. In this stage, StreamingT2V uses a Conditional Attention Module (CAM) and an Appearance Preservation Module (APM) to generate subsequent frames of the long video.
   • CAM utilizes short-term memory and focuses on the features of the previous video block through an attention mechanism, thereby achieving a smooth transition between blocks.
   • APM utilizes long-term memory to extract key visual features from the initial video chunks, ensuring consistency of scenes and objects throughout the video generation process.
3. Streaming Refinement Stage:
   • After generating sufficiently long videos (e.g., 80, 240, 600, 1200 frames or more), the model enters the refinement stage.
   • In this stage, the generated video is augmented using a high-resolution text-to-video model (e.g., MS-Vid2Vid-XL) through autoregression.
   • By using a random mixing method, 24 consecutive video blocks are enhanced while maintaining smooth transitions between blocks, thereby improving the overall quality and resolution of the video.