Snap Video - Snapchat's AI video generation model

What is Snap Video?

Snap Video is an AI video generation model developed by the research team at Snap (the company behind social media platform Snapchat). Its goal is to synthesize videos from text descriptions; users can input descriptive text, and the model can generate corresponding video content based on that text. Due to the high redundancy and complexity of video content, Snap Video aims to solve the challenge of generating high-quality, temporally coherent, and high-motion-fidelity videos from text descriptions. User research shows that Snap Video is comparable to, and in some cases even surpasses, video generation models such as Pika and Runway Gen-2.

Snap Video's official website

• Official project homepage:https://snap-research.github.io/snapvideo/
• arXiv research paper:https://arxiv.org/abs/2402.14797

Key features of Snap Video

• Video-first modelSnap Video is a model specifically designed for video content generation, with particular optimizations for handling the temporal continuity and motion of video. This differs from many other models, which are often extended from image generation models and may have limitations when processing video.
• Extended EDM frameworkThis model extendsEDM frameworkThis allows it to better handle spatial and temporal redundancy in video data, thereby improving the quality of video generation.
• High-efficiency Transformer architectureSnap Video employs a Transformer-based FIT architecture, which is highly effective in processing sequential data and capturing long-term dependencies, especially in video generation tasks and processing high-resolution video.
• Rapid training and inferenceCompared to traditional architectures such as U-Net, Snap Video trains and inferences faster, and can generate videos more efficiently.
• High-resolution video generationSnap Video is able to generate high-resolution video content, which is a challenge in previous text-to-video generation models because it needs to handle more details and complex motion.
• Joint space-time modelingSnap Video generates videos with complex motion and high temporal consistency by modeling both spatial and temporal dimensions.

Snap Video's technical architecture

• Extended EDM frameworkSnap Video extends the EDM framework to accommodate high-resolution video generation. Originally designed for image generation, the EDM framework generates new data samples by simulating a data diffusion process. Snap Video adjusts the forward diffusion process of EDM by introducing an input scaling factor (σin) to maintain the signal-to-noise ratio (SNR) at the original resolution, thus preserving performance in high-resolution video generation.
• Converter architectureSnap Video employs a Transformer-based architecture, a highly efficient deep learning model particularly well-suited for processing sequential data. The Transformer captures global dependencies in the input data through self-attention and cross-attention mechanisms. In Snap Video, the Transformer is used to process video frames to generate videos with coherent motion.
• FIT architectureSnap Video usesFIT (Far-reaching Interleaved Transformers)The architecture is a transformer architecture specifically designed for high-resolution image and video synthesis. FIT allows for an increase in input dimensionality without significantly impacting performance by learning a compressed representation of the input data and performing computations on a learnable latent space. This compressed representation enables Snap Video to efficiently process high-resolution video data.
  
• Joint space-time modelingSnap Video considers both spatial and temporal dimensions when generating videos, which helps capture dynamic changes and motion within the video. This joint modeling strategy improves the temporal consistency and motion quality of the video.
• Two-stage cascade modelTo generate high-resolution videos, Snap Video employs a two-stage cascaded model. The first stage generates low-resolution videos, while the second stage upsamples these videos to a higher resolution. This cascaded approach allows the model to generate high-quality videos while maintaining performance.
• Training and reasoning processSnap Video uses the LAMB optimizer and cosine learning rate scheduling during training. During inference, the model uses a deterministic sampler to generate video samples from Gaussian noise and applies classifier-free guidance to improve text-video alignment.
• Condition InformationSnap Video utilizes conditional information during the generation process, such as text descriptions, noise levels, frame rates, and resolutions. This information is provided to the model through additional read operations to control the generation process.