AtomoVideo - A high-fidelity image-to-video generation framework launched by Alibaba.

What is AtomoVideo?

AtomoVideo is a high-fidelity image-to-video (I2V) generation framework proposed by Alibaba's research team. It aims to generate high-quality video content from input static images. Based on multi-granularity image injection, high-quality datasets, and training strategies, this framework maintains high fidelity between the generated video and a given reference image, while achieving rich motion intensity and good temporal consistency.

Compared to Runway Gen-2 and Pika 1.0, AtomoVideo demonstrates certain advantages in preserving image details, generating dynamic videos, and providing personalized and controllable generation.

AtomoVideo's official website entrance

• Official project homepage:https://atomo-video.github.io/
• arXiv research paper:https://arxiv.org/abs/2403.01800

AtomoVideo Features

• High-fidelity image to video generationAtomoVideo can generate video content that is highly consistent with the static image input by the user. The generated video is highly similar to the original image in style, content and detail.
• Video frame predictionThis framework supports the generation of long video sequences. By iteratively predicting subsequent frames, it can generate longer video content from a series of initial frames.
• Time consistency and stabilityAtomoVideo prioritizes temporal continuity and stability when generating videos, ensuring smooth playback without abrupt transitions or disjointed visuals.
• Text to video generationBy combining advanced text-to-image models, AtomoVideo can also generate text-to-video content, allowing users to guide the creation of video content through text descriptions.
• Personalized and controllable generationBy combining personalized text-based image models and controllable generation models, AtomoVideo is able to generate customized video content based on users' specific needs.

How AtomoVideo works

1. Overall processAtomoVideo uses a pre-trained text-to-image (T2I) model as a base and adds new one-dimensional temporal convolutional and temporal attention modules after each spatial convolutional and attention layer. During training, only the parameters of the added temporal and input layers are updated, while the parameters of the T2I model remain fixed.
2. Image information injectionTo maintain consistency with the input image in the generated video, AtomoVideo injects image information at two distinct locations. First, the input image is encoded into a low-level representation using a VAE encoder, which is then combined with Gaussian noise. Simultaneously, a high-level semantic representation of the image is extracted using a CLIP image encoder and injected into the generation process via a cross-attention layer.
   
3. Video frame predictionTo generate long videos, AtomoVideo employs an iterative prediction method. Given a series of initial video frames, the model predicts the next frames. This approach allows for the generation of long video sequences under limited GPU memory constraints.
4. Training and reasoningDuring training, AtomoVideo uses its internal 15M dataset, where each video is approximately 10-30 seconds long, and the text descriptions of the videos are also input into the model. Zero terminal signal-to-noise ratio (SNR) and v-prediction strategies are employed during training to improve the stability of video generation. The model's input size is 512×512, containing 24 frames. During inference, the model performs classifier-free guidance, combined with image and text conditional injection, to improve the stability of the generated output.
5. Personalized video generationAtomoVideo fixes the parameters of the base 2D UNet during training, training only the added parameters, thus allowing it to be combined with popular custom models in the community. For example, it can be combined with T2I models like epiCRealism, which excels at generating light and shadow effects and tends to produce videos with light elements when used for I2V generation.