Video Alchemist - an AI video generation model with multi-agent open ensemble personalization capabilities.

What is Video Alchemist?

Video Alchemist, developed by companies like Snap, is a novel video generation model with multi-subject, open-set personalization capabilities. It can generate videos based on text prompts and reference images without requiring optimization during testing. The model is based on the Diffusion Transformer module, integrating reference image embedding and subject-level text prompts into the video generation process through dual cross-attention layers. Video Alchemist also introduces an automatic data construction pipeline and various data augmentation techniques to enhance the model's focus on subject identity and avoid the "copy-paste effect." To evaluate its performance, a new video personalization benchmark, MSRVTT-Personalization, is proposed.

Main functions of Video Alchemist

• Personalized video generationIt has built-in multi-subject, open set personalization capabilities, which can simultaneously personalize both the foreground and background objects without requiring optimization during testing.
• Conditional generation based on text prompts and reference imagesGiven a text cue and a set of reference images to conceptualize the entities in the cue, Video Alchemist can generate a corresponding video based on the text and reference images.
• Diffusion Transformer module applicationThe model is built on the new Diffusion Transformer module, which fuses each conditional reference image and its corresponding subject-level text cue through an additional cross-attention layer to generate multi-subject conditions and bind the text description of each subject to its image representation.

The technical principles of Video Alchemist

• Multi-subject open collection personalizationVideo Alchemist features built-in multi-subject, open-collection personalization capabilities, enabling simultaneous personalization of foreground and background objects without the need for optimization during testing. It can handle a variety of novel subject and background concepts without requiring separate optimization for each new subject or background.
• Diffusion Transformer moduleVideo Alchemist is built upon a new Diffusion Transformer module, which fuses each conditional reference image and its corresponding subject-level text cue through an additional cross-attention layer. Specifically, the model achieves multi-subject conditional generation through the following steps:
  • Input processing: Given a text prompt and a set of reference images, the model first encodes these inputs.
  • Cross-attention layer: By using a dual cross-attention layer, reference image embedding and subject-level text cues are integrated into the video generation process, enabling the generated video to naturally retain the subject's identity and background fidelity.
  • Subject-level fusion: A subject-level fusion mechanism is introduced to bind the text description of each subject to its image representation, ensuring the accuracy and consistency of the subjects in the generated video.
• Automated Data Construction Pipeline and Image EnhancementTo address the challenge of collecting reference image and video pairing datasets, Video Alchemist designed a novel automated data construction pipeline that incorporates a wide range of image augmentation techniques to enhance the model's focus on subject identity and avoid the "copy-paste effect."
  • Data collection: Collect main images from multiple frames and perform data augmentation processing.
  • Image enhancement: Through various data augmentation techniques, such as rotation, scaling, and color adjustment, the generalization ability of the model is enhanced and overfitting is reduced.
• MSRVTT-Personalization BenchmarkTo evaluate the performance of Video Alchemist, a new video personalization benchmark, MSRVTT-Personalization, was introduced. This benchmark accurately assesses subject fidelity and supports various personalization scenarios, including conditional modes based on face cropping, single or multiple arbitrary subjects, and combinations of foreground and background objects.

Video Alchemist project address

• Project official website:https://snap-research.github.io/open-set-video-personalization
• arXiv technical paper:https://arxiv.org/pdf/2501.06187

Application scenarios of Video Alchemist

• Short video creationIndividual users can transform creative stories and fantastical scenes into videos, create unique short videos, and share them on social media platforms to showcase their personality.
• Animation ProductionCreators can use Video Alchemist to generate animated characters and backgrounds, quickly creating animated short films without the need for complex animation software and skills.
• Historical eventsTeachers can generate videos of historical events to help students better understand the historical background and the course of events.
• Script SceneProducers and directors can generate preliminary video samples of scripted scenes for team communication and to present project concepts to investors.
• Character ActionsIt can generate character movements and expressions, helping actors and directors better understand the performance requirements of the characters.