Phantom - A consistent video generation framework launched by ByteDance
Phantom is a framework developed by ByteDance's Intelligent Creation Team for Subject-to-Video (S2V) generation. Based on cross-modal alignment technology, and combining text and image cues, it extracts the main subject from a reference image...
What is Phantom?
Phantom is a framework developed by ByteDance's Intelligent Creation Team for Subject-to-Video (S2V) generation. Based on cross-modal alignment technology, it combines text and image cues to extract subject elements from reference images and generate video content consistent with the text description. Phantom redesigns the joint text-image injection model based on existing text-to-video (T2V) and image-to-video (I2V) architectures, learning cross-modal alignment based on text-image-video triple data. The Phantom framework supports single-subject and multi-subject references, emphasizing subject consistency, particularly in human-generated tasks, covering existing identity-preserving video generation tasks and providing enhanced advantages.
Phantom's main functions
- Extracting subject elements from the reference imageIt identifies and extracts the main subject (such as people, animals, objects, etc.) from an image as the core content for generating a video.
- Generate video based on text prompts.Users can control the content and style of videos based on text commands, enabling highly customized video generation.
- Multi-subject video generationIt supports processing multiple subjects simultaneously, generating complex interactive scenarios, such as multi-person interaction and human-pet interaction.
- ID-Preserving): When generating videos, the identity features of the subject (such as face, clothing, etc.) are preserved, which is particularly suitable for scenarios such as virtual try-on and digital human generation.
- High-quality video outputThe generated videos perform exceptionally well in terms of visual effects, subject consistency, and text responsiveness, comparable to existing commercial solutions.
Phantom's technical principles
- Data structure designPhantom constructs a text-image-video triplet data structure to train models to understand the relationships between different modalities. The data is divided into two types: in-paired (image and video subjects are the same) and cross-paired (cross-video matching) to avoid the model simply copying the input image.
- Model ArchitectureBased on existing text-to-video (T2V) and image-to-video (I2V) architectures, a joint text-image injection model is redesigned. The model consists of an input head and a trainable DiT module. The input head is responsible for encoding video, text, and reference images, while the DiT module is responsible for cross-modal alignment and video generation.
- Cross-modal alignmentThe reference image is encoded based on a specific visual encoder (such as VAE and CLIP), then concatenated with video features and text features respectively, and input into the visual and text branches of the DiT module.
- Identity Preservation TechnologyWhen processing identity features such as faces, the similarity between the generated video and the reference image is evaluated based on a facial recognition model (such as ArcFace) to ensure the consistency of the subject's identity.
- Optimization and TrainingThe model is trained on large-scale triplet data to learn how to balance textual and image cues when generating videos. During pre-training, the model inherits the weights of the base model and is further fine-tuned based on cross-modal data to achieve high-quality video generation.
Phantom's project address
- Project official website:https://phantom-video.github.io/Phantom/
- GitHub repository:https://github.com/Phantom-video/Phantom
- HuggingFace model library:https://huggingface.co/bytedance-research/Phantom
- arXiv technical paper:https://arxiv.org/pdf/2502.11079
Phantom's application scenarios
- Virtual try-onGenerates dynamic display videos of clothing to help users preview the effects.
- Digital Human GenerationCreate virtual characters with specific appearances for use in scenarios such as virtual streamers.
- Advertising video productionQuickly generate product advertisements based on images and text, improving production efficiency.
- Film and animationGenerate character animation prototypes to assist in creative validation and reduce production costs.
- Education and TrainingGenerate educational videos featuring scientific experiments and historical scenes to enhance interactivity.