TransPixar - an open-source technology for generating transparent background videos, developed in collaboration with The Chinese University of Hong Kong, Adobe, and other organizations.
TransPixar is an advanced text-to-video generation method jointly open-sourced by the Chinese University of Hong Kong, Adobe Research, the Hong Kong University of Science and Technology, and SmartMoore. It extends pre-trained RGB video models to generate RGBA videos that include transparency information...
What is TransPixar?
TransPixar, an advanced text-to-video generation method jointly open-sourced by the Chinese University of Hong Kong, Adobe Research, the Hong Kong University of Science and Technology, and SmartMoore, extends pre-trained RGB video models to generate RGBA videos with transparency information. Based on a Diffusion Transformer (DiT) architecture, TransPixar achieves joint generation of RGB and alpha channels while maintaining high consistency by introducing alpha-specific tokens and LoRA-based fine-tuning. TransPixar optimizes the attention mechanism, retaining the advantages of the original RGB model, and can generate diverse and highly aligned RGBA videos with limited training data. TransPixar supports the creation of videos containing transparent elements such as smoke and reflections, providing highly realistic visual effects. TransPixar has broad application prospects in entertainment, advertising, and education, offering new possibilities for visual effects (VFX) and interactive content creation.
TransPixar's main functions
- RGBA video generationGenerate videos containing RGB color channels and alpha transparency channels from text descriptions to create complex visual effects.
- Transparency effect processingIt supports generating elements with transparent properties, such as smoke and reflections, which can be seamlessly integrated into the background scene and are suitable for applications such as visual effects (VFX).
- High-quality video generationWhile generating RGBA video, it retains the high quality of the original RGB video generation model, ensuring the clarity and detail of the video.
- Multi-scenario adaptabilityIt is suitable for video generation of various scenes and object types, including human movements, natural landscapes, dynamic effects, etc., and has good generalization ability.
- Text-driven content creationBased on the input text description, it generates matching video content, realizing automated text-to-video creation and improving the efficiency and creativity of content production.
TransPixar's technical principles
- Diffusion converter (DiT) architectureBased on the DiT model, a self-attention mechanism is used to capture long-range dependencies between video frames, enabling fine-grained modeling and generation of video content.
- Alpha channel generationIntroducing an alpha-specific token into the DiT model, which is then combined with the RGB token to generate the alpha channel, supports the output of RGBA video.
- LoRA fine-tuningThe LoRA (Low-rank Adaptation) fine-tuning scheme fine-tunes the projection of the alpha token, maintaining the quality of RGB generation while optimizing the generation of the alpha channel.
- Attention mechanism optimizationThe system analyzes and optimizes the attention mechanism in the RGBA generation process, including Text-attend-to-RGB, RGB-attend-to-Text, and RGB-attend-to-Alpha. Based on adjusting the attention calculation, it achieves strong alignment and high-quality generation between the RGB and alpha channels.
- Dataset Expansion and Training StrategiesTraining on a limited RGBA video dataset, based on reasonable data preprocessing and training strategies, improves the model's adaptability to diverse scenes and object types, and enhances the diversity and consistency of generated content.
TransPixar's project address
- Project official website:https://wileewang.github.io/TransPixar/
- GitHub repository:https://github.com/wileewang/TransPixar
- arXiv technical paper:https://arxiv.org/pdf/2501.03006
- Experience the demo online:https://huggingface.co/spaces/wileewang/TransPixar
Application scenarios of TransPixar
- EntertainmentQuickly generate planetary explosion effects clips to aid in the post-production of science fiction films.
- Advertising: Create advertising videos showcasing the appearance and driving dynamics of new electric vehicles to attract consumer attention.
- EducationGenerate videos of objects undergoing forces and motion to aid in explaining physical laws and improve students' understanding.
- Augmented Reality (AR)): Generates realistic panoramic videos of Paris, providing an immersive experience for VR tourism applications.
- Creative industriesCreate fantasy world videos to expand the expressive forms and creative space of digital art.