CustomVideoX - A personalized video generation framework jointly developed by the University of Science and Technology of China and Zhejiang University, among others.
CustomVideoX is an innovative personalized video generation framework jointly proposed by the University of Science and Technology of China (USTC) and Zhejiang University, among others. It generates high-quality, customized videos using reference images and text descriptions. The framework is based on a video diffusion transformer...
What is CustomVideoX?
CustomVideoX is an innovative personalized video generation framework jointly proposed by the University of Science and Technology of China (USTC) and Zhejiang University, among others. It generates high-quality, customized videos using reference images and text descriptions. Based on the Video Diffusion Transformer, the framework employs zero-shot learning, training only LoRA parameters to extract reference image features, achieving efficient personalized video generation. CustomVideoX's core technologies include: a 3D reference attention mechanism, supporting direct interaction between reference image features and video frames in spatial and temporal dimensions; a Temporally Aware Attention Bias (TAB) strategy, which enhances the temporal coherence of the generated video by dynamically adjusting the influence of reference features; and an Entity Region Aware Enhancement (ERAE) module, which highlights key entity regions through semantic alignment. This addresses the temporal inconsistency and quality degradation issues inherent in traditional methods.
Main functions of CustomVideoX
- Personalized video generationCustomVideoX can generate video content that closely matches the user-provided reference images and text descriptions. It generates videos that conform to the descriptions and preserves the detailed features of the reference images.
- High-fidelity reference image fusionThrough a 3D reference attention mechanism, CustomVideoX allows features of the reference image to interact seamlessly with video frames in both spatial and temporal dimensions. This ensures that the generated video matches the text description and preserves the main features and details of the reference image in each frame.
- Time coherence optimizationCustomVideoX introduces a Time-Aware Attention Bias (TAB) strategy, which optimizes the temporal coherence of generated videos by dynamically adjusting the influence of reference features at different time steps. This avoids the jumps or inconsistencies commonly found in traditional methods.
- Key Area EnhancementThrough the Entity Region Aware Enhancement (ERAE) module, CustomVideoX can identify and enhance key regions in a video that are relevant to the text description.
The technical principles of CustomVideoX
- 3D Reference Attention MechanismThe reference image is encoded using a 3D Causal VAE, which allows features to interact directly with video frames in both spatial and temporal dimensions, ensuring that the generated video retains the main features and details of the reference image in each frame.
- Time-perception attentional bias (TAB)In the denoising process of the diffusion model, the weights of reference features are dynamically adjusted using a parabolic temporal mask. The influence of reference features is reduced in the early stages of denoising, strengthened in the middle stages, and reduced again in the final stages, thus optimizing the temporal coherence and visual quality of the generated video.
- Enhanced Entity Region Awareness (ERAE)This method enhances the semantic alignment of videos by calculating the activation intensity of key entities in the text description and identifying regions in the video that are related to these entities. This highlights the target entities, maintains background diversity, and improves the semantic consistency and naturalness of the generated videos.
- Zero-shot learning and LoRA parameter trainingThis method extracts reference image features by training only a small number of LoRA parameters, avoiding the performance degradation that may result from traditional fine-tuning methods, while preserving the integrity of the pre-trained model and reducing training complexity.
- High-quality datasets and benchmarksResearchers have constructed a high-quality, customized video dataset and proposed the VideoBench benchmarking platform for training and evaluating the generalization ability and performance of models.
CustomVideoX project address
- Project official website:https://xiaobul.github.io/CustomVideoX/
- arXiv technical paper:https://arxiv.org/pdf/2502.06527
Application scenarios of CustomVideoX
- Art and DesignIt converts still images into dynamic videos, aiding in artistic creation and animation design.
- Advertising and MarketingGenerate personalized advertising videos to enhance brand promotion effectiveness.
- Film and EntertainmentIt assists in special effects production and animation creation, enhancing the visual effects of film and television content.
- Education and TrainingGenerate instructional animations to enhance the interactivity and comprehensibility of educational content.
- Game developmentQuickly generate character animations and game cutscenes to improve development efficiency.