DiTCtrl - A multi-prompt video generation method jointly launched by CUHK, Tencent, and other organizations.
DiTCtrl is a multi-cue video generation method based on the Multimodal Diffusion Transformer (MM-DiT) architecture, jointly developed by the Chinese University of Hong Kong, Tencent, and other institutions. DiTCtrl can generate multiple cues without additional training...
What is DiTCtrl?
DiTCtrl is a multi-cue video generation method based on the Multimodal Diffusion Transformer (MM-DiT) architecture, jointly developed by the Chinese University of Hong Kong, Tencent, and other institutions. DiTCtrl can achieve coherent video generation between multiple text cues without additional training, maintaining consistency in content and motion. Based on the analysis of the attention mechanism of MM-DiT, DiTCtrl employs key-value sharing and latent blending strategies to achieve smooth transitions between different cues, improving the coherence and naturalness of video generation. On the new benchmark MPVBench, DiTCtrl achieves state-of-the-art performance while maintaining computational efficiency, marking a significant advancement in multi-cue video generation technology.
DiTCtrl's main functions
- Multi-prompt video generationDiTCtrl can handle multiple text prompts and generate coherent video content, which is especially important for reflecting dynamic, multi-action scenes in the real world.
- No additional training requiredUnlike traditional video generation models, DiTCtrl can generate videos without additional training data or computational resources, achieving zero-shot multi-cue video generation.
- Smooth transitionDiTCtrl enables smooth transitions between different video clips, maintaining the continuity and visual consistency of the video content.
- Precise semantic controlThe system analyzes and utilizes MM-DiT-based attention mechanisms to precisely control video content, ensuring that the generated video closely corresponds to the text prompts.
The technical principles of DiTCtrl
- MM-DiT architectureBased on the Multimodal Diffusion Transformer (MM-DiT) architecture, text and video are mapped to a unified sequence for attention calculation, effectively processing multimodal data.
- Attention mechanism analysisThis study is the first to analyze the 3D full attention mechanism of MM-DiT and finds that it is similar to the cross/self attention module in the UNet-like diffusion model, providing a basis for precise semantic control.
- KV sharing mechanismIt adopts a key-value (KV) sharing mechanism to support the sharing of attention information among video segments with different prompts, thus maintaining the semantic consistency of key objects.
- Potential hybrid strategiesTo achieve smooth transitions between video segments, DiTCtrl uses a latent blending strategy, which applies a position-dependent weighting function to the overlapping area of adjacent video segments to achieve smooth transitions between different semantic segments.
- Mask-guided attention fusionExtract the attention mask of a specific object, and use the mask to guide attention fusion to generate attention features for new cues, thereby achieving consistency between different cues.
- MPVBench benchmarkTo evaluate the performance of multi-cue video generation, the MPVBench benchmark is introduced, which includes various transition types and specialized evaluation metrics.
DiTCtrl's project address
- Project official website:onevfall.github.io/project_page/ditctrl
- GitHub repository:https://github.com/TencentARC/DiTCtrl
- arXiv technical paper:https://arxiv.org/pdf/2412.18597
Application scenarios of DiTCtrl
- Film and video productionUsed to generate movie trailers, special effects scenes, or animated videos, reducing actual shooting costs and time.
- Game developmentIn game development, dynamic background videos or game story animations can be generated to enhance the game's immersion and storytelling.
- Advertising industryQuickly generate engaging video content based on advertising copy, enhancing the creativity and appeal of your ads.
- Social media content creationUsers generate personalized video content for use on social media platforms such as TikTok and Instagram.
- News and reportsCreate dynamic charts or event reenactments in news reports to make information delivery more intuitive and vivid.