project
Qihoo-T2X - A high-efficiency multimodal generative model open-sourced by 360 AI Research Institute and Sun Yat-sen University
Qihoo-T2X is a high-efficiency multimodal generative model based on Proxy Label Diffusion Transformer (PT-DiT), developed by 360 AI Research Institute and Sun Yat-sen University. Qihoo-T2X introduces a sparse proxy label attention mechanism, significantly reducing...
What is QIHOO-T2X?
Qihoo-T2X is a high-efficiency multimodal generative model based on surrogate-labeled diffusion Transformer (PT-DiT), developed by 360 AI Research Institute and Sun Yat-sen University. Qihoo-T2X introduces a sparse surrogate labeling attention mechanism, significantly reducing the redundancy in global self-attention computation of traditional diffusion Transformer, and enhances detail modeling capabilities by combining window attention and shifted window attention. Qihoo-T2X supports various tasks, including text-to-image (T2I), text-to-video (T2V), and text-to-multiview (T2MV) generation.
Main functions of Qihoo-T2X
- Text to ImageIt generates high-quality, high-resolution images based on the input text description, producing image content that is highly consistent with the text description. It is suitable for creative design, art generation, and other fields.
- Text to VideoIt generates coherent video content based on text descriptions, supports the generation of dynamic scenes and video sequences, and is suitable for video creation, animation production and other scenarios.
- Text to multiple viewsGenerates multi-view images of the same object or scene based on text descriptions, suitable for multi-view display of 3D objects, and supports virtual reality (VR) and augmented reality (AR) applications.
- High-efficiency generationBased on an optimized agent-based tokenized attention mechanism, it significantly reduces computational complexity, supports efficient generation of high-resolution images and long videos, and reduces training and inference costs.
Technical Principles of Qihoo-T2X
- Proxy-based attentionTraditional diffusion-based Transformers use a global self-attention mechanism, which is computationally complex and redundant. PT-DiT (the core architecture of Qihoo-T2X) calculates the average token within each spatiotemporal window as proxy tokens, and performs self-attention calculations on the proxy tokens, significantly reducing computational cost. Global semantic information between proxy tokens is injected into all potential tokens through cross-attention, ensuring the effective propagation of global information.
- Window attention and shifted window attentionTo enhance local detail modeling capabilities, PT-DiT introduces a window attention mechanism, performing self-attention calculations on markers within a local window. This shifted window attention mechanism avoids the "mesh effect" caused by window partitioning, further improving generation quality.
- Efficient computation of sparse proxy tagsBased on a sparse proxy tagging mechanism, PT-DiT significantly reduces computational complexity while maintaining the quality of generated content when processing high-resolution images and long videos.
- Multi-tasking adaptabilityPT-DiT's architecture is designed to seamlessly adapt to various tasks such as image generation, video generation, and multi-view generation without requiring major adjustments to the model structure.
Qihoo-T2X project address
- Project official website:https://360cvgroup.github.io/Qihoo-T2X/
- GitHub repository:https://github.com/360CVGroup/Qihoo-T2X
- arXiv technical paper:https://arxiv.org/pdf/2409.04005
Application scenarios of Qihoo-T2X
- Creative Design and Artistic CreationIt can quickly generate high-quality artistic images based on text descriptions, supports multiple styles, accelerates the design process, and provides inspiration for artists.
- Video content generationGenerates coherent animated videos suitable for advertising, promotion, and animation production, reducing video creation costs and time.
- Education and TrainingGenerate teaching images and videos to help students understand complex concepts and support virtual labs and dynamic teaching resources.
- Entertainment and Game DevelopmentGenerate virtual scenes, characters, and dynamic content for use in game development, virtual reality (VR), and augmented reality (AR) applications to enhance immersion.
- Advertising and MarketingQuickly generate personalized advertising images and videos to improve marketing effectiveness and meet brand visual content needs.