Inf-DiT - An ultra-high resolution image generation model jointly launched by Tsinghua University and Zhipu AI

What is Inf-DiT?

Inf-DiT, developed by Tsinghua University and Zhipu AI, is an image upsampling method based on a diffusion model, capable of generating ultra-high-resolution images. Inf-DiT introduces a unidirectional block attention mechanism (UniBA), reducing the space complexity of the generation process from O(N^2) to O(N), effectively solving the problem of excessive memory consumption in traditional diffusion models when generating large images. Using a Diffusion Transformer (DiT) structure, Inf-DiT can handle image upsampling tasks of various shapes and resolutions. Inf-DiT employs multiple techniques to enhance the local and global consistency of images, such as global image embedding and cross-attention mechanisms using neighboring low-resolution blocks, further improving the quality and consistency of the generated images. Experimental results show that Inf-DiT achieves state-of-the-art (SOTA) performance in both ultra-high-resolution image generation and super-resolution tasks.

Main functions of Inf-DiT

• Ultra-high resolution image generationGenerates ultra-high resolution images, breaking through the memory limitations of traditional diffusion models in high-resolution image generation, and is suitable for practical application scenarios such as complex designs, advertisements, posters and wallpapers that require fine details and rich textures.
• Flexible image upsamplingIt handles image upsampling tasks of various shapes and resolutions, providing strong technical support for image quality improvement for different needs.
• Enhanced local and global consistencyThe design incorporates various techniques, such as global image embedding and cross-attention mechanism of neighboring low-resolution blocks, to effectively enhance the local and global consistency of the generated image, ensuring that the generated image meets expectations in both detail and overall structure.
• Zero-sample text control capabilityIt has zero-sample text control capability, which guides and adjusts the generated image based on given text prompts, increasing the diversity and controllability of the generated image.

Inf-DiT technical principle

• Unidirectional Block Attention Mechanism (UniBA)The image is divided into multiple blocks, and these blocks are generated sequentially in batches during each diffusion step. Each batch generates a portion of the blocks simultaneously, and any number of blocks can be generated in parallel, provided memory allows. This mechanism reduces the space complexity of the generation process from O(N^2) to O(N), effectively reducing memory consumption and increasing the maximum resolution of the generated images.
• Diffusion converter (DiT) structureInf-DiT is based on the diffusion transformer structure as its basic architecture. It leverages the advantages of Vision Transformer (ViT) to use the attention mechanism as the main way for image patches to interact, which facilitates the implementation of unidirectional block attention mechanism and improves the performance and scalability of the model.
• Global Image EmbeddingTo enhance the global semantic consistency of generated images, Inf-DiT extracts global image embeddings from low-resolution images based on a pre-trained CLIP model and adds them to the temporal embeddings of the diffusion transformer, allowing the model to learn directly from high-level semantic information.
• Cross-attention mechanism of neighboring low-resolution blocksTo reduce the probability of generating discontinuous images when generating high-resolution images, Inf-DiT introduces a cross-attention mechanism for neighboring low-resolution blocks in the first layer of the transformer, allowing each block to perform cross-attention operations on the surrounding 3×3 low-resolution blocks, thereby better capturing neighboring low-resolution information and enhancing local consistency.

Inf-DiT project address

• GitHub repository:https://github.com/THUDM/Inf-DiT
• arXiv technical paper:https://arxiv.org/pdf/2405.04312

Application scenarios of Inf-DiT

• Design and Creative FieldsGenerate high-resolution architectural renderings that showcase architectural details and overall layout, helping clients and designers better understand the design scheme.
• Entertainment and Media IndustryIt improves the resolution and clarity of video images, enhances visual effects, and meets the needs of different playback media.
• Printing and Publishing IndustryUpsample low-resolution book illustrations and cover images to a high resolution suitable for printing to ensure print quality.
• Science and Research FieldTo improve the resolution of medical images and help doctors diagnose and analyze conditions more accurately.