LinFusion - National University of Singapore launches image generation model that can generate 16K images per minute with a single GPU.

What is LinFusion?

LinFusion, developed by a research team at the National University of Singapore, is an innovative image generation model based on a linear attention mechanism to handle high-resolution image generation tasks. This keeps the computational complexity linear when processing large numbers of pixels, significantly improving generation efficiency. LinFusion is highly compatible with existing pre-trained model components such as ControlNet and IP-Adapter, supporting zero-shot cross-resolution generation and generating images at previously unseen resolutions. It achieves image generation up to 16K resolution on a single GPU, providing powerful visual content generation capabilities for fields such as art creation, game design, and virtual reality.

LinFusion's main functions

• Text to Image GenerationGenerate a corresponding high-resolution image based on the text description provided by the user.
• High resolution support: Specifically optimized for generating high-resolution images, including resolutions not encountered during training.
• linear complexityThe linear attention mechanism is used to make the model more computationally efficient and consumes fewer resources when processing a large number of pixels.
• Cross-resolution generationGenerate images at different resolutions, including resolutions not seen during training.
• Compatible with pre-trained model componentsIt is compatible with pre-trained Stable Diffusion components such as ControlNet and IP-Adapter, and can be used without additional training.

LinFusion's technical principles

• Linear attention mechanismLinFusion employs a novel linear attention mechanism, unlike the quadratic complexity of self-attention in traditional Transformer-based models. This linear attention mechanism makes the computational complexity of the model linearly related to the number of pixels when processing large numbers of pixels, significantly reducing the demand for computational resources.
• Generalized linear attentionLinFusion introduces a generalized linear attention paradigm, extending existing linear complexity label mixers such as Mamba, Mamba2, and Gated Linear Attention. The generalized linear attention mechanism includes normalized perception and non-causal operations to accommodate the demands of high-resolution visual generation.
• Normalized perceptionThe normalized perception attention mechanism ensures that the sum of the attention weights for each token equals 1, thereby maintaining consistent performance across images at different scales.
• Non-causalityThis is a non-causal version of the linear attention mechanism, allowing the model to access all noisy spatial labels simultaneously during generation, rather than processing labels sequentially as is the case with traditional RNNs. This helps the model better capture the spatial structure of the image.

LinFusion's project address

• Project official websitelv-linfusion.github.io
• GitHub repository:https://github.com/Huage001/LinFusion
• arXiv technical paper:https://arxiv.org/pdf/2409.02097

Application scenarios of LinFusion

• Artistic CreationArtists and designers use LinFusion to generate high-resolution artworks from text descriptions, accelerating the creative process.
• Game developmentIn game design, it enables the rapid generation of game scenes, characters, or concept art, improving the efficiency of game art production.
• Virtual Reality (VR) and Augmented Reality (AR)In the creation of VR or AR content, LinFusion helps generate realistic background images or environments, enhancing the user experience.
• Film and video productionFilmmakers use LinFusion to generate concept art or special effects backgrounds for their movies, reducing pre-production time.
• Advertising and MarketingMarketing teams use LinFusion to quickly generate engaging ad images and social media posts, enhancing the appeal of their marketing content.