LayerDiffusion - A framework for AI-generated images with transparency.
LayerDiffusion, proposed by researchers from Stanford University, is a technique for generating transparent images using large-scale pre-trained latent diffusion models. It can help users generate single transparent images or multiple transparent layers.
What is LayerDiffusion?
LayerDiffusion (now renamed LayerDiffuse) is a technique proposed by researchers Lvmin Zhang (the author of ControlNet) and Maneesh Agrawala from Stanford University. It utilizes large-scale pre-trained latent diffusion models (such as Stable Diffusion) to generate transparent images, helping users generate single transparent images or multiple transparent layers. The core of this method lies in introducing the concept of "latent transparency," encoding the transparency information of the image's alpha channel into a latent space. This allows models originally used to generate non-transparent images to generate images with transparency.
With LayerDiffusion, users can quickly generate transparent, cutout images without first generating an image and then using AI technologies like Remove.bg for background removal.
LayerDiffusion official website entrance
- GitHub repository:https://github.com/layerdiffusion/LayerDiffusion(The model and source code will be available soon)
- LayerDiffusion SD WebUI version:https://github.com/layerdiffusion/sd-forge-layerdiffusion
- arXiv research paper:https://arxiv.org/abs/2402.17113
LayerDiffusion Features
- Generate transparent imagesLayerDiffusio can generate images with transparency, meaning it can create images with an alpha channel, where the alpha channel defines the transparency of each pixel in the image.
- Generate multiple transparent layersIn addition to a single transparent image, LayerDiffusion can also generate multiple transparent layers. These layers can be generated independently or based on specific conditions (such as foreground or background), and can be blended together to create complex scenes.
- Conditional generationLayerDiffusion supports conditional generation, allowing you to generate foreground or background based on transparent images, providing flexibility for creating images for specific scenes.
- Layer content structure controlUsers can also combine LayerDiffusion with the ControlNet control framework to provide structural control over layer content, guiding the shapes of layers, layouts, elements, and objects.
- Repeatedly iterate and combine layersLayerDiffusion can iteratively combine multiple layers by repeating the background condition foreground model to incrementally build a composite image with any number of transparent layers.
- High-quality image outputBy adding transparency as a latent offset to the latent space of a pre-trained model, LayerDiffusion is able to maintain the high-quality output of the pre-trained model without significantly altering the original latent distribution.
How LayerDiffusion works
- Preparation of potential space:
- First, LayerDiffusion uses the latent space of a pre-trained latent diffusion model (such as Stable Diffusion), which is obtained by encoding RGB images through a variational autoencoder (VAE).
- To support transparency, LayerDiffusion introduces an additional dimension (latent transparency) in the latent space to represent the alpha channel (transparency information) of the image.
- Encoding and Decoding of Potential Transparency:
- LayerDiffusion trains two independent neural networks: a latent transparency encoder and a latent transparency decoder.
- The encoder receives the RGB and alpha channels of the original image and encodes the transparency information into a latent offset, which is added to the representation in the latent space.
- The decoder then extracts transparency information from the adjusted latent representation and decodes it back to the original alpha channel.
- Adjustment of potential space:
- To ensure that the added transparency information does not disrupt the distribution of the original latent space, LayerDiffusion adjusts the latent representation using latent offsets.
- This process involves a "harmlessness" metric, which assesses whether the latent offset impairs the model's reconstruction ability by comparing the decoding results of the original pre-trained model's decoder to the adjusted latent representation.
- Fine-tuning of the diffusion model:
- After introducing transparency information into the latent space, LayerDiffusion fine-tunes the original diffusion model so that it can generate transparent images in the new latent space.
- This process involves training a diffusion model so that it can learn how to preserve transparency information while adding noise.
- Multi-layer generation:
- LayerDiffusion also extends its capabilities to support the generation of multiple transparent layers. This is achieved through a shared attention mechanism and low-rank adaptations (LoRAs), ensuring consistency and harmonious blending between different layers.
- Dataset preparation and training:
- To train the model, researchers collected a large number of transparent images (comprising 1 million transparent images covering a variety of content themes and styles), obtained through a human-participatory collection process. Based on this dataset, the model was trained to generate high-quality transparent images and layers.