DiffBrush - An image generation and editing framework jointly developed by Beijing University of Posts and Telecommunications, Tsinghua University, and other institutions.

What is DiffBrush?

DiffBrush, developed by Beijing University of Posts and Telecommunications, Tsinghua University, the Artificial Intelligence Research Institute of China Telecom, and Northwestern Polytechnical University, is a training-free image generation and editing framework that allows users to intuitively control image generation based on hand-drawn sketches. DiffBrush uses a pre-trained text-to-image (T2I) model, employing techniques such as color guidance, instance and semantic guidance, and latent space regeneration to precisely control the color, semantics, and instance distribution of the generated image. DiffBrush is compatible with various T2I models (such as Stable Diffusion and SDXL) and supports LoRA style adjustments. Users can simply draw the outlines and colors of instances on the canvas to generate images that meet their needs. DiffBrush overcomes the limitations of traditional T2I models that rely on text prompts, lowering the barrier to AI painting and providing users with a more intuitive and efficient creative method.

DiffBrush's main functions

• User-friendly image generationUsers can control the content of the generated image with hand-drawn sketches, without the need for complex text prompts or technical knowledge.
• Color controlBased on the color information drawn by the user, it precisely controls the color of the corresponding area in the generated image.
• Instances and Semantic Control: Draw the outline of the instance and label its semantic information, controlling the position and semantic attributes of specific objects in the generated image.
• Image editingEditing an existing image allows you to add, modify, or replace objects within it.
• Stylization generationIt supports integration with LoRA (Low-Rank Adaptation) style adjustments to generate images with different artistic styles, such as oil paintings and traditional Chinese paintings.
• Multi-model compatibilityIt is compatible with a variety of T2I models (such as Stable Diffusion, SDXL, Flux, etc.) and has wide applicability.

The technical principles of DiffBrush

• Guiding mechanism of diffusion modelThe diffusion process is based on a pre-trained T2I model (such as Stable Diffusion). The diffusion model progressively denoises, mapping random noise to the real image. DiffBrush modifies the denoising direction in the diffusion process, guiding the generated image to resemble the user's hand-drawn sketch.
• Color guidanceBased on the high similarity between the latent space and color space of the diffusion model, features in the latent space are adjusted to achieve precise control over the colors of the generated image. The color information drawn by the user is encoded into the latent space, and an energy function (such as MSE loss) guides the generated image to match the user's requirements.
• Examples and semantic guidanceThis approach leverages attention mechanisms (such as cross-attention and self-attention) in diffusion models to control instances and semantics. User-drawn instance outlines are used as the supervision target of the attention map. By adjusting the distribution of the attention map, the position and semantics of objects in the generated image are ensured to be consistent with the user's requirements.
• Potential Space RegenerationBased on iterative optimization of the initial noise distribution, which approximates the target distribution of the user's hand-drawn sketch, the potential space is adjusted in the early stages of the diffusion process to further optimize the quality of the generated image.
• User interaction and compatibilityDiffBrush combines user-drawn sketches with text prompts, enabling intuitive interaction through a simple user interface. It supports various pre-trained T2I models, allowing users to adjust hyperparameters such as guidance strength to achieve optimal generation results.

DiffBrush project address

• arXiv technical paper:https://arxiv.org/pdf/2502.20904

Application scenarios of DiffBrush

• Creative PaintingArtists and designers can quickly transform hand-drawn ideas into high-quality images, supporting multiple styles and facilitating the realization of artistic concepts.
• Image editingAdd, replace, or modify content on existing images using simple hand-drawn illustrations; suitable for both casual users and designers.
• Educational toolsUsed in art and design education to help students understand color, composition, and creative expression through hand-drawing.
• Game DesignQuickly generate game scenes, characters, or animation sketches, support stylized output, and facilitate creative iteration.
• Advertising designQuickly generate advertising images from creative sketches to meet clients' requirements for color and layout, thereby improving design efficiency.