DesignEdit - An open-source AI image layering and editing framework from Microsoft and others.

What is DesignEdit?

DesignEdit is an AI image editing framework jointly developed by researchers from Microsoft Research Asia and Peking University. It introduces the layer concept from the design field, employing multi-layer latent decomposition and fusion techniques to achieve accurate spatially aware image editing and processing without additional training. Through a key mask self-attention mechanism and artifact suppression scheme, DesignEdit can flexibly handle various objects in an image and perform complex operations such as moving, resizing, and removing them.

DesignEdit official website entrance

• Official project homepage:https://design-edit.github.io/
• arXiv research paper:https://arxiv.org/abs/2403.14487
• GitHub source code repository:https://github.com/design-edit/DesignEdit
• Hugging Face Demo:https://huggingface.co/spaces/YuhuiYuan/DesignEdit

DesignEdit's main functions

• Object removalDesignEdit can precisely remove specified objects from an image, whether single or multiple objects. Through multi-layered potential decomposition, the framework can process each object independently and naturally restore the background after removal.
• Object movementThe frame allows users to move one or more objects in an image to a new location. Through instruction-guided potential merging, objects can be repositioned on the canvas while maintaining harmony with their surroundings.
• Resizing and Flipping ObjectsDesignEdit allows users to scale and flip objects in an image, enabling them to change the size or orientation of objects without affecting other parts of the image.
• Camera pan and zoomSimulating changes in camera perspective, DesignEdit can achieve panning and zooming effects in images, allowing users to adjust the composition of images as if moving or adjusting the focus when viewing through a camera lens.
• Cross-image combinationDesignEdit allows you to combine elements from different images to create entirely new visuals. This feature is particularly useful for creative work, allowing you to combine elements from multiple images to create new visual content.
• Design Image EditingSpecifically designed for image/poster design, DesignEdit can handle editing tasks for text, decorations, and other design elements. It understands the specific needs of design images, such as typography and style adjustments, providing more precise editing control.

How DesignEdit works

DesignEdit works based on the combination of two core subtasks: Multi-Layered Latent Decomposition and Multi-Layered Latent Fusion.

1. Multilevel potential decomposition:
   • conceptDesignEdit divides the latent representation of the source image into multiple layers, each representing a different object or background part of the image.
   • Key mask self-attentionTo edit specific regions without disrupting other areas of the image, DesignEdit employs a special self-attention mechanism called key-masking self-attention. This mechanism allows the model to ignore or modify pixels within a masked region while preserving the contextual information of the surrounding area.
   • Background RepairAfter removing objects, DesignEdit uses the inherent healing capabilities of its self-attention mechanism to fill in blank areas in the background, ensuring the continuity and natural transition of the image.
2. Multi-layer potential fusion:
   • Command-guided fusionAfter the decomposition process, DesignEdit merges the edited multiple potential representation layers onto a new canvas according to the user's editing instructions. This process is performed according to a specific layer order and the layout arrangement specified by the user.
   • Artifact SuppressionTo improve editing quality, DesignEdit introduces an artifact suppression scheme in the latent space. This scheme helps reduce visual imperfections that may occur during the editing process, making images look more natural and realistic.
   • Harmonization treatmentDuring the fusion process, DesignEdit uses an additional denoising step to coordinate the multi-layered latent representation after fusion, further optimizing the integration of image edges and the smooth transition of the interface.

The entire editing process is training-free, meaning no additional training or fine-tuning is required for specific tasks. DesignEdit utilizes advanced deep learning models, such as GPT-4V, to assist in generating precise editing instructions and layout arrangements, thereby achieving efficient and accurate image editing.