D-Edit - A multi-functional image editing framework based on images, text, and masks.

What is D-Edit?

D-Edit is a multi-functional image editing framework based on images and text. It utilizes a pre-trained diffusion model and unique prompts to achieve precise control and editing of specific items within an image. The framework can handle various editing tasks, including image-based editing, text-based editing, mask-based editing, and item removal. D-Edit segments an image into multiple items and associates each item with a unique prompt, achieving decoupled control over the items. Users can achieve diverse editing effects by modifying the prompts, masks, or the association between items and prompts. D-Edit is the first framework capable of item editing based on mask editing and combining image and text editing.

Main functions of D-Edit

• Text-based editingThe user changes the text prompt associated with a specific item, replacing or editing the item in the image.
• Image-based editingIt allows users to replace items in a target image with items from a reference image.
• Mask-based editingUsers can edit the mask of specific items in an image, including moving, resizing, changing the shape, etc., to change the appearance of the items.
• Project removalBased on removing mask and cue pairs associated with the item, specific items are removed from the image, allowing the surrounding area to naturally fill the blank space.
• Multifunctional Image EditorThe above editing functions are implemented within a unified framework, providing flexible and diverse image editing capabilities.

The technical principles of D-Edit

• Project prompt interaction:D-Edit breaks down an image into multiple items and assigns a unique cue to each item. The cue is specifically learned to control the specific item in a pre-trained diffusion model.
• Decoupled cross-attention layerD-Edit introduces a decoupled cross-attention mechanism, which allows the model to process the control flow of each item separately from the corresponding prompts, enabling precise item-level editing.
• Two-step optimization process:
  • Injection promptInject new tokens representing each item into the vocabulary of the text encoder and randomly initialize the token embeddings.
  • Model fine-tuning: Optimize the embedding matrix of the text encoder and the weights of the UNet model to establish the association between items and prompts, so that the model can reconstruct the original image with the given item prompts.
  • Freedom of editing operationsAfter establishing the association between items and hints, D-Edit supports changing the hints, masks, or mappings between items and hints to perform various editing operations.
• Flexibility and control:D-Edit's design provides precise control over specific items during editing while maintaining a natural harmony with the overall image. Based on adjustment prompts and masks, users can achieve a range of editing effects, from minor adjustments to complete replacements.

D-Edit project address

• GitHub repository:https://github.com/collovlabs/d-edit
• arXiv technical paper:https://arxiv.org/pdf/2403.04880
• Experience the demo online:https://huggingface.co/spaces/Collov-Labs/d-edit

Application scenarios of D-Edit

• Digital art creationArtists and designers create new digital artworks by editing specific elements in images to achieve unique visual effects.
• Photo editingRegular users can edit their personal photos, such as changing the background, adjusting the pose, and changing the style of clothing.
• Advertising and MarketingMarketers can change specific elements in advertising images, such as products, logos, or text, to suit different marketing campaigns.
• Fashion and ClothingFashion designers showcase different versions of clothing designs, exploring various design possibilities by changing the color, texture, or style of the garments.
• Architecture and Urban PlanningArchitects and urban planners edit architectural design images to adjust building appearances or urban planning layouts.