AB
AB
AiBoss

All Categories

project

ICEdit - An instruction-based image editing framework jointly developed by Zhejiang University and Harvard University

ICEdit (In-Context Edit) is an instruction-based image editing framework developed by Zhejiang University and Harvard University. It leverages the powerful generation capabilities and context-awareness of the Massive Diffusion Transformer...

What is ICEdit?

ICEdit (In-Context Edit) is an instruction-based image editing framework developed by Zhejiang University and Harvard University. Leveraging the powerful generation and context-aware capabilities of the Massive Diffusion Transformer, it enables precise image editing using natural language commands. ICEdit requires only 0.1% of the training data and 1% of the trainable parameters, significantly reducing resource demands compared to traditional methods. It performs exceptionally well in multi-round and multi-task editing. ICEdit boasts advantages such as being open-source, low-cost, and fast (approximately 9 seconds per image), making it suitable for a wide range of applications.

ICEdit's main functions

  • Instruction-driven image editingIt enables precise modification of images based on natural language commands, such as changing the background, adding text, and changing people's clothing.
  • Multiple rounds of editingIt supports multiple consecutive edits, with each edit based on the previous result, making it suitable for complex creative needs.
  • Style conversionIt supports converting images into different art styles, such as watercolor paintings and comics.
  • Object replacement and additionReplace objects in an image or add new elements, such as replacing a person with a cartoon character.
  • High-efficiency processingIt has a fast processing speed (about 9 seconds per image), making it suitable for rapid generation and iteration.

ICEdit's technical principles

  • In-Context Editing FrameworkBased on "in-context prompting," editing instructions are embedded into generation prompts, and the model generates the edited image based on the processed prompts. This eliminates the need to modify the model architecture, avoiding the complex structural adjustments required in traditional methods. Based on contextual understanding, the model directly generates edited results that conform to the instructions.
  • LoRA-MoE Hybrid Fine-Tuning StrategyThis approach combines a parameter-efficient LoRA (Low-Rank Adaptation) adapter with a dynamic expert routing (Mixture-of-Experts, MoE) system. LoRA, based on low-rank matrix factorization, efficiently adjusts model parameters to adapt to different editing tasks. MoE, by dynamically selecting the most suitable expert module for the current task, further enhances editing quality and flexibility. It requires only a small amount of data (50K samples) for fine-tuning, significantly improving the editing success rate.
  • Early Filter Inference-Time ScalingDuring the inference phase, the visual language model (VLM) is used to evaluate the noise samples generated earlier and select the initial noise that best matches the editing instructions. The editing effect is evaluated in a few steps (e.g., 4 steps), quickly selecting the optimal initial noise to further improve editing quality.

ICEdit project address

Application Scenarios of ICEdit

  • Creative DesignTransform photos into artistic styles (such as watercolor paintings) or add creative elements for use in design and advertising.
  • Film and television productionQuickly generate character designs or scene concept art to assist in the early stages of film and television development.
  • social mediaEdit personal photos (such as changing backgrounds and adding effects) to create engaging social media content.
  • EducationGenerate teaching images, such as converting historical figures into cartoon style to aid teaching.
  • CommercialsQuickly create product promotional images, such as changing backgrounds and adding brand logos.
Back