SmartEraser - Image object removal technology developed by USTC and Microsoft Research Asia

What is SmartEraser?

SmartEraser is an image editing technology developed by the University of Science and Technology of China and Microsoft Research Asia, specifically designed for removing user-specified objects from images. Based on the innovative "Masked-Region Guidance" paradigm, SmartEraser differs from traditional "mask-and-inpaint" methods by preserving masked regions as guides in the removal process. This allows for more accurate identification and removal of target objects while effectively preserving the surrounding context. Trained on the large-scale, high-quality Syn4Removal dataset, and incorporating mask enhancement techniques and CLIP-based visual guidance, SmartEraser demonstrates superior performance in object removal tasks.

SmartEraser's main functions

• Target object identification and removalAccurately identify target objects specified by the user based on a mask and remove them from the image.
• Context preservationThis method removes the target object while preserving the details and structure of the surrounding environment, ensuring the visual coherence of the image.
• High-quality image generationThe generated image is visually consistent with the original image, without obvious distortion or artifacts.
• robustnessIt is highly robust to masks of different shapes and sizes provided by users and adapts to various input conditions.
• Suitable for complex scenariosEffectively remove target objects in complex scenes, such as images containing multiple objects and complex backgrounds.

SmartEraser's technical principles

• Masked region guidance paradigm:
  • Reserved mask areaUnlike traditional "masking and repair" methods, SmartEraser preserves the position of the masked region in the input, using it as a guide for the removal process. The model can accurately identify objects that need to be removed, reducing the risk of regenerating objects within the masked region.
  • Context informationUser-defined masks often extend beyond the target object itself, helping the model preserve the surrounding context in the final result, making the generated image more natural and realistic.
• Syn4Removal dataset:
  • Synthetic data generationThe method trains a model conforming to a new paradigm by constructing training data using a synthetic approach. This involves pasting object instances from different images onto different background images to form the input image. The mask of the pasted object serves as the input mask, and the original background image serves as the ground truth.
  • Large-scale datasetsThe Syn4Removal dataset contains 1 million pairs of image triples, covering a variety of scenes and object types, providing rich data support for model training.
• Framework based on text-to-image diffusion model:
  • Masking enhancementTo enhance the model's robustness to different mask shapes input by the user, various mask deformation methods are applied during training to simulate the shape of the user-input mask. These methods include original masks, erosion masks, dilation masks, convex hull masks, elliptical masks, and bounding box and Bézier curve masks.
  • CLIP-based visual guidanceThe visual features of the removed target are extracted using a pre-trained CLIP model and mapped into the feature space of the text encoder.
  • loss functionThe model's training loss function is based on a standard diffusion process, optimizing model parameters by minimizing the difference between predicted noise and actual noise.

SmartEraser project address

• Project official website:https://longtaojiang.github.io/smarteraser
• GitHub repository:https://github.com/longtaojiang/SmartEraser
• arXiv technical paper:https://arxiv.org/pdf/2501.08279

SmartEraser application scenarios

• Personal photo editingUsed to remove strangers from travel photos, unwanted people or objects from family photos, and to clean up the background of ID photos and social media profile pictures.
• Professional Image ProcessingIn advertising and marketing, remove distracting elements from product photography and optimize the background of advertising designs.
• graphic designIt helps graphic designers highlight focal points and remove unnecessary background elements in poster and cover designs.
• Cultural Relics RestorationIn the restoration of cultural relics, such as ancient paintings, this involves removing stains and damage.
• Medical and scientific researchIn medical image processing, such as X-ray and CT images, it removes equipment artifacts and markers; in scientific image processing, such as microscope and satellite images, it cleans up background noise and interference, improving the value of image analysis.