BrushEdit - An image editing framework jointly developed by Tencent, Peking University, and others, providing instruction-guided image editing and restoration.
BrushEdit is an advanced image editing framework jointly developed by Tencent and Peking University, among other institutions. It is an advanced iteration of the BrushNet model. The framework combines multimodal large language models (MLLMs) and a two-branch image inpainting model...
What is BrushEdit?
BrushEdit is an advanced image editing framework jointly developed by Tencent, Peking University, the Chinese University of Hong Kong, and Tsinghua University. It is an advanced iteration of the BrushNet model. The framework combines multimodal large language models (MLLMs) and a two-branch image inpainting model to achieve instruction-guided image editing and inpainting. It supports users to perform free-form, multi-round interactive editing operations using natural language commands. BrushEdit can handle significant modifications such as adding and removing objects while maintaining background continuity and natural editing effects, significantly improving the flexibility of image editing and the user experience.
BrushEdit's main functions
- Instruction-guided image editingUsers can use natural language commands to guide image editing tasks, such as adding, deleting, or modifying objects in an image.
- Multi-round interactive editingIt supports multiple rounds of interaction during the editing process, allowing users to gradually adjust and improve the editing results.
- Free-form mask editingUsers can freely draw masks to specify the editing area without the need for precise segmentation tools.
- Background and foreground processingThe frame can distinguish between the editing area (foreground) and the non-editing area (background), ensuring that editing operations do not affect non-target parts of the image.
- Image restorationAutomatically fills and repairs missing or specified areas in an image, such as removing unwanted objects or filling holes.
BrushEdit's technical principles
- Multimodal Large Language Models (MLLMs): Use pre-trained MLLMs to parse the user's free-form editing instructions and identify the editing type and target object.
- Dual-branch image restoration modelThe framework uses a two-branch architecture: one branch is responsible for generating the image of the masked region, and the other branch handles the background information of the unmasked region.
- Proxy Collaboration FrameworkBased on the collaboration between agents (agent instructors and agent commanders), it enables editing category classification, main object identification, mask acquisition, and editing area repair.
- Feature fusionThe system integrates user instructions and mask information into the image inpainting model, guiding the model to generate content that matches the instructions within the masked area.
- Zero convolutional layers and feature insertionThe frozen pre-trained model is connected to the trainable BrushEdit model using zero convolutional layers, which reduces noise in the early training stage and integrates features layer by layer to achieve fine control.
- Hybrid fine-tuning strategyThe fine-tuning strategy that combines random masks and segmented masks allows the model to handle a variety of masking tasks without being limited by a specific mask type.
BrushEdit project address
- Project official website:liyaowei-stu.github.io/project/BrushEdit
- GitHub repository:https://github.com/TencentARC/BrushEdit
- HuggingFace model library:https://huggingface.co/TencentARC/BrushEdit
- arXiv technical paper:https://arxiv.org/pdf/2412.10316
Application scenarios of BrushEdit
- Content creation and editingArtists and designers can use it for creative image editing to quickly achieve complex visual effects and artistic creations.
- Media and EntertainmentIn film and video production, this involves restoring old or damaged footage or removing unwanted elements during post-production.
- Advertising and MarketingAdvertisers can quickly change the product or background in advertising images to adapt to different marketing strategies.
- social mediaUsers share edited images on social media, such as holiday decorations and virtual clothing try-ons.
- e-commerceEditing product images on e-commerce platforms, such as changing product backgrounds, adjusting product colors, or adding promotional labels.