Lego-Edit - Xiaomi's open-source image editing framework - AiBoss

What is Lego-Edit?

Lego-Edit is an open-source, instruction-based image editing framework from Xiaomi. It leverages the generalization capabilities of a Multimodal Large Language Model (MLLM) to enable flexible image editing. Employing a model-level toolkit, it includes various efficiently trained models capable of performing diverse image operations. Lego-Edit utilizes a three-stage progressive reinforcement learning training strategy: supervised fine-tuning (SFT), task-specific reinforcement learning (RL), and finally, additional RL training with a large number of unlabeled instructions to enhance its ability to handle flexible instructions. Its strengths lie in its powerful generalization capabilities, achieving state-of-the-art (SOTA) performance on multiple benchmarks. It supports local, global, and multi-step editing and accepts masked input for precise control of the editing area. Lego-Edit can integrate new tools without retraining, facilitating easy feature expansion.

Main functions of Lego-Edit

Powerful image editing capabilitiesLego-Edit can perform a variety of complex image editing tasks according to user instructions, including but not limited to local editing, global editing, and multi-step editing, to meet the image processing needs in different scenarios.
Flexible instruction comprehension and executionWith the generalization capabilities of the Multimodal Large Language Model (MLLM), Lego-Edit can understand and execute open-domain instructions, and can process even unfamiliar instructions through its powerful reasoning ability.
Efficient use of model-level toolkitsThe framework includes a variety of model-level tools that can be efficiently trained on limited data. These tools have a variety of image manipulation functions. MLLM can use these tools to perform fine-grained editing operations, achieving efficient and accurate image modification.
Integrate new tools without retrainingLego-Edit supports the integration of newly introduced editing tools without additional tweaks, making its functionality easily expandable to adapt to ever-changing image editing needs.
Mask input supports precise editingOne of the highlights of Lego-Edit is that it accepts mask input. Users can use masks to precisely specify the image area to be edited, achieving more accurate local editing effects.
Open source and ease of useThe Lego-Edit code is open source under the Apache 2.0 license, and the model is open source under the CC BY-NC 4.0 license, making it easily accessible and usable by users. With simple environment setup and pre-trained model download, users can begin image editing using the Gradio WebUI, lowering the barrier to entry.

The technical principles of Lego-Edit

Model-level toolkitLego-Edit integrates multiple efficiently trained models, each responsible for specific image manipulation functions such as color adjustment and object replacement, forming a feature-rich toolkit that provides basic support for complex image editing tasks.
Multimodal Large Language Model (MLLM) drivenLeveraging the powerful generalization and reasoning capabilities of MLLM, it understands user commands and coordinates different models in the model-level toolkit to complete the transformation from command parsing to specific operations, enabling flexible image editing.
Three-stage progressive reinforcement learning training strategyFirst, supervised fine-tuning (SFT) is performed to allow the model to learn basic image editing knowledge. Then, reinforcement learning (RL) is used to train the model on specific editing tasks to build reasoning ability and tool usage knowledge. Finally, additional RL training is performed using a large number of unlabeled instructions, with feedback provided by a large-scale critical model to further enhance the ability to handle flexible instructions.
Masked input mechanismIt supports users to specify the image area to be edited by using a mask, making the editing operation more precise. It can modify specific parts of the image without affecting other areas, improving the flexibility and accuracy of editing.
Tool integration that does not require retrainingLego-Edit can directly integrate newly introduced editing tools without additional tweaks, quickly adapting to new editing needs, expanding the scope of functionality, and maintaining the system's efficiency and scalability.

Lego-Edit project address

Project official website: https://xiaomi-research.github.io/lego-edit/
Github repository: https://github.com/xiaomi-research/lego-edit
arXiv technologypaper: https://arxiv.org/pdf/2509.12883

Application scenarios of Lego-Edit

Creative Design FieldDesigners can use Lego-Edit to quickly realize creative ideas and complete complex operations such as image compositing and style transformation with simple commands, improving design efficiency and inspiring more creative ideas.
Content creation and editingIn scenarios such as video production, advertising design, and social media content creation, Lego-Edit can help creators quickly modify image materials, such as adjusting colors, replacing backgrounds, and adding effects, to meet diverse content creation needs.
E-commerce and product displayE-commerce merchants can use Lego-Edit to optimize product images, such as removing blemishes, adjusting lighting, and adding virtual scenes, to improve product presentation and increase users' willingness to buy.
Education and TrainingIn the field of education, Lego-Edit can serve as a teaching tool to help students learn image editing skills and cultivate creativity and aesthetic appreciation. At the same time, teachers can also use it to quickly create image materials for teaching courseware.
Personal photo enhancementRegular users can use Lego-Edit to enhance their personal photos, such as removing backgrounds, adjusting skin tones, and adding decorative elements, easily creating satisfactory photos for social media sharing or personal collection.
Virtual Reality and Game DevelopmentIn virtual reality and game development, Lego-Edit can be used to quickly generate and modify image resources in games, such as character appearances and scene elements, improving development efficiency and enriching game visual effects.