OmniEdit - A general-purpose image editing model open-sourced by the University of Waterloo and other institutions.
OmniEdit is an advanced image editing technology that trains a general-purpose model using supervision from multiple expert models to handle various image editing tasks. It can handle images with different aspect ratios and seven different image editing tasks...
What is OmniEdit?
OmniEdit is an advanced image editing technology that trains a general-purpose model under the supervision of multiple expert models to handle various image editing tasks. It can handle images with different aspect ratios and seven different image editing tasks, including object replacement, removal, and addition, supporting arbitrary aspect ratios and resolutions. Supervised training based on seven expert models ensures task coverage, and its use in large-scale multimodal model scoring improves the quality of training data. Employing the novel EditNet architecture, it improves the success rate of editing. OmniEdit significantly outperforms existing models (including InstructPix2Pix, MagicBrush, UltraEdi, etc.) in both automatic and human evaluation, accurately following instructions while maintaining the fidelity of the original image.
OmniEdit's main functions
- Multi-task editing capabilitiesOmniEdit can perform seven different image editing tasks, including object replacement, object removal, object addition, attribute modification, background replacement, environment change, and style transfer.
- Expert Model SupervisionOmniEdit is trained under supervision based on seven different expert models to ensure task coverage.
- Supports arbitrary aspect ratios and resolutionsThe model can handle images with different aspect ratios and resolutions, making it suitable for various real-world scenarios.
- Instruction-driven editingUsers can guide OmniEdit to perform specific image editing based on text commands, improving editing flexibility and user control.
- High-quality image outputDuring the editing process, OmniEdit maintains the high fidelity of the original image while reducing noise and artifacts.
- Data quality control: Use large-scale multimodal models to score synthetic samples to improve the quality of training data.
OmniEdit's technical principles
- Supervised learning from expert to general modelA general editing model is trained using supervision signals from multiple expert models, each focusing on a specific editing task.
- Importance SamplingLarge-scale multimodal models (such as GPT-4o) score the quality of synthetic samples and perform importance sampling to improve the quality of the training dataset.
- EditNet architectureBased on a diffusion-transformer architecture, it uses intermediate representations for interaction, supports the interaction between control branches and original branches, and enhances the model's understanding of editing tasks.
- Supports any aspect ratioDuring training, images with different aspect ratios are included to ensure that the model can adapt to any image aspect ratio.
OmniEdit project address
- GitHub repository:https://github.com/TIGER-AI-Lab/OmniEdit
- HuggingFace model library:https://huggingface.co/collections/TIGER-Lab/omniedit-6732d8e381c3e56b0a2106d5
- arXiv technical paper:https://arxiv.org/pdf/2411.07199
Application scenarios of OmniEdit
- Professional Image EditingDesigners and artists perform efficient image editing work, including advertising design, artistic creation, and photo restoration.
- Social media content creationSocial media users can quickly edit and enhance images to increase the appeal of their content.
- e-commerceOnline merchants can edit product images, such as changing backgrounds and adjusting styles, to enhance the product's market appeal.
- News and MediaNews organizations can quickly adjust news photos to adapt to different publishing needs and styles.
- Education and trainingIn the field of education, it serves as a teaching tool to help students learn image editing and visual design.