DreamOmni - A unified image generation and editing model jointly launched by CUHK, ByteDance, and other organizations.

What is DreamOmni?

DreamOmni is a unified image generation and editing model jointly developed by the Chinese University of Hong Kong, ByteDance, and the Hong Kong University of Science and Technology. The model integrates text-to-image (T2I) generation with various editing tasks, including imperative editing, restoration, drag-and-drop editing, and reference image generation. DreamOmni addresses the challenge of creating high-quality editing data through an efficient synthetic data pipeline, supporting model training and scaling. By jointly training T2I and editing tasks, it enhances concept understanding and improves image generation quality. In extensive experimental evaluations, DreamOmni demonstrates significant advantages in image generation and editing tasks with superior performance.

DreamOmni's main functions

• Unified image generation and editingDreamOmni can handle text-to-image (T2I) generation as well as a variety of image editing tasks, such as instructional editing, repair (e.g., repair and expansion), drag-and-drop editing, and reference image generation.
• Synthetic Data PipelineUsing sticker-like elements, it efficiently and accurately synthesizes large-scale, high-quality edit data, supporting the training of a unified model.
• Joint trainingBy combining T2I data and data from various editing tasks for training, the model's understanding of specific concepts can be improved, the generation quality can be enhanced, and the editing performance can be improved.
• Multitasking supportThe model can understand and perform operations such as adding, removing, and replacing, as well as handle editing tasks such as image translation, rotation, and scaling.

DreamOmni's technical principles

• Framework DesignIntegrating the T2I model with multiple editing tasks enables multi-task learning.
• Visual-Language Model (VLM)Based on VLM unified coding of visual and linguistic cues, it combines coded cues with latent representations of noise to achieve joint computation.
• Synthetic data generationBased on a composite collage data pipeline, DreamOmni can create precise editing data, supporting add, delete, and replace operations, as well as drag-and-drop editing and reference image generation.
• Multimodal input compatibilityThe framework is simple in design and compatible with multimodal input, enabling DreamOmni to handle complex prompts and image conditions.
• Training strategyDreamOmni employs a phased training strategy, training step-by-step from low resolution to high resolution to optimize model performance and training efficiency.
• Optimization technology: Use techniques such as Rectified Flow to optimize the model, and perform a forward process between noise and data in a linear interpolation manner to improve the quality and efficiency of generation.

DreamOmni's project address

• Project official website:zj-binxia.github.io/DreamOmni-ProjectPage
• arXiv technical paper:https://arxiv.org/pdf/2412.17098

DreamOmni application scenarios

• Digital art creationArtists and designers can generate or edit images to quickly transform creative concepts into visual works.
• Game developmentGame developers create game assets, such as characters, environments, and items, or edit existing game elements.
• Film and entertainment industryGenerate special effects backgrounds or edit existing scene images during film production, saving costs and time.
• Advertising and MarketingMarketers can quickly generate compelling ad images and marketing materials that adapt to different advertising channels.
• Education and trainingIn the field of education, it is used to create teaching materials, such as diagrams and simulated scenarios, to enhance the learning experience.