DreamO - A custom image generation framework jointly developed by ByteDance and Peking University

What is DreamO?

DreamO is a unified framework for customized image generation, jointly developed by ByteDance's creative team and the School of Electronic and Computer Engineering at Peking University Shenzhen Graduate School. Based on a pre-trained Diffusion Transformer (DiT) model, it enables flexible customization for various image generation tasks. DreamO supports seamless integration of multiple conditions such as identity, subject, style, and background, and improves the consistency and condition decoupling capabilities of generated results based on feature routing constraints and placeholder strategies. DreamO employs a staged training strategy to ensure efficient convergence and high-quality generation even in complex tasks. The framework is widely applicable to scenarios such as virtual try-on, style transfer, and subject-driven generation, providing powerful customization capabilities for image generation.

DreamO's main functions

• Multi-condition integrationIt supports customization of various conditions such as identity, subject, style, and background, and seamlessly integrates these conditions into image generation.
• High-quality generationBased on a phased training strategy, this ensures high-quality generated images and corrects biases introduced by low-quality data.
• Flexible condition controlIt allows users to precisely control the position and layout of conditions in the generated image.
• Wide applicabilityIt supports handling complex multi-condition scenarios and is suitable for various image generation tasks such as virtual try-on, style transfer, and subject-driven generation.

DreamO's technical principles

• Diffusion Converter (DiT) FrameworkThe core architecture uses a diffusion transformer to achieve image customization based on unified processing of different types of inputs (such as text, images, conditions, etc.). The diffusion model generates images by progressively removing noise, and the transformer architecture enhances the model's understanding and processing capabilities of input conditions.
• Feature routing constraintsTo improve the consistency between the generated result and the reference image, feature routing constraints are introduced. These constraints are based on an attention mechanism that optimizes the relationship between the conditional image and the generated image, ensuring that specific regions in the generated image correspond to the conditional image and avoiding coupling between conditions.
• Placeholder strategyBy adding placeholders (such as [ref#1]) to the text description, the conditional image is associated with a specific object in the text description, enabling precise control over the conditional position in the generated image.
• Phased training strategyThis approach utilizes a phased training method, including an initial phase (simple tasks), a full training phase (multi-tasks), and a quality alignment phase (correcting biases). This helps the model converge smoothly under complex data distributions and maintain high-quality generation.
• Large-scale training dataTo achieve broad generalization capabilities, a large-scale training dataset covering various tasks (such as identity customization, subject-driven, virtual try-on, style transfer, etc.) is constructed to ensure that the model can learn generative capabilities under different conditions.

DreamO's project address

• Project official website:https://mc-e.github.io/project/DreamO/
• GitHub repository:https://github.com/bytedance/DreamO
• arXiv technical paper:https://arxiv.org/pdf/2504.16915

Application scenarios of DreamO

• Virtual try-onUsers upload their own photos and clothing images to generate a try-on effect.
• Style transferIt can transform ordinary photos into artistic images or generate different styles of visual effects based on design sketches, making it suitable for artistic creation and design inspiration exploration.
• Subject-driven generationIt generates personalized avatars or virtual characters based on user-uploaded photos, supports multi-subject fusion, and can be used in social media, games, and animation production.
• Identity customizationGenerates images containing specific character images, supports the preservation and fusion of identity features, and is suitable for virtual social interaction and personalized content creation.
• Creative content generationIt generates creative advertisements, film and television special effects, or educational scene images based on text descriptions and conditional images, supports a variety of complex customization tasks, and meets creative needs.