OmniBooth - An image generation framework jointly developed by Huawei Noah's Ark and HKUST

What is OmniBooth?

OmniBooth is an image generation framework jointly developed by Huawei Noah's Ark Lab and the research team at HKUST. It supports spatial control and instance-level customization based on text prompts or image references. The framework uses user-defined masks and associated text or images to precisely control the position and attributes of objects in an image, improving the controllability and practicality of text-to-image synthesis technology. The core of OmniBooth lies in its innovative latent control signal, a high-dimensional spatial feature that seamlessly integrates spatial, textual, and image conditions to achieve fine-grained image synthesis control.

OmniBooth's main functions

• Multimodal command controlIt supports controlling image generation with text prompts or image references, enabling image synthesis under multimodal commands.
• Space control and instance-level customizationUser-defined masks and text or image guidance allow for precise control over the position and attributes of objects in an image, enabling instance-level customization.
• High-dimensional potential control signalsBased on latent control signals, it seamlessly integrates spatial, textual, and image conditions to provide a unified representation method.
• Flexibility and practicalityUsers can choose text or images as multimodal conditions as needed, enhancing the flexibility and practicality of generated images.

OmniBooth's technical principles

• Multimodal embedding extraction:
  • Text embeddingExtract the embedding vector of the text prompt using the CLIP text encoder.
  • Image embeddingThe DINOv2 feature extractor is used to extract the embedding vector of the image reference, preserving the image's identity and spatial information.
• Potential control signalsText and images are embedded into vectors and plotted into high-dimensional latent control signals, which contain spatial information and rich latent features.
• Space deformation technologyUsing spatial deformation technology, images are effectively transformed and integrated into potential control signals while preserving the details and structure of the images.
• Feature alignment network and edge loss function:
  • Develop a feature alignment network to inject conditions into latent features.
  • We propose edge loss to enhance supervision in high-frequency regions, thereby improving the quality and structural alignment of the generated images.
• Multi-scale training and random mode selection strategyDuring the training phase, the model employs multi-scale training and random modality selection strategies to enhance its adaptability to different resolutions and modal inputs.

OmniBooth's project address

• Project official website:len-li.github.io/omnibooth
• GitHub repository:https://github.com/EnVision-Research/OmniBooth
• HuggingFace model library:https://huggingface.co/lilelife/OmniBooth
• arXiv technical paper:https://arxiv.org/pdf/2410.04932

Application scenarios of OmniBooth

• Dataset generationGenerate synthetic datasets needed to train machine learning models, especially when real-world data is difficult to obtain.
• Content creationArtists and designers create new visual content, such as illustrations and concept art, using text or images as guidance to realize their creative ideas.
• Games and entertainmentIn game development, it is used to quickly generate prototype designs for game environments, characters, and items.
• Virtual Reality (VR) and Augmented Reality (AR)Create realistic backgrounds and objects for virtual environments to enhance the user experience.
• Advertising and MarketingQuickly generate advertising images and marketing materials, and customize them according to customer needs.