MultiBooth - A multi-concept image generation method jointly developed by Tsinghua University and Meta and other institutions.

What is MultiBooth?

MultiBooth is a multi-concept image generation method developed by Tsinghua University Shenzhen International Graduate School, Meta, Hong Kong University of Science and Technology, and other institutions. It can generate images containing multiple user-specified concepts from text. MultiBooth divides the generation process into two stages: single-concept learning and multi-concept integration. In the single-concept learning stage, a concise and discriminative embedding representation is learned for each concept based on a multimodal image encoder and adaptive concept normalization technology, with LoRA technology used to improve concept fidelity. In the multi-concept integration stage, a Region Customization Module (RCM) is used to generate each concept within a specified region based on bounding boxes and region cues, ensuring accurate interaction between different concepts based on basic cues. MultiBooth achieves efficient multi-concept image generation while maintaining high image fidelity and text alignment capabilities, and has low costs in both training and inference phases.

Main functions of MultiBooth

• Multi-concept image generationGenerate an image containing multiple specified concepts based on text prompts provided by the user.
• High fidelity and text alignmentThe generated images are highly faithful, clearly displaying the detailed features of each concept, and are highly aligned with the text prompts, ensuring that the image content is consistent with the user's intent.
• Efficient ReasoningIn the process of generating multiple concepts, the reasoning cost is low and the reasoning time does not increase significantly with the increase in the number of concepts, making the generation of multiple concept images more efficient.
• Plug-in generationIt supports combining different single-concept modules in a plug-in manner to generate multi-concept images without retraining the model for each concept combination, thus improving the model's flexibility and scalability.

The technical principles of MultiBooth

• Single concept learning stage:
  • Multimodal image encoderUsing the QFormer encoder, input images and concept names (such as "dog"), based on the interaction of self-attention and cross-attention layers, it generates customized embedding representations aligned with text, learning a concise and discriminative embedding for each concept.
  • Adaptive Concept Normalization (ACN)Adjusting the L2 norm of the customized embedding makes it comparable to other word embeddings in the prompt, solves the problem of inter-domain differences in the embedding space, and improves the ability to generate multiple concepts.
  • High-efficiency concept coding technologyLoRA technique is used to perform low-rank decomposition of the attention layer in U-Net, avoiding language drift caused by fine-tuning U-Net, improving the concept fidelity of single concept learning, and reducing the need for additional parameter storage.
• Multi-concept integration phase:
  • Regional Customization Module (RCM)In the cross-attention layer, image features are divided into different regions based on bounding boxes and region cues obtained by user definition or automated processes. Each region generates a concept guided by a corresponding single-concept module and cues. Based on the basic cues, the interaction between concepts in different regions is ensured, and multiple concepts are accurately fused in the same image.
  • Parallel generation and interactionIn RCM, multiple single-concept modules can be generated simultaneously, and parallel interaction between concepts is achieved based on the cross-attention mechanism, avoiding the increase in concept fusion and reasoning costs.

MultiBooth project address

• Project official website:multibooth.github.io
• GitHub repository:https://github.com/chenyangzhu1/MultiBooth
• arXiv technical paper:https://arxiv.org/pdf/2404.14239

Application scenarios of MultiBooth

• Entertainment and Creative Industries: Quickly generate mysterious ancient tomb scene images for adventure games, showcasing internal mechanisms and murals, enriching the game's exploration elements.
• Advertising and MarketingCreate advertising posters for cosmetic face masks, showcasing the rejuvenated facial skin of young women after using the product, conveying the product's efficacy and brand positioning.
• Education and LearningGenerate images of medieval castles, clearly showing the structure of towers and walls, helping students understand the characteristics of castles and deepen their memory of historical knowledge.
• e-commerceGenerate styling images for summer dresses, showcasing how floral dresses look paired with straw sandals and straw hats, to attract customers to make a purchase.
• Scientific Research and EngineeringThe project generates schematic diagrams of novel nanomaterial structures, showcasing their ultralight weight and high strength properties, and helps the public understand the innovativeness of scientific research results.