GroundingBooth - Adobe, in collaboration with several universities, launches a custom framework for themes and text-to-image mapping.

What is GroundingBooth?

GroundingBooth is an advanced text-to-image customization framework developed by a research team from Washington University in St. Louis, Adobe, and Purdue University. Based on a text-image alignment module and a masking cross-attention layer, it achieves precise spatial alignment of foreground subjects and background objects. The framework generates images that meet personalized needs in terms of layout, identity preservation, and text-image coherence, supports multi-theme customization, and maintains high accuracy in complex scenes. GroundingBooth is the first work to achieve joint grounding of theme-driven foreground generation and text-driven background generation, enabling highly customized visual content creation.

Main functions of GroundingBooth

• Single theme customizationGenerates a custom image that matches a user-provided text description and a single theme image.
• Multi-theme and text entity joint customizationIt supports customization of multiple themes and text entities simultaneously, generating complex images containing multiple objects and text descriptions.
• Spatial alignmentEnsure that the objects in the generated image are spatially consistent with the input layout information.
• Identity PreservationPreserve the identity features of the subject during image generation.
• Text-Image AlignmentEnsure that the generated image content is consistent with the text description.

GroundingBooth's technical principles

• Feature extractionFeature embeddings for text and images are extracted based on CLIP text encoder and DINOv2 image encoder, respectively.
• Grounding moduleGrounding markers are generated by combining text and image features with the input layout information through positional encoding.
• Masking Cross Attention LayerMasked cross-attention layers are used in each Transformer block of U-Net to control the combination of foreground and background features, ensuring the accuracy of feature injection.
• Precise layout controlBy using a masked cross-attention layer during the training and inference phases, precise control over the size and position of objects in the generated image can be achieved.
• Model trainingDuring the training phase, the model learns how to generate accurate image layouts based on text descriptions and reference objects.
• Model InferenceDuring the inference phase, the model processes multiple reference objects and achieves multi-topic customization through replicated masking cross-attention layers.
• Avoid context mixingDistinguish between topic-driven foreground generation and text-driven background generation to avoid contextual confusion during the generation process.

GroundingBooth project address

• Project official websitegroundingbooth.github.io
• arXiv technical paper:https://arxiv.org/pdf/2409.08520v1

Application scenarios of GroundingBooth

• Personalized product customizationUsers can generate customized product images based on their preferences, such as customized T-shirts, cups, phone cases, etc., with specific patterns or text printed on them.
• Artistic CreationArtists and designers use GroundingBooth to generate artworks with specific styles and elements.
• Game DesignGame developers can use this framework to quickly generate personalized characters, scenes, or items in their games.
• Advertising and MarketingMarketers create custom images that match the advertising copy to enhance the appeal of their ads.
• Social media content creationUsers share customized images on social media, which are related to specific topics or events.
• Educational and training materialsEducators generate instructional images with specific information and layouts to enhance the appeal and effectiveness of learning materials.