InstanceAssemble - An image generation framework jointly launched by Xiaohongshu and Fudan University

What is InstanceAssemble?

InstanceAssemble is a lightweight layout-to-image generation framework jointly developed by Xiaohongshu and Fudan University. Through an innovative "instance assembly attention" mechanism, the framework achieves accurate image generation from simple to complex, and from sparse to dense layouts. Users only need to provide the bounding box positions of objects and content descriptions, and the AI can generate semantically correct images at the corresponding locations. InstanceAssemble is based on a diffusion transformer architecture and adopts a lightweight adaptation method, requiring only a few additional parameters to adapt to mainstream models, significantly lowering the barrier to entry.

The main functions of InstanceAssemble

• Precise layout controlBy specifying the location (boundary box) and content description of each object, the AI can generate semantically correct image content at the corresponding location.
• Layout generation from simple to complexFor both simple objects and complex, dense scenes, InstanceAssemble can maintain high-precision layout alignment and semantic consistency.
• Multimodal content controlIt supports defining the content of each instance through multiple modalities such as text description, reference image, depth map, and edge map, further improving the accuracy and detail of the generated image.
• Lightweight adapterIt can adapt to a variety of mainstream diffusion models without retraining the entire model, using only a small number of additional parameters (such as only about 3.46% of the additional parameters required to adapt to the Stable Diffusion 3-Medium model), thus lowering the barrier to entry.
• Strong generalization ability: When training, only sparse layout (≤10 instances) is used, which can maintain robust performance on dense layout (≥10 instances) and adapt to layout conditions of different complexities.

Technical principles of InstanceAssemble

• Diffusion Model BasicsInstanceAssemble is based on current mainstream diffusion transformer architectures (such as Multimodal Diffusion Transformer, MMDiT), and uses the powerful generative capabilities of diffusion models to generate high-quality images through a stepwise denoising process.
• Instance assembly attention mechanism:
  • Layout EncoderThe layout conditions provided by the user (such as bounding boxes, text descriptions, etc.) are encoded into instance tokens. For each instance, a corresponding instance token is generated by combining its location information (enhanced by DenseSample) and content description (text or visual content).
  • Assemble Attention Module (Assemble-Attn)During the generation process, image tokens interact with instance tokens. For each instance, only the image region within the bounding box is considered, and the region features are updated through an attention mechanism. The updated features are then fused into the overall image through weighted assembly, ensuring layout and semantic consistency for each instance.
  • Cascade structureIt adopts a cascading mechanism, using the basic MMDiT model to process global text prompts and image features, and then using the Assemble-MMDiT module to process instance layout conditions, achieving precise layout control while maintaining global generation quality.
• Lightweight adapter (LoRA)To efficiently adapt to existing diffusion models, InstanceAssemble uses low-rank adaptation (LoRA) technology, which introduces only a small number of trainable parameters (such as low-rank matrices) into the attention module, and achieves layout control without changing the capabilities of the basic model.
• Evaluation and BenchmarkingTo more accurately measure the matching degree between layout and image, InstanceAssemble proposed the "Layout Grounding Score" (LGS) evaluation metric and created the "DenseLayout" benchmark set containing 5,000 images and 90,000 instances to evaluate the generation performance under complex layout conditions.

Project address of InstanceAssemble

• GitHub repositoryhttps://github.com/FireRedTeam/InstanceAssemble
• arXiv technical paper: https://arxiv.org/pdf/2509.16691

Application scenarios of InstanceAssemble

• Design and AdvertisingIt helps designers and advertising companies quickly generate design sketches or advertising images that conform to specific layouts and styles, and precisely control the position and content of elements.
• Content creationIt provides content creators with high-quality image generation tools for social media, video production, and more, enhancing the appeal and professionalism of their content.
• Game developmentThe framework can quickly generate game scenes and character equipment layouts, helping game developers efficiently design realistic backgrounds and levels.
• Education and TrainingThe framework can generate teaching materials and virtual laboratory scenarios, helping teachers to convey knowledge more intuitively and improve teaching effectiveness.
• Architectural DesignIt assists interior designers and architects in generating layout drawings and renderings of interior spaces and building exteriors, and optimizes design schemes.