CreatiLayout - Fudan University and ByteDance jointly launch innovative layout-to-image generation technology

What is CreatiLayout?

CreatiLayout is an innovative layout-to-image (L2I) technology jointly proposed by Fudan University and ByteDance. It utilizes the large-scale layout dataset LayoutSAM, containing 2.7 million image-text pairs and 10.7 million entity annotations, each with detailed descriptions of attributes such as color, shape, and texture. The core of CreatiLayout lies in its SiamLayout framework, which treats layout information as an independent modality. Through MM-DiT's native MM-Attention, it facilitates interaction between the layout and image modalities, effectively resolving the modality competition problem. Furthermore, CreatiLayout introduces LayoutDesigner, which uses a large language model to help users generate and optimize layouts, supporting multiple input methods such as center points, masks, sketches, and text descriptions.

CreatiLayout's main functions

• High-quality image generationBased on the Siamese Multimodal Diffusion Transformer, CreatiLayout can generate high-quality and fine-grained images. It can accurately render complex attributes such as color, texture, and shape.
• Layout generation and optimizationWith LayoutDesigner, CreatiLayout can generate and optimize layouts based on various user inputs, such as center points, masks, sketches, and text descriptions. This allows users to express their design intent more flexibly and generate harmonious and aesthetically pleasing layouts.
• Large-scale dataset supportCreatiLayout has built a large-scale layout dataset called LayoutSAM, which contains 2.7 million image-text pairs and 10.7 million entity annotations. Each entity is described in detail with attributes such as color, shape, and texture, providing rich data support for the model.
• Multimodal interactionIn terms of model architecture, CreatiLayout treats layout information as an independent modality and achieves the interaction between layout and image modality through MM-DiT's MM-Attention.

CreatiLayout's technical principles

• Twin multimodal diffusion converterCreatiLayout uses this transformer to achieve high-quality and fine-grained image generation. It can effectively handle multimodal data, including images, text, and layout information.
• SiamLayout frameworkThe framework treats layout information as an independent modality, equally important as text and image modalities. It utilizes MM-DiT's native MM-Attention to facilitate interaction between the layout and image modalities, mitigating modal competition and enhancing the guidance role of the layout. Specifically, the interaction between the image, text, and layout modalities is decoupled into two twin branches: an image-text interaction branch and an image-layout interaction branch. This ensures that the guidance of text and layout on image content is independent and does not interfere with each other.
• LayoutDesignerIt performs layout planning and optimization based on a large language model, and supports users to generate and optimize layouts through various input methods (such as center point, mask, sketch, text description, etc.).

CreatiLayout's project address

• Project official website:https://creatilayout.github.io
• Github repository:https://github.com/HuiZhang0812/CreatiLayout
• arXiv technical paper:https://arxiv.org/pdf/2412.03859
• Experience the demo online:https://huggingface.co/spaces/HuiZhang0812/CreatiLayout

Application scenarios of CreatiLayout

• Promotional poster productionDesigners can use CreatiLayout to quickly generate high-quality promotional posters that meet their expectations.
• Furniture placement optimizationCreatiLayout can accurately analyze the position and size of each piece of furniture to generate realistic interior renderings.
• Visual effects creationIt can help developers explore new visual effects and styles, and create more creative and attractive game visuals by precisely controlling layout and image details.
• Teaching material productionTeachers can use CreatiLayout to present knowledge points in a visually appealing way.