EliGen - A new entity-level controllable image generation framework jointly launched by Zhejiang University and Alibaba.

What is EliGen?

EliGen is a novel entity-level controllable image generation framework jointly developed by Zhejiang University and Alibaba Group. By introducing a region attention mechanism, it can seamlessly integrate entity cues and spatial masks of arbitrary shapes into the diffusion transformer without additional parameters. EliGen includes a dataset of 500,000 high-quality annotated samples for training models to achieve robust and accurate entity-level manipulation. EliGen proposes a restoration fusion pipeline that can be extended to multi-entity image restoration tasks.

EliGen's main functions

• Physical precision controlThrough its region attention mechanism, EliGen can precisely control each entity in an image, including its position, shape, and semantic attributes.
• Multi-entity image restorationEliGen proposes a repair fusion pipeline that enables multi-entity image repair tasks. Through a region attention mechanism, multiple entities can be repaired in a single forward pass.
• Stylized entity controlCombined with the IP-Adapter, EliGen can generate target images based on the style of reference images, enabling stylized entity control.
• Interactive Image Design and EditingBy integrating with MLLM (Multimodal Language Model), EliGen can perform image design and editing based on dialogue, allowing users to generate or modify images through text descriptions.
• Integration with community modelsEliGen can seamlessly integrate with open-source models such as IP-Adapter, In-Context LoRA, and MLLM, unlocking new creative possibilities. It enables stylized entity control, entity transfer, and dialogue-based image design and editing.
• Strong generalization abilityEliGen maintains good generalization ability under different random seeds, continuously changing entity positions, and unreasonable position inputs, generating high-quality and compliant images.

EliGen's technical principles

• Regional attention mechanismEliGen introduces a region attention mechanism, which extends the attention module of the Diffusion Transformer (DiT) to handle entity masks of arbitrary shapes. EliGen combines global and local cue embeddings to form an extended cue sequence, constructing a joint attention mask (including entity-latent, inter-entity, and intra-entity masks) based on spatial conditions. No additional parameters are required, allowing modification of entity details before training.
• Building high-quality datasetsTo train EliGen, researchers constructed a dataset containing 500,000 high-quality annotated samples. The samples were generated as images using Flux, with global cues and entity information annotations based on the Qwen2-VL visual language model. This high-quality dataset ensures the model's robustness and accuracy in entity-level manipulation.
• Training and fine-tuningEliGen employs the LoRA (Low-Rank Adaptation) method for efficient fine-tuning, ensuring rapid model convergence. LoRA weights are applied to the linear layers of each block of DiT, including the projection layer of the attention layer and the linear layers within the adaptive LayerNorm.
• Repair fusion pipeEliGen proposes an inpainting fusion pipeline for multi-entity image inpainting tasks. Through region-based noise fusion operations, the pipeline enables EliGen to inpaint multiple entities in a single forward pass while preserving the quality of un-inpainted regions.

EliGen's project address

• arXiv technical paper:https://arxiv.org/pdf/2501.01097

EliGen application scenarios

• Virtual scene generationEliGen can generate high-quality virtual scenes, supports entity control with arbitrary shape masks, and is suitable for scene design in virtual reality and game development.
• Character and prop designUsers can precisely control the position, shape, and style of characters and props, providing powerful tools for character and prop design in the virtual world.
• Massive Custom Data SynthesisEliGen's ability to control arbitrary-shape masks enables it to efficiently generate large-scale customized data, suitable for data augmentation and machine learning tasks.
• Product Display and PromotionEliGen can generate high-quality product display images, supporting precise entity control and stylized design to meet the needs of advertising and product design.