ACE - An all-around image generation and editing model launched by Alibaba Tongyi Labs

What is ACE?

ACE (All-round Creator and Editor) is an all-around image generation and editing model based on diffusion transform, launched by Tongyi Lab of Alibaba Group. ACE introduces Long Context Conditional Units (LCUs) and a unified conditional format, enabling it to understand and execute natural language instructions and achieve a wide range of visual generation tasks. The ACE model supports multimodal input and can handle complex tasks including image generation, editing, and multi-turn interactions, providing a unified solution to improve the efficiency and flexibility of visual content creation.

Main functions of ACE

• Multimodal visual generationA generates images based on text instructions, supporting various visual generation tasks such as style transfer, object addition or deletion, etc.
• Image editingThe model edits existing images, including semantic editing, element editing (such as adding or removing text and objects), and inpainting.
• Long context processingBased on Long Context Conditional Units (LCUs), it understands and performs image editing tasks in multi-turn dialogues while maintaining the coherence of the dialogue history.
• Data collection and processingEmploying efficient data collection methods, pairing images are acquired based on a synthesis or clustering pipeline, and accurate text instructions are generated using a fine-tuned large-scale multimodal language model.
• Single-model multi-task processing: Avoid the cumbersome processes used in visual proxies, and use a single model backend to respond to any image creation request, thereby improving efficiency.

ACE's technical principles

• Long Context Conditional Unit (LCU)Introducing LCU, a unified conditional format, which combines historical information with current text instructions to better understand user requests and generate the desired image.
• Diffusion model based on TransformerWe construct a diffusion model based on Transformer, using LCU as input, and jointly train it for various generation and editing tasks to improve the model's multi-task processing capabilities.
• Conditional tokenizingThe model encodes text instructions and visual information (such as images and masks) into sequences and merges them to achieve alignment of multimodal information.
• Image indicator embeddingTo ensure that the image order mentioned in the text instructions matches the image sequence in the CU, the image order is indicated by predefined text tags.
• Long-context Attention BlockThe module distinguishes different spatial and frame-level image embeddings based on time-step embedding (T-Emb) and 3D rotational position coding (RoPE), ensuring that text embeddings and image embeddings are aligned frame by frame in self-attention and cross-attention layers.

ACE project address

• Project official website:ali-vilab.github.io/ace-page
• GitHub repository:https://github.com/ali-vilab/ACE/
• arXiv technical paper:https://arxiv.org/pdf/2410.00086

Application scenarios of ACE

• Artistic Creation and DesignArtists and designers can generate or edit images to realize their creative ideas and improve their creative efficiency.
• Media and EntertainmentIn filmmaking, it involves generating keyframes or creating auxiliary visual effects. In game development, it involves rapid prototyping and generating game assets.
• Advertising and MarketingMarketers can quickly generate attractive advertising images and marketing materials.
• Education and TrainingEducators create customized teaching materials and visual aids to enhance students' learning experience.
• e-commerceE-commerce platforms generate product images or provide personalized product displays based on customer needs.