ACE++ - An upgraded image generation and editing model launched by Alitongyi

What is ACE++?

ACE++ is an advanced image generation and editing tool developed by Alibaba's Tongyi Lab. Through instructional and context-aware content-filling technologies, it achieves high-quality image creation and editing capabilities. ACE++ offers multiple models for different tasks: ACE++ Portrait generates consistent portraits; ACE++ Subject maintains subject consistency across different scenes; and ACE++ LocalEditing redraws specific areas of an image while preserving its original structure. The upcoming ACE++ Fully will support even more instructional editing and reference generation tasks.

ACE++ main functions

• Image generation
  • Portrait generation:pass ACE++ Portrait The model can generate high-quality portraits based on user input while maintaining consistency in facial features and style.
  • Topic generation:ACE++ Subject The model can generate images with specific themes in different scenarios, such as placing a sign or object in different backgrounds.
• Image editing
  • Partial editing:ACE++ LocalEditing The model allows for the redrawing or modification of specific areas of an image while preserving the original image's structure and style. For example, it can modify a person's clothing, background, or the appearance of a specific object.
  • Stylized editingUsers can use commands to stylize images, such as converting ordinary photos into artistic styles or specific visual effects.
• Context-aware content fillingIt can intelligently fill in missing or modified parts based on the context information of the image, ensuring that the generated image is visually natural and consistent.
• Instruction-driven interactionUsers can control the image generation and editing process through simple natural language commands. For example, users can specify the generation of portraits in a particular style, or request the addition, deletion, or modification of a certain element in an image.
• Multitasking supportACE++ supports a variety of image processing tasks, including but not limited to:
  • Virtual try-on:pass ACE++ Subject The model achieves a virtual try-on effect.
  • Sign stickerPlace the logo or brand elements in different items or scenarios.
  • Photo restorationRepair and enhance damaged or blurry photos.
  • Movie Poster EditingGenerate or modify movie posters while maintaining consistency between characters and scenes.

Technical Principles of ACE++

• Improved Long Context Conditional Unit (LCU++)ACE++ proposes the LCU++ input paradigm, which concatenates the input image, mask, and noise along the channel dimension to form a Conditional Unit (CU) feature map, unlike the traditional sequence concatenation method. This improvement reduces interference from context-aware frameworks and lowers the model adaptation cost. The LCU++ input format can be extended to both 0-ref and N-ref tasks, enhancing the model's adaptability to different tasks.
• Two-stage training programACE++ employs a two-stage training approach. In the first stage, the model is pre-trained based on a text-to-image model, focusing on 0-ref tasks and leveraging the generative capabilities of the base model to quickly adapt to conditional inputs. In the second stage, the model is fine-tuned on all data, supporting general instructions while optimizing its ability to reconstruct input reference images and generate target images.
• Model ArchitectureThe overall architecture of ACE++ integrates the LCU++ paradigm, mapping CU feature maps to serialized markers through the x-embed layer, which then serve as input to the Transformer layer. The model training objective is to minimize the mean squared error between the predicted speed and the true speed, endowing the model with context-aware generation capabilities.
• Task support and model optimizationACE++ provides a toolkit supporting various image editing and generation tasks, including portrait consistency, subject consistency, and local editing. For common application scenarios, ACE++ trains lightweight, domain-stable fine-tuning models, such as the LoRA strategy, to improve model performance on specific tasks.

ACE++ project address

• Project official website:https://ali-vilab.github.io/ACE_plus
• Github repository:https://github.com/ali-vilab/ACE_plus
• HuggingFace model library:https://huggingface.co/ali-vilab/ACE_Plus
• arXiv technical paper:https://arxiv.org/pdf/2501.02487

Application scenarios of ACE++

• Virtual try-onUsing the ACE++ Subject model, users can place clothing or accessories on different character models to achieve a virtual try-on effect. Different clothing styles and combinations can be displayed on virtual characters, helping designers quickly evaluate design effectiveness or providing e-commerce platforms with personalized try-on experiences.
• Brand logo stickerIn product design or advertising production, ACE++ Subject models can embed brand logos or design elements into different backgrounds or objects.
• Photo editingACE++ supports various editing operations on existing photos, including style conversion, adding or deleting elements, and replacing backgrounds.
• Movie Poster EditingThe ACE++ Portrait model allows for stylization and modification of portraits in movie posters. It enables adjustments to facial expressions, clothing styles, and the addition of specific artistic effects to meet diverse film promotion needs.
• Partial editing:The ACE++ LocalEditing model can redraw or modify specific areas of an image while preserving the original image's structure and style. It can be used to repair scratches and stains in photos, or enhance specific parts of a person's image.
• Artistic Creation and DesignArtists and designers can use ACE++'s generation and editing features to quickly realize creative ideas. They can generate initial design sketches based on text descriptions or stylize existing designs, improving creative efficiency.