AnyText - Alibaba's open-source visual image text generation and editing model

What is AnyText?

AnyText is a diffusion-based multilingual visual text generation and editing model developed by the Alibaba Research Institute of Intelligent Computing. It focuses on rendering accurate and coherent text in images. AnyText comprises a diffusion pipeline with two main elements: an auxiliary latent module and a text embedding module. The former uses inputs such as text glyphs, positions, and mask images to generate latent features for text generation or editing. The latter uses an OCR model to encode stroke data, which is combined with image caption embeddings from a word segmenter to generate text that seamlessly blends with the background. This technology addresses the challenge of synthesizing blurry, unreadable, or erroneous text when generating text regions in images, improving the accuracy of text writing in images.

GitHub project:https://github.com/tyxsspa/AnyText

Paper address:https://arxiv.org/abs/2311.03054

ModelScope:https://modelscope.cn/studios/damo/studio_anytext

Hugging Face:https://huggingface.co/spaces/modelscope/AnyText

AnyText Features

• Multilingual support:AnyText can generate text in multiple languages, including Chinese, English, Japanese, and Korean.
• Multiline text generation:Users can specify multiple locations on an image to generate text.
• Writing in deformed areas:AnyTextIt can generate text in horizontal, vertical, or even curved or irregular areas.
• Text editing capabilities:AnyTextIt provides the ability to modify the text content at a specified location in an image while maintaining consistency with the style of the surrounding text.
• Plug and play:AnyText can be seamlessly integrated into existing diffusion models, providing the ability to generate text.

How AnyText works

Through the collaborative work of the following modules, AnyText can accurately generate and edit multilingual text within images while maintaining seamless integration with the background, as detailed below:

1. Text-controlled diffusion pipeline:
   • AnyText uses a variational autoencoder (VAE) to encode the input image, generating a latent representation.
   • Then, noise is gradually added to this latent representation using a diffusion algorithm to generate a series of noisy latent images over time steps.
   • At each time step, AnyText applies a network (TextControlNet) to predict the noise that should be added to the noise potential image in order to control the generation of text.
2. Auxiliary latent modules:
   • This module takes text glyphs, positions, and a masked image as input and generates an auxiliary latent feature map.
   • The glyph information is generated by rendering the text onto the image, the position information marks the position of the text in the image, and the mask image indicates the image area that should be preserved during the diffusion process.
3. Text embedding module:
   • This module uses a pre-trained OCR model (such as PP-OCRv3) to extract stroke information from the text and encode it as an embedding.
   • These embeddings are combined with image title embeddings from the tokenizer to generate a fused intermediate representation, which is then mapped to the intermediate layer of UNet via a cross-attention mechanism.
4. Text perception loss:
   • During training, AnyText uses text-aware loss to further improve the accuracy of text generation.
   • This loss is achieved by comparing the text regions in the generated image and the original image, focusing only on the correctness of the text itself and excluding factors such as background, character position deviation, color, or font style.
5. Training and optimization:
   • The training objective of AnyText is to minimize the weighted sum of the text control diffusion loss and the text perception loss.
   • During training, the model balances these two losses by adjusting the weight ratio (λ).

How to generate text using AnyText

1. Access AnyTextModelScope spaceorHugging Face demo
2. Enter the prompt word, then select the desired text location.
3. Finally, click "Run" and wait for the image and text to be generated.