TextDiffuser-2 - An AI image and text rendering fusion framework launched by Microsoft and others.

What is TextDiffuser-2?

Text-Diffuser 2 is a newly developed text rendering method based on a diffusion model, developed by researchers from Microsoft Research, Hong Kong University of Science and Technology, and Sun Yat-sen University. It aims to address the limitations of image diffusion models in terms of flexibility, automation, layout prediction capabilities, and style diversity when generating text, thereby improving the quality and diversity of visual text in generated images.

TextDiffuser-2's innovation lies in its use of the powerful capabilities of language models to automatically plan and encode text layout, thereby increasing the diversity and visual appeal of generated images while maintaining text accuracy. Compared to the first-generation TextDiffuser, it has been improved and optimized in several aspects, such as improved layout planning, line-level text encoding, dynamic adjustment of text layout during chat interaction, optimized text rendering, and more diverse text styles.

TextDiffuser-2 official website entry

• Official project homepage:https://jingyechen.github.io/textdiffuser2/
• Hugging Face Demo:https://huggingface.co/spaces/JingyeChen22/TextDiffuser-2
• GitHub address:https://github.com/microsoft/unilm/tree/master/textdiffuser-2
• arXiv research paper:https://arxiv.org/abs/2311.16465

Features of TextDiffuser-2

• Text layout planning:It automatically infers keywords from user input suggestions and plans the text layout within images.Allows users to specify keywords and determine their location within the image. Also supports...By engaging in interactive chat with users, the text layout can be dynamically adjusted, such as by regenerating, adding, or moving text elements.
• Text layout encoding:In the diffusion model, a language model is used to encode the location and content of the text to generate a text image.Use line-level text encoding instead of character-level encoding to provide greater flexibility and stylistic diversity.
• Text Image Generation:Generate an image containing accurate and visually appealing text based on the planned text layout.It supports a variety of text styles, including handwriting and artistic fonts, to enhance the visual diversity of images.
• Text template image generation:When a template image is provided, TextDiffuser-2 can directly use existing OCR tools to extract text information and use it as a conditional input into the diffusion model, without having to predict the layout from the language model.
• Text Repair:Similar to the first-generation TextDiffuser, TextDiffuser-2 can be adapted for text inpainting tasks by training the model by modifying the input convolutional kernel channels of U-Net to fill in text regions in an image.
• Textless natural image generation:Even with fine-tuning on text data, TextDiffuser-2 retains its generative capabilities in the original domain (such as the COCO dataset), generating images without text.
• Handling overlapping layouts:TextDiffuser-2 exhibits greater robustness in handling overlapping text boxes in predicted layouts and is able to generate more accurate text images.

How TextDiffuser-2 works

1. User input:The user provides a descriptive prompt, which can be a textual description of the desired image content and layout.
2. Layout planning:Using a pre-trained large language model (such as GPT-4), fine-tuned to automatically infer text content and layout based on user prompts, the model can handle two scenarios: automatically generating text and layout without user-provided keywords; and determining the placement of keywords provided by the user.The layout information output by the language model includes the coordinates of the text lines, such as the coordinates of the top left and bottom right corners of each text line.
3. Layout coding:Based on the layout planning results, TextDiffuser-2 uses another language model to encode text layout information. This model combines user prompts and layout information to form a format suitable for the diffusion model to process.To encode text position, TextDiffuser-2 introduced special markers to represent coordinates, such as "[x5]" and "[y70]" which represent the x and y coordinate values, respectively.
4. Diffusion model training:The TextDiffuser-2 diffusion model is trained using a denoising L2 loss to learn how to generate images based on encoded text layout information. This process involves progressively guiding the generation of the target image from a state of random noise.
5. Image text generation:During the generation phase, the diffusion model generates an image based on the encoded text layout information. This process typically involves multiple steps, each of which gradually brings the generated image closer to the final text layout and content.
6. User Interaction:TextDiffuser-2 allows users to interact with the model through multiple rounds of chat to further adjust the text layout. Users can request to regenerate the layout, add or remove keywords, or move keywords to new positions.
7. Evaluation and optimization:The generated images will be evaluated to ensure the accuracy of the text and the visual quality of the images. This may include using OCR tools to assess the readability and accuracy of the text, and using user research to evaluate the aesthetics and usability of the images.