TextHarmony - A multimodal generative model jointly developed by ByteDance and East China Normal University

What is TextHarmony?

TextHarmony, a multimodal generative model jointly developed by East China Normal University and ByteDance, excels at understanding and generating visual text. Based on Slide-LoRA technology, the model dynamically aggregates modality-specific and modality-independent LoRA experts, partially decoupling the multimodal generation space and coordinating visual and linguistic generation within a single model instance. TextHarmony achieves a more unified generation process between visual and linguistic modalities. The research team has released a high-quality image captioning dataset, DetailedTextCaps-100K, based on advanced closed-source MLLM synthesis, further enhancing the model's visual text generation capabilities.

Main functions of TextHarmony

• Visual text understandingTextHarmony can understand text information in images and perform tasks such as scene text detection, recognition, document understanding, visual question answering (VQA), and key information extraction (KIE).
• Visual text generationThe model generates images based on text descriptions, and renders accurate and coherent text in the generated images.
• Visual text editingTextHarmony replaces or renders text at a specific location on an image while maintaining background consistency.
• Visual text perceptionThe model has basic optical character recognition (OCR) capabilities, enabling it to detect and recognize text in images.

The technical principles of TextHarmony

• Slide-LoRAA method that dynamically aggregates modality-specific and modality-independent LoRA (Low-Rank Adaptation) experts partially decouples the multimodal generation space, making the model more consistent when generating visual and linguistic modalities in a single instance.
• Multimodal pre-trainingTextHarmony is pre-trained on an image-text corpus containing rich text to learn how to generate multimodal outputs.
• Comprehensive fine-tuningBuilding upon pre-training, TextHarmony fine-tunes the model for text-centric tasks, enhancing its ability to generate multimodal expressions and follow human instructions.
• High-quality datasetsCreate the DetailedTextCaps-100K dataset, which provides detailed image caption pairs. These caption pairs focus on both visual and textual elements in image generation tasks, significantly improving the quality of generated images.
• Model ArchitectureTextHarmony's architecture comprises a visual encoder, an LLM (Large Language Model), and an image decoder. These components work together to generate visual and textual content. It maximizes conditional probabilities to generate interwoven token sequences of text and images.

TextHarmony's project address

• GitHub repository:https://github.com/bytedance/TextHarmony
• arXiv technical paper:https://arxiv.org/pdf/2407.16364

Application scenarios of TextHarmony

• Document AnalysisTextHarmony is used to automate the processing and understanding of text content in documents, such as automatically identifying and extracting key information from documents, thereby improving the efficiency of document management.
• Scene text recognitionIn fields such as urban surveillance and autonomous driving, TextHarmony can recognize and understand text information in scenarios such as street views, billboards, and traffic signs.
• Visual Question Answering (VQA)TextHarmony can answer questions about image content and has wide applications in education, entertainment, and intelligent assistants.
• Image editing and enhancementIn design and media production, TextHarmony allows you to add or modify text in images as needed, increasing the flexibility of creative work.
• Information retrievalTextHarmony helps develop smarter search engines by understanding the text content in images to provide more accurate search results.