mPLUG-DocOwl 1.5 - Alibaba's open-source multimodal large-scale language model

What is mPLUG-DocOwl 1.5?

mPLUG-DocOwl 1.5 is a multimodal large-scale language model launched by Alibaba Group, focusing on OCR-free document understanding. Based on unified structure learning, the model enhances its ability to understand the structural information of rich textual images such as documents, tables, and charts. mPLUG-DocOwl 1.5 includes structure-aware parsing tasks and multi-granularity text localization tasks, covering five domains: documents, web pages, tables, charts, and natural images. The H-Reducer module of mPLUG-DocOwl 1.5 merges horizontally adjacent image patches based on convolutional layers, reducing visual feature length while preserving layout information, allowing the model to efficiently process high-resolution images. In multiple visual document understanding benchmarks, the model demonstrates industry-leading OCR-free performance, improving state-of-the-art performance by more than 10 points.

Main features of mPLUG-DocOwl 1.5

• Structure-aware document parsingIt identifies and parses the text structure in a document, such as line breaks and spaces, to understand how the document is organized.
• Table to MarkdownConvert table images to Markdown format for easier processing and reading.
• Chart to MarkdownConvert chart images to Markdown format while preserving key data and structural information.
• Natural Image AnalysisIt analyzes images in natural scenes to identify and understand textual information within them.
• Multi-granular text positioning: Locate text at different granular levels (words, phrases, lines, blocks) to enhance the model's ability to identify text locations.

Technical Principles of mPLUG-DocOwl 1.5

• Unified Structure LearningBased on structure-aware parsing and multi-granularity text localization tasks, the model learns how to understand and process text-rich images.
• H-Reducer Visual-Text ModuleBy merging horizontally adjacent visual features through convolutional layers, feature length is reduced while maintaining layout information, enabling large language models to process high-resolution images more effectively.
• Multimodal Large Language Model (MLLM)Combining a visual encoder and a large language model, using vision-to-text modules (such as H-Reducer) allows the model to understand and generate language descriptions related to visual content.
• Training on large datasetsUsing large-scale labeled datasets, such as DocStruct4M and DocReason25K, the model can learn textual structure and semantic information from various documents and images.
• Two-stage training frameworkFirst, a unified structure is learned, and then multi-task adjustments are made to enable the model to perform well in various downstream tasks.

Project address for mPLUG-DocOwl 1.5

• GitHub repository:X-PLUG/mPLUG-DocOwl/tree/main/DocOwl1.5
• arXiv technical paper:https://arxiv.org/pdf/2403.12895

Application scenarios of mPLUG-DocOwl 1.5

• Automated document processingIn enterprises or government agencies, automating the parsing and understanding of large amounts of documents, such as contracts, invoices, reports, and forms, improves work efficiency and reduces human intervention.
• Intelligent Search EngineIntegrating mPLUG-DocOwl 1.5 into the search engine enhances the ability to search for text content within images, providing more accurate search results.
• Aid reading and comprehensionIt helps users better understand the content of complex documents, especially for visually impaired individuals, by providing easily accessible information based on parsing the document structure.
• Education and academic researchIn the field of education, it helps students and researchers understand complex information in textbooks, academic papers, and research materials.
• Customer service and supportIn the customer service system, mPLUG-DocOwl 1.5 is used to parse user-uploaded documents, automatically extract key information, and provide faster service response.