mPLUG-DocOwl2 - Alibaba launches a multimodal large model for multi-page document understanding, requiring only 324 tokens per page.
mPLUG-DocOwl 2 is a multimodal large-scale language model for multi-page document understanding, developed by the mPLUG team at Alibaba Tongyi Lab. It achieves high-resolution document understanding without relying on Optical Character Recognition (OCR) technology...
What is mPLUG-DocOwl2?
mPLUG-DocOwl 2 is a multimodal large-scale language model for multi-page document understanding, developed by the mPLUG team at Alibaba Tongyi Lab. Without relying on Optical Character Recognition (OCR) technology, it achieves efficient understanding and processing of document images through high-resolution document image compression. mPLUG-DocOwl 2 has achieved a new state-of-the-art (SOTA) standard in multi-page document understanding benchmarks, consuming only 324 tokens per page of document images, reducing memory usage and initial packet time, and improving processing speed. The model's training consists of three stages: single-page pre-training, multi-page pre-training, and multi-task instruction fine-tuning. mPLUG-DocOwl 2 supports understanding single-page documents and can also handle complex problems in multi-page documents, such as cross-page content association and structural parsing.
Main functions of mPLUG-DocOwl2
- Understanding multi-page documentsExtracting and understanding information directly from multi-page document images without relying on OCR technology.
- High-resolution image processingThe high-resolution document image compression module compresses each page of document images into 324 visual tokens, reducing video memory usage and initial packet time.
- Multi-page question-and-answer capabilityAnswer questions about multi-page documents, providing detailed explanations and relevant page numbers.
- Document structure analysisIt parses and represents the hierarchical structure of multi-page documents, outputting them in JSON format for easier data processing and analysis.
- Cross-page content associationIt helps understand and connect content across pages in a multi-page document, providing an understanding of page structure.
- High-efficiency processingOn a single A100-80G GPU, it can process up to 60 pages of high-resolution document images simultaneously, improving processing efficiency.
Technical Principles of mPLUG-DocOwl2
- High-resolution document image compression (High-resolution DocCompressor)Based on low-resolution global visual features as guidance, a cross-attention mechanism is used to compress high-resolution document images into fewer visual tokens.
- Shape-adaptive CroppingThe adaptive cropping module cuts the document according to its shape and size to fit the layout of different pages.
- Visual feature extractionVisual features of each slice are extracted using a visual encoder (such as ViT), and feature merging and dimension alignment are performed using the H-Reducer module.
- Cross-attention mechanismDuring the compression process, global graph features are used as queries and slice features are used as key-value pairs. Feature compression is achieved through a cross-attention layer.
- Combining global and local visual featuresBy combining global visual features (capturing layout information) and local visual features (preserving text and image details), more accurate document understanding can be achieved.
Project address for mPLUG-DocOwl2
- GitHub repository:https://github.com/X-PLUG/mPLUG-DocOwl/tree/main/DocOwl2
- arXiv technical paper:https://arxiv.org/pdf/2409.03420v2
Application scenarios of mPLUG-DocOwl2
- Legal document analysisIt automatically parses legal documents and cases, extracts key information, and supports legal research and case preparation.
- Medical Record ManagementExtracting critical data from medical records and reports to support patient care, research, and administration.
- academic researchIt helps researchers quickly understand and summarize large amounts of literature, accelerating scientific discovery and knowledge innovation.
- Financial Report AnalysisAutomates the processing of annual reports, financial statements, and other financial documents, extracting key financial metrics and trends.
- Government document processingAutomated processing of government announcements, regulations, and policy documents improves the efficiency of government services.