Qianfan-OCR - Baidu Qianfan's end-to-end intelligent document model

What is Qianfan-OCR?

Qianfan-OCR is an end-to-end intelligent document model launched by Baidu Qianfan. Based on a 4B parameter visual language architecture, it integrates document parsing, layout analysis, text recognition, and semantic understanding. The model ranked first among end-to-end models in the OmniDocBench v1.5 benchmark with a score of 93.12. It achieves explicit modeling of page structure through a Layout-as-Thought mechanism and supports the understanding of complex tables and charts. The model is open source and can be efficiently deployed with a single A100 GPU.

Main functions of Qianfan-OCR

• Document Image AnalysisThe model supports extracting structured text content directly from scanned documents or images without preprocessing.
• Page layout analysis and understandingIt supports automatically recognizing elements such as titles, paragraphs, tables, and charts in a document and their spatial relationships.
• Text recognition and conversionIt accurately converts printed or handwritten text in images into editable text.
• Key information extractionIt supports locating and extracting specific fields from complex documents, such as dates, amounts, and names.
• Chart reasoning analysisThe model can understand the numerical meaning and trends of visualizations such as bar charts and line charts.
• Multi-format outputSupports generating structured data formats such as Markdown, JSON, and HTML.

The technical principle of Qianfan-OCR

• End-to-end unified architectureQianfan-OCR employs a unified visual language architecture, replacing the traditional multi-stage "detection-recognition-understanding" pipeline with an end-to-end approach. The model directly maps document images into structured output, avoiding error accumulation and visual information loss caused by staged processing.
• Layout-as-Thought mechanismTo address the lack of explicit layout modeling in end-to-end models, the team introduced the Layout-as-Thought mechanism. Before outputting the final result, the model uses... <think> The token triggers the structural thinking stage, first generating layout information such as element position, type, and reading order. Based on this prior knowledge, the content is parsed, possessing both structural awareness and semantic understanding capabilities within a unified framework.

Key information and usage requirements of Qianfan-OCR

• Model size4B Parameter Visual Language Architecture
• Evaluation resultsOmniDocBench v1.5 end-to-end model ranked first (93.12 points), surpassing Gemini 3-Pro on the KIE leaderboard.
• Core InnovationIt adopts the Layout-as-Thought mechanism, supporting explicit modeling of page layout structure.
• Deployment performanceSingle A100 GPU (W8A8 quantization) throughput: 1.024 pages/second
• Open source statusThe model weights have been released to HuggingFace, along with the accompanying Skills toolchain.
• Hardware environmentIt is recommended to equip an NVIDIA A100 or equivalent GPU for inference deployment.
• Software DependencyRequires installation of the vLLM inference framework, supporting W8A8 quantization to reduce VRAM usage.
• Access method: Can be invoked online via Baidu Qianfan platform, or deployed privately based on open-source authority.
• Input formatSupports common document and image formats (PDF, PNG, JPG, etc.)
• Output formatConfigure structured output such as Markdown, JSON, and HTML according to requirements.

Qianfan-OCR's core advantages

• Leading architectureThe model adopts an end-to-end unified visual language architecture, replacing the traditional multi-stage pipeline, eliminating the accumulation of errors between modules and greatly simplifying the complexity of system deployment and operation.
• Page layout understanding: A unique Layout-as-Thought mechanism, through <think> Tokens explicitly model the position, type, and reading order of document elements, significantly improving the parsing accuracy in complex layout scenarios.
• Top performanceIt ranked first among end-to-end models in the OmniDocBench v1.5 benchmark with a score of 93.12, and achieved the best scores in 5 out of 6 tasks, including graph understanding.
• Outstanding efficiencyA single A100 GPU, combined with W8A8 quantization, can achieve a throughput of 1.024 pages per second, saving the cost of CPU detection and heterogeneous orchestration of multiple models compared to traditional solutions.
• Ready to use right out of the boxSupports online invocation of Baidu Qianfan platform and private deployment of HuggingFace open source weights, providing a complete Skills toolchain and multi-format output capabilities.

How to use Qianfan-OCR

• Online callAccess the Baidu Qianfan platform console, select the Qianfan-OCR built-in model in the model center, create an application to obtain the API Key, and then upload the document image through the standard HTTP interface to obtain the structured parsing results in real time.
• Private Deployment Download the open-source model weights from HuggingFace, install the vLLM inference framework and configure the W8A8 quantization parameters, start the model service on a server equipped with an A100 GPU, and implement offline calls via the local API.
• Toolchain Integration Clone the official GitHub Skills repository, perform secondary development based on the provided document intelligence toolkit, embed OCR capabilities into existing business systems, and support custom output formats and batch document processing workflows.

Qianfan-OCR project address

• GitHub repository: https://github.com/baidubce/Qianfan-VL
• HuggingFace model libraryhttps://huggingface.co/baidu/Qianfan-OCR
• arXiv technical paper: https://arxiv.org/pdf/2603.13398

Comparison of Qianfan-OCR with similar competing products

Application scenarios of Qianfan-OCR

• Enterprise document digitizationSupports batch processing of scanned documents such as contracts, invoices, and reports, automatically extracting key fields and generating a structured database.
• Financial Bill ReviewThe model can identify amounts, dates, and account information in bank statements, insurance policies, and account statements to assist in risk control and compliance review.
• Medical record managementIt analyzes symptoms, diagnoses, and medication records in handwritten or printed medical records to enable rapid archiving and retrieval of electronic medical records.
• Academic paper processingConvert PDF documents to Markdown format, preserving formulas, charts, and citation structure, facilitating knowledge base construction.
• Archival and historical document restorationThe model can recognize text in low-quality images such as ancient books and old newspapers, assisting in the digital preservation of cultural heritage.