FireRed-OCR - Xiaohongshu's open-source visual language model for document structure parsing

What is FireRed-OCR?

FireRed-OCR is a lightweight visual language model for document structure parsing, open-sourced by the Xiaohongshu team. With only 2B parameters, it achieved a 92.94% overall score in the authoritative OmniDocBench v1.5 benchmark, surpassing large models such as GPT-5.2, Gemini-3.0 Pro, and Qwen3-VL-235B, achieving a breakthrough of "small model beating large model." The model is based on the Qwen3-VL-2B-Instruct architecture and employs a three-stage progressive training strategy. FireRed-OCR is specifically designed to solve the "structural illusion" problem in document parsing, accurately extracting complex tables, mathematical formulas, hierarchical headings, and other content, converting them into standard Markdown format.

FireRed-OCR's main functions

• Extracting complex tablesIt accurately identifies and extracts table structures from messy PDFs and scanned documents, maintaining row and column correspondence and avoiding the table misalignment problems common in traditional OCR.
• Mathematical Formula AnalysisIt accurately identifies mathematical formulas in documents and converts them into standard LaTeX or Markdown formats, ensuring the validity and readability of the formula syntax.
• Hierarchical structure restorationIt intelligently identifies heading levels (H1-H6), paragraph indentation, list symbols, etc. in a document and generates a Markdown hierarchical structure that conforms to the specifications.
• Multi-format document conversionIt supports converting various document formats such as PDF, scanned images, academic papers, and financial reports into structured Markdown text with one click.
• Anti-structural illusionBy optimizing through GRPO reinforcement learning, common document parsing errors such as content fabrication, line order disorder, and hierarchical confusion are significantly reduced.
• Multi-scenario adaptationSuitable for professional scenarios such as financial report digitization, academic paper analysis, contract document structuring, and book content extraction.
• Lightweight deploymentWith a 2B parameter scale, it supports local deployment and API calls, reducing computing costs and making it suitable for small and medium-sized enterprises and individual developers.

FireRed-OCR Technical Principles

• InfrastructureIt is based on the Qwen3-VL-2B-Instruct multimodal large model construction, inheriting its powerful visual understanding and text generation capabilities.
• Three-stage progressive training strategy:
  • Phase 1 (Multi-task pre-alignment)Simultaneously train three tasks: region detection, region recognition, and layout to Markdown, to establish the model's ability to perceive the spatial layout of documents.
  • Phase 2 (Specialized SFT)Supervised fine-tuning is performed on high-quality, standardized Markdown datasets to ensure consistent output logic and accurate hierarchical representation.
  • Phase 3 (Format Constraints GRPO)The Group Relative Policy Optimization reinforcement learning algorithm is applied to optimize output quality through a format reward mechanism.
• Four major reward mechanisms:
  • Formula syntax validity bonus: Ensures that mathematical formulas conform to LaTeX syntax specifications.
  • Table integrity bonus: Ensures that the row and column structure of the table is completely consistent.
  • Hierarchical closure reward: Verifies that Markdown heading level tags are correctly closed.
  • Text accuracy bonus: Improves text recognition accuracy and content fidelity.
• Structural hallucination inhibitionTo address common issues in document parsing such as disordered table rows, fabricated formulas, and chaotic hierarchies, a combination of format constraints and reinforcement learning is used for optimization, significantly reducing the occurrence rate of illusions.
• End-to-end optimizationIt generates structured Markdown directly from visual input, eliminating the need for the multi-stage pipeline of traditional OCR (detection → recognition → layout analysis → formatting), thus reducing error accumulation.

FireRed-OCR project address

• Github repository: https://github.com/FireRedTeam/FireRed-OCR

Application scenarios of FireRed-OCR

• Digitalization of financial reportsIt accurately extracts complex tables and financial data from listed companies' financial statements and audit reports, converts them into structured Markdown, and facilitates financial analysis and data storage.
• Analysis of academic papersIt identifies mathematical formulas, figure and table titles, and reference levels in research papers, and generates standard academic format text to aid in literature management and knowledge extraction.
• Contract document structuringConvert scanned contracts and legal documents into editable structured text, retaining the hierarchical structure and key information, thus improving the efficiency of legal document processing.
• Digitalization of books and magazinesProcess scanned books and periodicals, restore the table of contents and text layout, and quickly build a searchable digital library.
• Educational materials organizationIt analyzes formulas and tables in textbooks, test papers, and lecture notes, converting them into a structured format suitable for online learning, thus supporting content development on educational platforms.
• Archive digitizationIt helps businesses and organizations convert historical paper archives and handwritten notes into structured electronic documents, enabling permanent preservation and intelligent retrieval of archives.