Nanonets-OCR-s - An OCR model launched by Nanonets

What is Nanonets-OCR-s?

Nanonets-OCR-s (Nanonets OCR Small) is an image-to-Markdown OCR model from Nanonets, supporting the conversion of document content from images into structured Markdown format. The model can extract text and intelligently recognize and process complex document elements such as LaTeX equations, image descriptions, signatures, watermarks, checkboxes, and complex tables. Based on a deep learning model trained on a large dataset, Nanonets-OCR-s supports various document types, including research papers, financial documents, and medical forms. The output Markdown format content can be directly processed by large language models and is widely used in academic, legal, financial, and corporate fields, significantly improving the efficiency and accuracy of document processing.

Main functions of Nanonets-OCR-s

• LaTeX Equation RecognitionAutomatically converts mathematical equations and formulas into correctly formatted LaTeX syntax, including conversions of inline mathematical expressions and displayed equations.
• Intelligent Image DescriptionUse structured tags to describe images in a document, making them processable by large language models. Supports describing the content, style, and context of single or multiple images (such as logos, charts, graphics, QR codes, etc.), predicting image descriptions in the `<img>` tag, and predicting page numbers in the `<page_number>` tag.
• Signature detection and isolationIdentifying signatures in isolated documents is crucial for legal and business document processing. The model predicts the signature text from the <signature> tag.
• Watermark ExtractionSimilar to signature detection, the model supports detecting and extracting watermark text from documents, with the predicted watermark text located in the <watermark> tag.
• Smart checkbox processingThis function converts checkboxes and radio buttons in forms to standardized Unicode symbols for consistent processing. The model predicts the state of checkboxes within the `<checkbox>` tag.
• Extracting complex tablesExtract complex tables from documents and convert them into Markdown and HTML tables.

The technical principle of Nanonets-OCR-s

• Visual-Language Model (VLM)Nanonets-OCR-s is based on a vision-language model (VLM), which simultaneously understands and processes visual information (such as images, tables, charts, etc.) and linguistic information (such as text content). The model leverages joint learning of visual and linguistic features to better understand the structure and content of documents.
• Dataset planning and trainingTo train the model, a dataset of over 250,000 pages of documents was curated, covering various document types such as research papers, financial documents, legal documents, medical documents, tax forms, receipts, and invoices. The documents contain elements such as images, charts, equations, signatures, watermarks, checkboxes, and complex tables. Training was conducted using both synthetic and manually labeled datasets. The model was first trained on the synthetic dataset and then fine-tuned on the manually labeled dataset. The synthetic dataset provides a large number of training samples, while the manually labeled dataset improves the model's performance on real-world documents.
• Base model selectionThe Qwen2.5-VL-3B model was selected as the base model for the Visual-Language Model (VLM), and fine-tuned on a curated dataset to improve its performance on document-specific optical character recognition (OCR) tasks.
• Intelligent content recognition and semantic taggingNanonets-OCR-s can identify various elements in a document and semantically tag them. Based on this method, the model converts unstructured document content into a structured, context-rich Markdown format, providing higher-quality input for downstream tasks.
• Model optimization and tuningDuring training, the model's parameters and structure are continuously optimized to improve performance across various document types and scenarios. Specific adjustments and optimizations are made to the model to meet different functional requirements, ensuring its accuracy and reliability in practical applications.

Nanonets-OCR-s project address

• Project official website: https://nanonets.com/research/nanonets-ocr-s/
• HuggingFace model libraryhttps://huggingface.co/nanonets/Nanonets-OCR-s

Application scenarios of Nanonets-OCR-s

• Thesis digitizationThis tool converts academic papers containing LaTeX equations and tables into a structured Markdown format, making it easier for researchers to organize, cite, and further analyze the literature.
• Research Data CompilationIt can quickly extract key information from research papers, such as experimental data, charts, and conclusions, making it easier for researchers to quickly review and compare them.
• Academic PublishingIt helps publishers convert academic documents in print or PDF format into formats suitable for online publication, improving the accessibility and searchability of the documents.
• Legal document analysisIt can quickly identify and extract important clauses, case citations, and legal provisions from legal documents, improving the efficiency of legal research and case analysis.
• Financial statement processingExtracting data from financial statements, such as revenue, expenses, and balance sheets, facilitates financial analysis and report generation.