AB
AB
AiBoss

All Categories

project

SmolDocling - A lightweight multimodal document processing model

SmolDocling (SmolDocling-256M-preview) is a high-efficiency, lightweight, multimodal document processing model. It can convert document images end-to-end into structured text, supporting the recognition of various elements such as text, formulas, and charts...

What is SmolDocling?

SmolDocling (SmolDocling-256M-preview) is a high-efficiency, lightweight, multimodal document processing model. It can convert document images end-to-end into structured text, supporting the recognition of various elements such as text, formulas, and charts. It is suitable for various document types, including academic papers and technical reports. The model has only 256M parameters, boasts fast inference speed (only 0.35 seconds per page processing time on an A100 GPU), is fully compatible with Docling, and can export to multiple formats.

Main functions of SmolDocling

  • Multimodal document conversionIt can efficiently convert image documents into structured text, supporting both scientific and non-scientific documents.
  • Rapid reasoningOn the A100 GPU, processing a single page of a document takes only 0.35 seconds and uses less than 500MB of video memory.
  • OCR and Layout RecognitionSupports optical character recognition (OCR), preserving document structure and element bounding boxes.
  • Complex element recognitionIt can recognize complex document elements such as code blocks, mathematical formulas, charts, and tables.
  • Seamless integration with DoclingSupports converting results to multiple formats (such as Markdown, HTML, etc.) and is compatible with Docling.
  • Command supportIt supports a variety of commands, such as converting pages to Docling format, converting charts to tables, and converting formulas to LaTeX.

SmolDocling's technical principles

  • Lightweight designSmolDocling-256M-preview is a visual language model containing only 256M parameters, designed specifically for document optical character recognition (OCR) and conversion. It can process documents quickly on consumer-grade GPUs, using less than 500MB of video memory, with a processing time of only 0.35 seconds per page.
  • Visual backbone networkThe model uses SigLIP base patch-16/512 as the visual backbone network, with 93M network parameters, enabling efficient processing of image input. Through pixel compression technology, each 512×512 image patch is compressed into 64 visual tags, significantly reducing computational resource requirements.
  • Text encoderSmolDocling-256M-preview uses SmolLM-2 as a text encoder with 135M encoder parameters, capable of processing text input and fusing it with visual information.
  • Multimodal fusion and outputThe model can receive multimodal inputs of images and text, and generate structured text output. It supports various document processing functions, including converting document images into structured text, extracting information from charts and tables, and converting mathematical formulas into LaTeX format.
  • Optimized dataset and training strategyThe training dataset for SmolDocling-256M-preview includes both scientific and non-scientific documents, with document comprehension accounting for 41%. A higher pixel labeling rate (4096 pixels/label) was used during training, significantly improving efficiency compared to the previous 1820 pixels/label.

SmolDocling's project address

Application scenarios of SmolDocling

  • Document Conversion and DigitizationSmolDocling-256M-preview efficiently converts image-based documents into structured text formats while preserving the original document layout and complex elements (such as code blocks, mathematical formulas, and charts). It supports multiple output formats, including Markdown and HTML, making it suitable for digital document processing.
  • Scientific and Non-scientific Document ProcessingIt can process non-scientific content (such as business documents, patent documents, etc.). It can identify and extract key information from documents, such as formulas, charts, and tables.
  • Fast OCR and Layout RecognitionSmolDocling-256M-preview provides efficient optical character recognition (OCR) capabilities, accurately extracting text from images while preserving the document's structure and element bounding boxes.
  • Mobile and low-resource device supportSmolDocling-256M-preview can run on mobile devices or in resource-constrained environments, such as smartphones or laptops.
Back