GOT-OCR2.0 - An open-source end-to-end OCR model supporting multilingual and multimodal recognition, and diverse input/output.
GOT-OCR 2.0 is an advanced optical character recognition (OCR) model that propels OCR technology into the 2.0 era. The GOT-OCR 2.0 end-to-end model consists of a high-compression encoder and a long context decoder, capable of processing text, ...
What is GOT-OCR2.0?
GOT-OCR 2.0 is an advanced Optical Character Recognition (OCR) model, propelling OCR technology into the 2.0 era. The end-to-end GOT-OCR 2.0 model consists of a high-compression encoder and a long context decoder, capable of processing a variety of optical characters, including text, mathematical formulas, molecular formulas, charts, musical scores, and geometric figures. GOT-OCR 2.0 supports multiple languages, especially Chinese and English, and can output results in various formatting formats, such as Markdown and LaTeX. The model features interactive OCR capabilities, including region-level recognition and dynamic resolution strategies, multi-page OCR technology, and is suitable for high-resolution images and batch document processing. GOT-OCR 2.0 has 580M parameters and a model size of 1.43GB, providing an accurate and efficient OCR solution.
Main functions of GOT-OCR2.0
- Multilingual and multimodal recognitionIt supports text recognition in multiple languages, including Chinese and English, as well as handwritten and printed text.
- Diverse input/outputIt can handle various input formats such as photos, documents, and slices, and supports output formats such as plain text, Markdown, TikZ, SMILES, and Kern.
- Long text processingThe decoder supports tokens up to 8K in size, making it suitable for processing long text documents such as academic papers and legal documents.
- Interactive OCR functionRegion-level recognition guided by coordinates or colors provides a more flexible user experience.
- Dynamic resolution strategyIt adapts to ultra-high resolution images, such as large posters or stitched PDF pages, while maintaining recognition accuracy.
- Multi-page OCR technologyBatch processing of multi-page documents improves the efficiency of processing long PDF files or documents with many images.
Technical Principles of GOT-OCR2.0
- Encoder-decoder architecture:
- encoder: Responsible for compressing the input image into a series of image tokens, which capture visual information in the image.
- decoder: Receives the image token output from the encoder and converts it into text output. The decoder supports long context and can handle long text.
- High compression rate encoderThe encoder compresses a 1024×1024 pixel image into a 256×1024 image token, which helps in processing high-resolution images.
- Long Context DecoderThe decoder supports token sequences up to 8K and can handle long documents containing a large amount of text.
- Multi-stage training strategy:
- Pre-training phaseThe encoder is pre-trained on a large amount of text data to learn a visual representation of the text.
- Joint training phaseThe encoder is trained together with the new decoder to adapt to a wider range of OCR tasks.
- Post-training phaseFurther training of the decoder supports advanced features such as fine-grained OCR, dynamic resolution, and multi-page OCR.
Project address for GOT-OCR2.0
- GitHub repository:https://github.com/Ucas-HaoranWei/GOT-OCR2.0
- HuggingFace model library:https://huggingface.co/stepfun-ai/GOT-OCR2_0
- arXiv technical paper:https://arxiv.org/pdf/2409.01704
Application scenarios of GOT-OCR2.0
- Document digitizationConvert paper documents (such as books, manuscripts, legal documents, academic papers, etc.) into electronic formats for easy storage, retrieval, and editing.
- Scene text recognitionIt identifies and extracts text in natural scenes, such as street signs, billboards, and menus.
- Bill processingAutomatically recognizes and extracts text information from invoices, receipts, and bills, simplifying financial and accounting processes.
- Authentication and securityIn scenarios where personal identity verification is required, such as banking transactions and airport security checks, it can identify information on passports, ID cards, or driver's licenses.
- Logistics and TransportationIt automatically identifies barcodes and address information on packages, improving the efficiency of logistics sorting and delivery.
- Medical Record Management: Identify and digitize doctors' handwritten prescriptions, medical records, and other medical documents.