GLM-OCR - A lightweight multimodal OCR model from Zhipu Open Source

• General text recognitionSupports photos, screenshots, scanned documents, and PDFs; recognizes special text such as printed text, handwritten text, seals, and code.
• Complex table parsingIt accurately understands the structure of merged cells, multi-level headers, etc., and directly outputs HTML code without the need for secondary table creation.
• Information Structure ExtractionIt intelligently extracts key fields from cards, receipts, and forms, outputs them in standard JSON format, and integrates with business systems.
• Formula and code recognitionIt supports accurate recognition of professional and technical content such as mathematical formulas and program code.
• Multilingual and mixed-language supportSupports handling complex layouts such as vertical text and mixed multilingual text.
• Batch document processingIt supports batch document recognition and outputs well-formatted data, providing a high-quality data foundation for RAG.

GLM-OCR Technical Principles

• Overall ArchitectureGLM-OCR adopts the classic "encoder-decoder" architecture design, which is inherited from the GLM-V series. The architecture consists of three core modules: the CogViT visual encoder (400M parameter scale) on the vision side, the lightweight connection layer responsible for cross-modal information fusion, and the GLM-0.5B language decoder on the back end.
• Visual encodingThe visual encoder adopts the CogViT architecture developed by Zhipu, and introduces the CLIP contrastive learning strategy for large-scale pre-training on billions of text-image pairs. This enables the model to have powerful text detection and layout semantic understanding capabilities, and can effectively handle challenges such as multi-column layouts, mixed text and images, and rotated text in complex documents.
• Cross-modal fusionTo achieve efficient fusion of visual and linguistic information, GLM-OCR employs a lightweight and efficient connection layer structure. By integrating the SwiGLU activation mechanism and introducing a 4x downsampling strategy, it accurately filters and retains key visual tokens, efficiently compressing and transmitting high-density visual semantic information to the backend language decoder, supporting high-precision OCR recognition output.
• Training optimizationGLM-OCR pioneered the introduction of Multi-Token Prediction Loss (MTP) into OCR model training. By simultaneously predicting multiple future tokens, it enhances the loss signal density, significantly improving model learning efficiency. Through continuous and stable full-task reinforcement learning training, it further optimizes the model's overall recognition accuracy and cross-domain generalization ability in complex document scenarios.
• Reasoning processAt the system level, GLM-OCR adopts a two-stage technical paradigm of "layout analysis → parallel recognition". Based on PP-DocLayout-V3, it performs document layout analysis to accurately locate areas such as text, tables, and images; and performs OCR recognition in parallel, ultimately achieving stable, high-quality, and efficient parsing results in document scenarios with diverse layouts and complex structures.

GLM-OCR project address