Qianfan-VL - Baidu's open-source visual understanding model

What is Qianfan-VL?

Qianfan-VL is a large-scale visual understanding model launched by Baidu AI Cloud Qianfan, designed for enterprise-level multimodal applications. The model is available in 3B, 8B, and 70B sizes, boasting excellent general-purpose capabilities and specialized enhancements for vertical scenarios such as OCR and education. Based on an open-source model, the model completes all computational tasks on Baidu's self-developed Kunlun P800 chip, demonstrating outstanding performance and efficiency. Qianfan-VL supports multimodal tasks such as complex graph understanding, visual reasoning, and mathematical problem-solving, providing high-precision visual understanding solutions for enterprise applications.

Main functions of Qianfan-VL

• Multi-size modelIt offers three model specifications: 3B, 8B, and 70B, to meet the diverse needs of enterprises and developers of different sizes, and is applicable to everything from real-time scenarios on the edge to complex inference computing scenarios.
• OCR and Enhanced Document UnderstandingIt possesses full-scene OCR recognition capabilities, accurately recognizing handwriting, mathematical formulas, and natural scene text, and supports structured extraction of information from cards, certificates, and invoices. Furthermore, it excels in understanding complex document layouts, automatically analyzing layout elements, accurately parsing tables and charts, and enabling intelligent question answering and structured document parsing.
• Thinking and reasoning abilityThe 8B and 70B models support the activation of thought chain capabilities through special tokens, covering a variety of complex scenarios such as complex chart understanding, visual reasoning, and mathematical problem solving. They can combine visual information with external knowledge for combined reasoning, providing clear problem-solving ideas and step-by-step demonstrations.
• general abilityIt performs well in general multimodal tasks such as object recognition, image description, and visual question answering. It supports mixed Chinese and English understanding and has good cross-modal alignment capabilities, providing strong support for intelligent applications in different scenarios.

The technical principle of Qianfan-VL

• Multimodal architectureThe 3B model is based on the Qwen 2.5 architecture, while the 8B and 70B models are based on the Llama 3.1 architecture. Both models utilize 3T Chinese and English corpora for vocabulary expansion and localization enhancement, supporting mixed Chinese and English comprehension.InternViT initialization supports dynamic block processing of images of different resolutions, with a maximum support for 4K resolution input.Achieving seamless bridging of visual and language modalities through an MLP adapter ensures the accuracy and efficiency of information delivery.
• Capability Enhancement Training Pipeline:
  • Four-stage training strategyThe model's general and domain capabilities are gradually improved through four stages: cross-modal alignment, general knowledge injection, domain-enhanced knowledge injection, and post-training.
  • High-precision data synthesis technology: Construct a large-scale data synthesis pipeline for multimodal tasks, covering core tasks such as document recognition, mathematical problem solving, graph understanding, table recognition, formula recognition, and natural scene OCR. Through refined pipeline design and intermediate process data construction, it achieves large-scale production of high-quality training data.
• Massive parallel training:based onThe three-dimensional parallel combination of data parallelism (DP), tensor parallelism (TP), and pipeline parallelism (PP) significantly improves training efficiency through dynamic load balancing, gradient synchronization optimization, and ZeRO-3 state partitioning technology.Based on Baidu's self-developed Kunlun P800 chip, the hardware separation design of communication operators and matrix multiplication operators enables parallel communication and computing, significantly improving hardware utilization.
• Inference optimizationThe model performs high-efficiency inference on chips such as Kunlun Core and GPU, supporting parallel computing on a scale of 5000 cards per task, ensuring the model's high-efficiency processing capability in practical applications.

Qianfan-VL's project address

• Project official website: https://baidubce.github.io/Qianfan-VL/
• GitHub repository: https://github.com/baidubce/Qianfan-VL
• HuggingFace model library: https://huggingface.co/collections/baidu/qianfan-vl-68d0b9b0be8575c17267c85c
• arXiv technical paper: https://github.com/baidubce/Qianfan-VL/blob/main/docs/qianfan_vl_report_comp.pdf

Application scenarios of Qianfan-VL

• OCR recognition scenariosThe model can accurately recognize text information in various documents, invoices, handwritten notes, etc., and supports multiple fonts and complex backgrounds, providing efficient solutions for enterprise document processing and data entry.
• Mathematical problem-solving scenariosIt uses visual recognition to identify math problems and perform reasoning calculations, supporting various question types such as geometry and algebra. It provides intelligent tutoring tools for the education field, helping students understand and solve math problems.
• Document understanding scenariosAutomatically parses document structure, extracts key information, supports the understanding and analysis of complex tables and charts, and improves the efficiency of enterprise document management, information retrieval and knowledge management.
• Chart analysis scenariosIt extracts and analyzes data from charts such as bar charts, line charts, and pie charts, supporting trend prediction and correlation reasoning, and providing strong support for data analysis and decision-making.