AB
AB
AiBoss

All Categories

project

ERNIE-4.5-VL - Baidu Wenxin's open-source next-generation multimodal AI model

ERNIE-4.5-VL is a new generation of multimodal AI model open-sourced by Baidu Wenxin. Based on the ERNIE-4.5-VL-28B-A3B architecture with 3B activation parameters, it focuses on visual language understanding and cross-modal inference. The model is trained using massive amounts of visual-language data...

What is ERNIE-4.5-VL?

ERNIE-4.5-VL is a new generation of multimodal AI model open-sourced by Baidu Wenxin. Based on the ERNIE-4.5-VL-28B-A3B architecture with 3B activation parameters, it focuses on visual language understanding and cross-modal inference. The model strengthens semantic alignment through training with massive amounts of visual-language data and employs multimodal reinforcement learning to improve stability. It adds visual localization and "image thinking" functions, supporting scenarios such as document parsing and video analysis. The model supports 2-bit lossless quantization and a 128K ultra-long context window, optimizing memory usage and inference speed, and can handle extremely long text content.

Main functions of ERNIE-4.5-VL

  • Multimodal understanding and generationERNIE-4.5-VL can process text and image information simultaneously, enabling functions such as image description generation, image-text question answering, and image classification, and supporting the comprehensive understanding and creation of multimodal content.
  • Document and Chart UnderstandingThe model possesses powerful document parsing capabilities, capable of handling complex charts, flowcharts, and other data, and is widely used in engineering drawing understanding and data analysis in manufacturing, scientific research, and other fields.
  • Intelligent Reasoning and Decision MakingIt supports complex multimodal reasoning tasks, such as visual question answering and logical reasoning, and can perform in-depth analysis and decision-making by combining text and image information.
  • Multilingual interactionIt supports interaction in more than 100 languages and is suitable for cross-language multimodal tasks, such as multilingual image description and cross-language text and image question answering.
  • Long text processingIt features an ultra-long context window (supporting up to 128K), enabling it to handle large-scale text content and is suitable for long document analysis and complex task processing.
  • Image-based thinking functionIt supports the use of tools such as image zoom and search, enhancing the interactivity and flexibility of the model in visual tasks.
  • Efficient Deployment and Optimization: Through 2-bit quantization technology, it significantly reduces memory usage and improves inference speed, supports single-card deployment, and is suitable for resource-constrained scenarios.

ERNIE-4.5-VL model version

  • ERNIE-4.5-21B-A3B-Thinking
    • Parameter size21 billion parameters, with 3 billion parameters activated each time.
    • FeaturesIt supports different thinking modes, has stronger reasoning ability, and is suitable for handling complex multimodal tasks.
    • Context lengthSupports 128k context length.
  • ERNIE-4.5-VL-28B-A3B
    • Parameter size28 billion parameters, with 3 billion parameters activated each time.
    • ArchitectureIt adopts a heterogeneous hybrid expert architecture (MoE) that includes text, vision and shared experts, and improves the performance of vision tasks through a modal isolation routing mechanism.
    • performanceIt performs exceptionally well in multimodal tasks, particularly in visual perception, document and chart understanding.
    • Context lengthSupports 32k context length.
  • ERNIE-4.5-Turbo-VL-Preview
    • FeaturesThe image understanding, creation, translation, and coding capabilities have been significantly improved, and for the first time, a 32k context length is supported, with a significant reduction in first token latency.
    • Context lengthSupports 16k context length.
  • ERNIE-4.5-Turbo-VL-32K-Preview
    • FeaturesIt further improves performance based on ERNIE-4.5-Turbo-VL-Preview and supports longer context lengths.
    • Context lengthSupports 32k context length.
  • ERNIE-4.5-VL-424B-A47B
    • Parameter size424 billion parameters, 47 billion parameters activated.
    • performanceIt performs exceptionally well in multimodal evaluation benchmarks, especially in challenging reasoning tasks.
    • FeaturesIt supports thinking modes, has strong reasoning capabilities, and also maintains good performance in perception tasks.
  • ERNIE-4.5-0.3B
    • Parameter sizeOnly 300 million parameters.
    • FeaturesLightweight model, suitable for deployment on edge devices, fast inference speed, and performance close to that of large models.

Technical Principles of ERNIE-4.5-VL

  • Heterogeneous Hybrid Expert Architecture (MoE)ERNIE-4.5-VL employs a heterogeneous hybrid expert architecture, dividing parameters into text experts, vision experts, and shared experts. This architecture allows the model to dynamically select the optimal combination of experts when handling different modalities, improving computational efficiency and performance.
  • Modal isolation routing mechanismBy separating the processing paths for text and images and introducing router orthogonal loss and multimodal balance loss, the model achieves decoupled training of text and visual modalities, significantly improving the performance of visual tasks.
  • 2-Bit lossless quantization technologyIt employs a "convolutional code quantization" algorithm to achieve lossless inference with 2-bit precision. This allows the model to maintain performance while significantly reducing memory usage and increasing inference speed, supporting more efficient deployment.

Project address for ERNIE-4.5-VL

  • Github repositoryhttps://github.com/PaddlePaddle/ERNIE
  • HuggingFace model libraryhttps://huggingface.co/collections/baidu/ernie-45
  • Technical Papers: https://yiyan.baidu.com/blog/publication/ERNIE_Technical_Report.pdf

Application scenarios of ERNIE-4.5-VL

  • Document and Chart UnderstandingIt is used to parse complex engineering drawings, scientific charts and business documents, helping users quickly extract key information. It is applied in manufacturing, scientific research and finance and other fields.
  • Smart meter maintenanceEmbedded in smart meter devices, it generates text descriptions of power consumption anomalies in real time and reports faults, improving operation and maintenance efficiency and accuracy.
  • Multilingual interactionIt supports interaction in more than 100 languages and is suitable for cross-language tasks such as text and image Q&A and image description, facilitating global applications.
  • Intelligent Customer ServiceBy combining text and image information, we can provide users with more precise customer service support, such as handling product inquiries and troubleshooting.
  • Content creationGenerates high-quality graphic and text content, such as image descriptions and creative copy, suitable for the advertising, media, and creative industries.
  • Education: To assist teaching, using a combination of text and graphics to help students better understand and learn knowledge, such as explaining complex scientific charts.
Back