CogVLM2 - A new generation of multimodal large models launched by Zhipu AI

What is CogVLM2?

CogVLM2, developed by Zhipu AI, is a next-generation multimodal large-scale model that achieves significant performance improvements in visual and language understanding. It supports text lengths up to 8K and image inputs of 1344*1344 resolution, demonstrating powerful document image understanding capabilities. The model employs a 5 billion-parameter visual encoder and a 7 billion-parameter visual expert module, optimizing the interaction between visual and language modalities through a deep fusion strategy. This ensures that language processing capabilities are maintained while visual understanding is enhanced. The open-source version of CogVLM2 supports both Chinese and English, with a model size of 1.9 billion parameters, but approximately 12 billion parameters are activated during actual inference, showcasing its high efficiency in multimodal tasks.

Improvements to CogVLM2

The improvements of the CogVLM2 model compared to its predecessor mainly include the following aspects:

• Performance improvementCogVLM2 has shown significant performance improvements across several key benchmark tests, including OCRbench and TextVQA. For example, it has improved by 32% in OCRbench and 21.9% in TextVQA.
• Document Image UnderstandingCogVLM2 enhances the understanding of document images and question-answering capabilities, especially performing exceptionally well in the DocVQA benchmark test.
• Supports high-resolution imagesThe model supports image resolutions up to 1344*1344 pixels, enabling it to process higher-resolution images.
• Support for long textsCogVLM2 supports text inputs up to 8K, which enables the model to handle longer documents and more complex language tasks.
• Bilingual supportCogVLM2 provides an open-source model version that supports both Chinese and English, enhancing the model's multilingual capabilities.

CogVLM2 model information

CogVLM2 has released two open-source versions of CogVLM2 based on the Meta-Llama-3-8B-Instruct language base model: cogvlm2-llama3-chat-19B and cogvlm2-llama3-chinese-chat-19B. Interested users can download or try them online from GitHub, Hugging Face, or the Moda community.

CogVLM2 model architecture

The CogVLM2 model architecture has been optimized and innovated based on the previous generation model, with the following specific features:

1. Visual encoderCogVLM2 employs a visual encoder with 5 billion parameters, responsible for feature extraction and encoding of the input image.
2. Visual Expert ModuleA 7 billion-parameter vision expert module is integrated into the large language model. This module, through unique parameter settings, finely models the interaction between visual and language sequences.
3. Deep integration strategyCogVLM2 employs a deep fusion strategy, which allows the visual and linguistic modalities to be more closely integrated, thereby enhancing the model's visual understanding capabilities while maintaining its advantages in language processing.
4. MLP AdapterThe model uses an MLP (Multilayer Perceptron) Adapter to adjust and adapt features between different modalities.
5. downsampling moduleTo better process and understand high-resolution documents or web images, CogVLM2 introduces a dedicated downsampling module after the visual encoder, which effectively extracts key information and reduces the sequence length input into the language model.
6. Word EmbeddingThe model includes a Word Embedding layer, which converts text into numerical vectors that the model can understand.
7. Multi-expert module structureCogVLM2 employs a multi-expert module structure, which results in only about 12 billion parameters being activated during inference. This design ensures both model performance and improves inference efficiency.
8. Language base modelCogVLM2 uses Meta-Llama-3-8B-Instruct as its language foundation model, providing the model with powerful language understanding and generation capabilities.

CogVLM2 model performance

The CogVLM2 team conducted quantitative evaluations on a range of multimodal benchmarks, including TextVQA, DocVQA, ChartQA, OCRbench, MMMU, MMVet, and MMBeen. As shown in the table below, despite their smaller model size, CogVLM2's two models achieved state-of-the-art (SOTA) performance on multiple benchmarks; and in other performance aspects, they also reached levels comparable to closed-source models (such as GPT-4V and Gemini Pro).