ParGo - A multimodal large-scale model connector jointly launched by ByteDance and Sun Yat-sen University

What is ParGo?

ParGo is an innovative multimodal large language model connector proposed by ByteDance in collaboration with Sun Yat-sen University. It improves the alignment of visual and linguistic modalities in multimodal large language models (MLLMs). By combining local and global tokens and using a carefully designed attention mask to extract local and global information respectively, it enhances the modeling of relationships between local regions while controlling the number of tokens. It considers both image details and the global perspective, overcoming the problem of neglecting details in traditional methods.

ParGo's main functions

• Efficiently connect visual and language modalitiesParGo employs a combined global and local perspective approach. Through two key modules, Partial-Global Perception Block (PGP) and Cascaded Partial Perception Block (CPP), visual features are mapped to partial and global tokens, respectively extracting local and global information from the image. This achieves an efficient connection between visual features and the Large Language Model (LLM), overcoming the problem of over-focusing on salient regions in traditional methods.
• Improve the performance of multimodal large language modelsIt performs exceptionally well in multiple MLLM benchmarks, such as achieving a 259.96% improvement over traditional Q-Former projectors in the MME benchmark. Particularly in tasks emphasizing detail perception, ParGo significantly outperforms other projectors, demonstrating more accurate text recognition, better depiction of image details, and more effective identification of local elements.
• Self-supervised learning enhances context understanding.Introducing a self-supervised learning strategy during the training phase enhances the model's understanding of context by predicting the content of occluded parts, thereby improving the model's generalization performance and reducing its dependence on large-scale labeled data.

ParGo's technical principles

• Global + Local PerspectivesParGo employs two types of learnable tokens, based on an attention mechanism, to map visual features to a large language model (LLM) from both local and global perspectives.
• Partial-Global Perception Block (PGP)In ParGo, the features of the visual encoder are mapped to two different types of tokens: Partial token and Global token, which can extract local and global information of the image, respectively.
  • Partial tokensEach token interacts with only a portion of the visual features, focusing on local information of the image.
  • Global tokensThe global token interacts with all visual features to capture global information about the image.
• Cascaded Partial Perception Block (CPP)ParGo introduced the Cascaded Partial Perception (CPP) module before the Partial-Global Perception module. The core of the CPP module is a self-attention mechanism with a specially designed mask. As the number of layers increases, each partial token can access more neighboring tokens, gradually expanding its perception range.
• Self-supervised learning strategyDuring the training phase, ParGo introduced a self-supervised learning strategy, which enhances the model's ability to understand the context by predicting the content of the occluded parts.

ParGo's project address

• Github repository:https://github.com/bytedance/ParGo
• arXiv technical paper:https://arxiv.org/pdf/2408.12928

ParGo application scenarios

• Visual question answering systemParGo can understand visual cues in images and parse semantic information in text, performing exceptionally well in visual question answering tasks.
• Image caption generationParGo excels particularly in tasks such as COCO Caption, generating high-quality image captions. It can describe global information about an image and also capture local details, producing more accurate and richer captions.
• Cross-modal retrievalParGo can be used for cross-modal retrieval tasks, helping users find relevant images through text queries or relevant text through image queries.
• Sentiment AnalysisParGo can understand sentiment information in images and text, providing more accurate results in sentiment analysis tasks. Image content understanding: ParGo can be used for image content understanding tasks, helping the system better understand details and global information in images.