Mini-Monkey - A multimodal AI model jointly developed by Huazhong University of Science and Technology and South China University of Technology

What is Mini-Monkey?

Mini-Monkey is a lightweight multimodal AI model jointly developed by Huazhong University of Science and Technology and South China University of Technology. Employing a multi-scale adaptive segmentation strategy (MSAC) and a scale compression mechanism (SCM), it effectively addresses the jagged edges caused by traditional image segmentation, improving performance in high-resolution image and document understanding tasks. Mini-Monkey has achieved excellent results in multiple benchmark tests, demonstrating strong potential in multimodal understanding and document intelligence.

Mini-Monkey Features

• Multi-scale adaptive segmentation strategy (MSAC)By generating representations at different scales, the model is allowed to select unsegmented objects, thereby improving the ability to recognize small or irregularly shaped objects.
• Scale compression mechanism (SCM)A training-free and parameter-free mechanism to reduce computational overhead by extracting key visual features by focusing on visual markers at the detail layer.
• High-resolution image processingIt optimizes the processing capabilities for high-resolution images and reduces semantic incoherence issues caused by image segmentation.
• Document comprehension taskIt performs exceptionally well in document comprehension, effectively avoiding the problems associated with traditional segmentation strategies when handling small text and irregular shapes in documents.

Mini-Monkey's technical principles

• Multi-scale adaptive segmentation strategy (MSAC):The strategy provides features at different scales by segmenting the image into grids of different scales and layering them according to the aspect ratio of the grids.The detail layer provides high-resolution image details, while the adaptive layer avoids repeated cutting on the same object based on the aspect ratio generated by the detail layer, ensuring semantic consistency between different layers.
• Scale compression mechanism (SCM):SCM is used to reduce the computational overhead of MSAC by generating attention maps to identify and extract key visual features.Visual tokens are selected using the first few layers of a pre-trained large language model (LLM), without the need for additional training or parameters.
• Attention weight allocation:In SCM, visual tokens of different importance are distinguished by assigning high attention weights and low attention weights.Tokens with low attention weight will be filtered out, while important tokens will be retained for further processing.
• Shared LLM layer:SCM uses LLM block layers to share computations for processing different visual tokens, which helps improve the model's efficiency and performance.

Mini-Monkey's project address

Mini-Monkey Application Scenarios

• Document UnderstandingWhen processing document images, it can accurately identify and understand the text content in the document, including complex documents such as ancient books and handwritten notes.
• Image recognitionIn scenarios where it is necessary to identify small or irregularly shaped objects in images, such as medical image analysis and satellite image interpretation, Mini-Monkey can provide more accurate recognition results.
• Multimodal information processing: Scenarios that combine image and text information for comprehensive analysis, such as social media content analysis and evaluation of the matching degree between advertising images and copy.
• Content generationIn applications that require generating descriptions or stories based on image content, Mini-Monkey can provide richer and more accurate text content.
• Decision Support SystemIn decision support systems that require rapid processing and analysis of large amounts of visual data, such as security monitoring and traffic management.