MM1.5 - Apple's upgraded multimodal large model

What is MM1.5?

MM1.5 is a multimodal large-scale language model from Apple, designed to enhance rich image understanding, visual reference and localization, and multi-image reasoning capabilities. The model utilizes a data-centric training approach, achieving high performance from 1B to 30B parameters through large-scale pre-training, continuous pre-training on high-resolution OCR data, and fine-tuning with optimized visual instructions. MM1.5 includes intensive and MoE variants, demonstrating how small-scale models can achieve powerful performance through sophisticated data curation and training strategies. MM1.5 also introduces dedicated variants, MM1.5-Video and MM1.5-UI, optimized for video understanding and mobile UI understanding, providing in-depth insights into the training process and decision-making based on empirical research, guiding the future development of multimodal AI technology.

Main functions of MM1.5

• Text-rich image understandingMM1.5 can understand the text content in an image and the relationship between the text and the image content.
• Visual reference and positioningThe model identifies specific objects in images and understands references to objects in text, such as "that red ball".
• Multi-image reasoningMM1.5 can analyze multiple images, understand the relationships between them, and perform logical reasoning.
• Video UnderstandingBased on the MM1.5-Video variant, the model can understand video content, including actions, events, and time sequences.
• Understanding Mobile UIMM1.5-UI variant is specifically designed for understanding, recognizing, and manipulating interface elements in mobile applications.

Technical principles of MM1.5

• Deep learning and natural language processingBy combining deep learning-based visual models with natural language processing techniques, the model can understand and generate text related to image content.
• Coordinate tokens and visual attention mechanisms: Use coordinate tokens to locate objects in an image, and focus on specific regions of the image based on visual attention mechanisms.
• Image segmentation and multimodal fusionIt segments images into multiple parts and merges them with text information, supporting multi-image reasoning.
• Video frame sampling and timing analysisThe process involves sampling video frames, analyzing the temporal relationships between frames, and understanding the video content.
• Interface element recognitionUse image recognition technology to identify elements on a mobile interface, such as buttons and icons.

MM1.5 project address

• arXiv technical paper:https://arxiv.org/pdf/2409.20566v1

Application scenarios of MM1.5

• Image and video understandingMM1.5 can understand and analyze image and video content, and can be applied to fields such as image annotation, video content analysis, and security monitoring.
• Visual searchIn e-commerce or digital libraries, MM1.5 helps users search for specific products or documents based on descriptions or query images.
• Assisted driving and autonomous drivingIn the automotive industry, MM1.5 is used to understand and analyze road conditions and assist driving decisions.
• Smart AssistantIn smartphones and smart home devices, MM1.5 provides a more natural and intuitive way to interact, understanding users' voice or text commands.
• Education and trainingMM1.5 serves as an educational tool to help students understand complex concepts and provides a personalized learning experience.