Molmo 2 - Ai2 open-source AI video analysis model

What is Molmo 2?

Molmo 2 is an open-source video analytics model developed by the Allen Institute for Artificial Intelligence (Ai2), focusing on video understanding, pointing, and tracking. Based on the Qwen 3 and Olmo architectures, the model boasts powerful video analysis capabilities, accurately locating events, tracking multiple targets, and generating detailed captions within videos. Molmo 2 outperforms several open-source and closed-source models (such as Gemini 3) in video tracking and question answering tasks, while also providing efficient inference capabilities. The model has a rich training dataset covering various video and image tasks, supporting research and educational applications and contributing to the development of multimodal intelligence.

Main features of Molmo 2

• Video Comprehension and Q&AIt can provide an in-depth understanding of video content and answer various questions related to the video, including descriptive questions and event reasoning questions.
• Video pointing and positioningIt supports precise spatial and temporal location of specific events or objects in a video, such as answering "when and where a certain event occurred".
• Multi-target trackingTrack multiple targets in a video and maintain stable tracking when a target is occluded or re-enters the frame.
• Dense video subtitle generationGenerate detailed and descriptive captions for videos, providing searchable narrative content for long videos.
• Anomaly and artifact detectionDetects abnormal events in videos or generates artifacts in videos, such as inconsistent lighting or errors in object geometry.
• Multiple image input supportIt supports single images, multiple images, and video clips of varying lengths as input, providing flexible multimodal processing capabilities.
• Cross-modal reasoningIt combines visual and linguistic information for reasoning, supporting complex multimodal tasks such as image or video queries based on text descriptions.

The technical principles of Molmo 2

• Model ArchitectureThe architecture of Molmo 2 consists of a visual encoder, a language model (LLM), and a connector. The visual encoder converts input image or video frames into visual tags, extracting spatial and temporal information; the language model, based on Qwen 3 or Olmo, processes visual tags and textual information to enable cross-modal reasoning; the connector interweaves visual tags with timestamps, image indices, and text, enabling the model to jointly process spatial, temporal, and linguistic information.
• Two-stage trainingMolmo 2 employs a two-stage training approach. The first stage involves pre-training through image captioning and image pointing tasks to enhance the alignment and localization capabilities of visual and linguistic modalities. The second stage involves supervised fine-tuning on multimodal datasets, covering image, multi-image, video, and plain text tasks, to further improve the model's generalization ability.
• Data processing and samplingIn video input, Molmo 2 samples up to 128 frames at a low frame rate (≤2fps) and encodes them through a Vision Transformer. Visual markers are pooled by time windows (e.g., 3×3), interleaved with text and temporal information, and then input into the language model, supporting cross-frame visual marker interaction.
• Optimization technologyMolmo 2 employs a label-weighting scheme during the fine-tuning phase to balance learning across different tasks and improve model performance in multi-task scenarios. Simultaneously, it introduces sequence packing and message tree scheduling to increase throughput, and enhances localization and tracking capabilities through a bidirectional attention mechanism between visual labels.
• Dataset and Task DesignMolmo 2 has built a multimodal dataset with over 9 million samples, covering tasks such as dense captioning generation, video question answering, localization, and tracking. The training data includes a mix of tasks such as image captioning, video QA, pointing, and tracking, improving the model's adaptability in complex scenarios.

Molmo 2 project address

• Project official websitehttps://allenai.org/blog/molmo2
• GitHub repository:https://github.com/allenai/molmo2
• HuggingFace model libraryhttps://huggingface.co/collections/allenai/molmo2
• Technical PapersLink: https://www.datocms-assets.com/64837/1765901660-molmo_v2_2026-techreport-3.pdf

Application scenarios of Molmo 2

• Autonomous driving and traffic monitoringMolmo 2 can be used for traffic flow analysis, accident detection and early warning, improving traffic management and the safety and efficiency of vehicle operation.
• Robotics and AutomationIn robot vision navigation and industrial automation inspection, it helps robots better understand the environment and detect product quality problems.
• Scientific Research and EducationIt assists in the analysis of scientific experiments and the development of educational tools, providing researchers and students with detailed analysis and understanding of dynamic processes.
• Content creation and mediaIt automatically generates video subtitles and assists in video editing, improving the efficiency and accessibility of content creation.
• Security and surveillanceReal-time detection of abnormal behavior and personnel tracking in security monitoring ensures the safety of public places and specific areas.