SlowFast-LLaVA-1.5 - Apple's multimodal long video understanding model

What is SlowFast-LLaVA-1.5?

SlowFast-LLaVA-1.5 (SF-LLaVA-1.5 for short) is a high-efficiency video language model designed specifically for long video understanding. Based on the two-stream (SlowFast) mechanism, it balances processing more input frames with reducing the number of tokens per frame, capturing detailed spatial features and efficiently handling long-term temporal motion information. The model includes models with parameter sizes from 1B to 7B, trained using a simplified two-stage training process and high-quality public datasets. The model performs exceptionally well in long video understanding tasks, maintains strong performance in image understanding tasks, and demonstrates significant advantages with small-scale models, providing strong support for lightweight and mobile-friendly video understanding applications.

Main functions of SlowFast-LLaVA-1.5

• High-efficiency long video understandingIt can efficiently process complex spatiotemporal information in long videos, capture long temporal context, and is suitable for understanding and analyzing long video content.
• Multimodal fusionIt combines video and image inputs to provide comprehensive visual understanding capabilities, supports joint training of video and image tasks, and improves the model's performance in various visual tasks.
• Lightweight and mobile-friendlyThe model design emphasizes lightweight design, making it suitable for deployment in resource-constrained environments such as mobile devices, and meeting the needs of edge computing and real-time applications.
• Strong reasoning abilityBased on the architecture of Large Language Model (LLM), it has powerful natural language processing capabilities, and can generate detailed descriptions of video content and answer questions related to the video.
• ScalabilityIt provides models with parameter sizes ranging from 1B to 7B, allowing users to choose the appropriate model size based on their specific needs, thus achieving a balance between performance and resources.

Technical Principles of SlowFast-LLaVA-1.5

• SlowFast dual-stream mechanism:
  • Slow flowIt processes video at a lower frame rate, capturing detailed static spatial features, and is suitable for processing keyframe information in video.
  • Fast StreamIt processes video at a high frame rate, but has fewer features per frame, focusing on capturing motion information and is suitable for handling dynamic changes in video.
• Two-stage training process:
  • Phase 1 (Image Understanding)Supervised fine-tuning (SFT) using image data provides the model with general knowledge and reasoning capabilities, ensuring that the model has good basic performance on image tasks.
  • Phase Two (Joint Training of Video and Image)Building upon the first stage, joint training is conducted using image and video data to further improve the model's performance in video understanding tasks while maintaining a strong understanding capability for image tasks.
• High-quality data mixing:
  • Image dataThis includes general, text-rich, and knowledge-based datasets, such as LLaVA Complex Reasoning, ShareGPT-4v, and Coco Caption.
  • Video dataIt covers large-scale video data and long video understanding tasks, such as LLaVA-Hound, ShareGPT4Video, ActivityNet-QA, etc., ensuring that the model performs well in a variety of video tasks.
• Model ArchitectureUsing Oryx-ViT as the visual encoder and Qwen2.5 series as the language model (LLM), different projectors are designed for video and image inputs to adapt to the input characteristics of different modalities.

Project address for SlowFast-LLaVA-1.5

• GitHub repositoryhttps://github.com/apple/ml-slowfast-llava
• arXiv technical paper: https://arxiv.org/html/2503.18943v1

Application scenarios of SlowFast-LLaVA-1.5

• Understanding and summarizing long video contentAutomatically generates summaries for long videos, helping users quickly grasp the core content and saving time.
• Video Q&A SystemUsers ask questions in natural language, and the model generates accurate answers based on the content of long videos, improving the interactive experience.
• Video editing and creationAutomatically edit key segments from long videos to generate short videos, improving content creation efficiency.
• Video surveillance and analysisIt can identify abnormal behaviors in surveillance videos in real time, such as crowds gathering, thereby improving the level of intelligence in surveillance.
• Multimedia content recommendationBased on users' viewing history, recommend relevant long-form video content to enhance user engagement.