Apollo-Meta, in collaboration with Stanford University, has developed a large-scale multimodal model.

What is Apollo?

Apollo is a large-scale multimodal model (LMM) developed in collaboration between Meta and Stanford University, focusing on video understanding. Based on systematic research, Apollo reveals the key driving factors for video understanding in LMMs, introducing the phenomenon of "Scaling Consistency," which states that design decisions made on smaller models can effectively scale to larger models. The Apollo project introduces ApolloBench, an efficient benchmark for evaluating video understanding, and a series of advanced Apollo models that demonstrate superior performance at different scales, particularly in handling long videos, efficiently understanding videos lasting several hours. The Apollo-3B and Apollo-7B models outperform models with more parameters in multiple benchmark tests, marking a new advancement in video LMM research.

Apollo's main functions

• Improved video comprehensionApollo enhances the understanding of video content by using large multimodal models (LMMs), including the capture and processing of spatiotemporal features.
• Exploring Design SpaceSystematically explore the design space of video LMMs, including video sampling, architecture, data composition, training plan, etc.
• Performance optimizationBased on the discovery of the "Scaling Consistency" phenomenon, Apollo can make design decisions on smaller models and effectively transfer these decisions to larger models, reducing computational costs.
• High-efficiency assessmentIntroducing ApolloBench, an efficient video understanding evaluation benchmark, to quickly and accurately evaluate model performance.
• Model familyIntroducing the Apollo model family, which demonstrates outstanding performance across different scales, especially in processing long videos.

Apollo's technical principles

• Scaling ConsistencyThe discovery that effective design and training decisions on smaller models can be transferred to larger models is known as "Scaling Consistency".
• Video sampling strategyApollo found that frame rate (fps) sampling outperforms uniform sampling during both training and inference.
• Encoder selectionBased on experiments, the optimal single encoder and encoder combination were determined to achieve the optimal video representation.
• Token resamplingUse Perceiver Resampler to resample visual tokens, reducing the number of tokens per frame and improving model efficiency.
• Data MixingThe study investigated the mixing ratios of different text, image, and video data and found that including an appropriate amount of text data and maintaining a slight video weight resulted in the best performance.
• Training PlanBased on a multi-stage training plan, different components are gradually unfrozen to optimize the model training dynamics.

Apollo's project address

• Project official website:apollo-lmms
• GitHub repository:https://github.com/Apollo-LMMs/Apollo
• HuggingFace model library:https://huggingface.co/Apollo-LMMs
• arXiv technical paper:https://arxiv.org/pdf/2412.10360

Apollo's application scenarios

• Video content analysisIt analyzes video content, identifies objects, scenes, and events within the video, and provides support for the automatic annotation and indexing of video content.
• Video search and recommendationBased on understanding video content, we improve video search engines, providing more accurate search results and personalized video recommendations.
• Intelligent monitoringIn the field of security monitoring, it identifies abnormal behavior and provides real-time analysis and response.
• autonomous drivingApollo's video understanding capabilities are used in autonomous driving systems to help vehicles better understand their surroundings.
• Education and trainingIn the education field, we analyze instructional videos to provide personalized learning suggestions and feedback.