StreamBridge - A large-scale video language model framework jointly developed by Apple and Fudan University.

What is StreamBridge?

StreamBridge, a video-LLMs framework jointly developed by Apple and Fudan University, helps AI understand live video streams in real time. The framework supports long-contextual interactions based on memory buffers and round-decay compression strategies, and introduces a lightweight activation model to achieve proactive response functionality. The research team released the Stream-IT dataset containing approximately 600,000 samples to enhance streaming video understanding capabilities. Tests on mainstream offline models such as LLaVA-OV-7B, Qwen2-VL-7B, and Oryx-1.5-7B show that StreamBridge significantly improves the model's ability in multi-round real-time understanding and proactive response, demonstrating strong potential in the field of streaming video understanding.

StreamBridge's main functions

• Multi-turn real-time understandingIt supports multi-turn interactions with long contexts, preserving historical visual and dialogue contexts when processing the latest video clips.
• Active responseThe model can proactively monitor video streams like a human and provide timely feedback without explicit instructions.
• Flexible integrationIt supports seamless integration into existing video language models without requiring large-scale modifications to the underlying model.
• Data supportIt provides Stream-IT, a large-scale streaming video understanding dataset containing approximately 600,000 samples, and supports diverse instruction formats for training and optimizing models.

StreamBridge's technical principles

• Memory bufferIt stores and retrieves the embedding information of video frames, supporting multi-turn interactions. Each new video frame is independently encoded and appended to the buffer. When a user query is received, the contents of the buffer are flattened into a single input embedding sequence, which is then fed into the language model for response generation.
• Round decay compression strategyBefore each response is generated, if the length of the input embedding exceeds a predefined maximum length, the model merges visual tags frame by frame, starting from the earliest dialogue turn, until the total length is below the maximum length. The merging operation is implemented based on average pooling to ensure that the most recent visual context is preserved.
• Lightweight activation modelThe activation model is a standalone, lightweight multimodal large language model (MLLM) that runs in parallel with the main video large language model. The activation model receives the current frame (and the first few frames for user queries and options) as input and outputs a binary signal indicating whether the main model generates a response. The activation model uses a scorehead for binary classification (response or not) and introduces a learnable activation label <ACT> during training to supervise activation timing.
• Stream-IT datasetThis dataset, consisting of approximately 600,000 samples, is designed to simulate real-time user interaction by generating multi-turn question-and-answer sequences from a large-scale video caption corpus. It supports various task formats, such as dense video captioning, sequential step recognition, and video-based question answering.

StreamBridge project address

• arXiv technical paper:https://arxiv.org/pdf/2505.05467

Application scenarios of StreamBridge

• Real-time video interactionEnhance the real-time interactive experience in scenarios such as video conferencing and online education.
• Autonomous driving assistanceReal-time processing of road condition video to assist autonomous driving decision-making.
• Intelligent monitoringReal-time analysis of surveillance video to quickly detect abnormal behavior.
• Robot VisionIt helps robots understand their environment in real time and achieve natural interaction.
• Content creationFunction: To assist in video creation and editing, and provide real-time content analysis.