MOSS-VL-Realtime - OpenMOSS open-source visual language model - AiBoss

What is MOSS-VL-Realtime?

MOSS-VL-Realtime is an open-source 11-parameter streaming visual language model from OpenMOSS, designed specifically for real-time video understanding. The model supports on-the-fly responses, real-time correction, and active silencing, upgrading video understanding from watching recordings to watching live streams. Through a dual-channel cross-attention architecture and absolute timestamp encoding, the model achieves parallel perception and generation, keeping AI synchronized with the physical world. It achieves state-of-the-art performance in multiple streaming benchmarks.

Main functions of MOSS-VL-Realtime

Real-time video understandingIt continuously receives video streams and generates answers while watching, without waiting for the video to end.
Instant Q&A ResponseUsers can insert questions at any time, and the model will answer them instantly based on the current frame.
Active silence mechanismWhen there is insufficient information or no key events, the model autonomously remains silent and waits.
Dynamic answer correctionWhen the scene changes, the model adjusts and corrects its previous answers in real time.
Spatiotemporal positioning perception: By using absolute timestamp encoding, we can accurately perceive the physical time flow and spatial location.
Long context processingSupports continuous understanding and interaction with long video streams.

The technical principle of MOSS-VL-Realtime

Dual-channel cross-attention architectureVisual features enter through the side channel, decoupled from text generation, allowing perception and generation to run in parallel without blocking each other.
Decoupling cross attentionSeparate the cross-attention calculation between video feature extraction and text generation to reduce end-to-end latency for high-frequency frames.
Absolute timestamp encoding: Embed absolute time information into each frame, enabling the model to perceive the physical time flow and break free from the dependence on a fixed frame rate.
XRoPE Cross-Dimensional Rotation Position Encoding: Unify the spatial dimension (h, w) and temporal dimension (t) of the mapped video features to construct a unified spatiotemporal representation.
Streaming SFT training paradigmThe multimodal corpus was reconstructed, and dense captions were converted into question-and-answer data with real-time revisions. The model was then trained to predict when to speak and when to remain silent.

How to use MOSS-VL-Realtime

Obtain model resourcesVisit the OpenMOSS official website https://openmoss.ai/MOSS-VL/ to browse demo cases and download open-source weights and code.
Preparing the hardware environmentIt is recommended to equip a single H200 graphics card, which supports real-time inference and low-latency response for 256 frame video sequences.
Deploy model weightsLoad the 11B parameter model into the local environment and complete the dependency installation and weight initialization configuration.
Access video streamConnect to a real-time camera or import a video stream input to ensure that frame data can be continuously transmitted to the model.
Start real-time interactionYou can enter text questions at any time during video playback, and the model will generate or correct answers in real time based on the current screen.

The core advantages of MOSS-VL-Realtime

Streaming open source SOTAIt achieves the best performance among open-source technologies in multiple streaming video understanding benchmarks, and its real-time interactive capabilities are industry-leading.
Ultra-low latency and high throughputThe first token generation time on a single H200 card is increased by about 5 times, and the decoding throughput is increased by 2.7 times.
Offline capabilities remain undiminishedV*Bench 89.0 and BLINK 78.0 still align with top open-source baselines for understanding long videos and document diagrams.
Human-like interaction rhythmIt can autonomously determine when to speak and naturally supports natural dialogue behaviors such as interruption, immediate correction, and voluntary silence.
Unified Flow Cytometry CharacterizationVideo frames, user questions, and model answers are interwoven into a single token sequence, enabling seamless real-time interaction.

MOSS-VL-Realtime project address

Project official websitehttps://openmoss.ai/MOSS-VL/
HuggingFace model libraryhttps://huggingface.co/OpenMOSS-Team/MOSS-VL-Realtime

Comparison of similar products from MOSS-VL-Realtime

Dimension	MOSS-VL-Realtime	Qwen2.5-VL-7B
Architecture	Cross-attention dual-channel, decoupled parallel processing of perception and generation	Decoder-only, visual token embedding autoregressive sequence
Interaction mode	Streaming in real time, allowing you to watch, talk, and correct instantly.	Offline batch processing; please watch the video before answering.
First token delayed	Approximately 5 times faster, with millisecond-level response time.	Baseline level, blocked by visual processing
Decoding Throughput	2.7x improvement, smooth high frame rate generation	benchmark level
proactive behavior	Supports active silencing, instant correction, and arbitrary interruption.	No real-time behavior control capability
Time perception	Absolute timestamp encoding, perceiving the physical time flow	Relative position encoding, which relies on a fixed frame rate
Model size	11B Open Source Weight	7B Open Source Weight

Application scenarios of MOSS-VL-Realtime

Real-time monitoring and alarmsAutomatically silent in street surveillance scenes when nothing is happening, and triggers an alarm the moment a target falls or an abnormal intrusion occurs.
Live commentary of the eventThe system captures key actions such as passing, shooting, and scoring in real time during sports events like football and provides simultaneous professional commentary.
Presentation slide readingFollow the PPT slides in real time, providing an impromptu and coherent explanation of each slide's content from beginning to end.
Experimental process record: Observe plant growth through time-lapse photography to accurately capture key changes and report the specific number of days that occurred.
Real-time teaching and tutoringWatch the handwritten solution process; each line of formula is transcribed and the derivation logic is explained simultaneously.