Open-o3 Video - A video inference model jointly developed by Peking University and ByteDance.

What is Open-o3 Video?

Open-o3 Video is an open-source video inference model jointly developed by Peking University and ByteDance. It achieves accurate video inference by integrating explicit spatiotemporal evidence (key timestamps and bounding boxes). Through a carefully curated STGR dataset and a two-stage SFT-RL training strategy, it achieves state-of-the-art performance on the V-STAR benchmark. Its agentless framework design efficiently handles complex spatiotemporal relationships, resulting in outstanding performance in video inference tasks. The training process includes two phases: cold start initialization and reinforcement learning. This approach allows the model to better adapt to different video inference scenarios.

Main functions of Open-o3 Video

• Spacetime reasoningIt can integrate explicit spatiotemporal evidence, including key timestamps and bounding boxes, to accurately perform video reasoning and effectively handle the temporal and spatial relationships in videos.
• Dataset planning and training strategiesWe meticulously designed the STGR dataset and adopted a two-stage SFT-RL training strategy, first performing cold start initialization and then optimizing model performance through reinforcement learning, resulting in outstanding performance in the V-STAR benchmark test.
• Non-proxy framework designIt adopts a non-proxy framework to efficiently handle complex spatiotemporal relationships, thereby improving the accuracy and efficiency of video reasoning.
• Open source and scalabilityIts open-source nature allows researchers and developers to easily use and improve the model, driving the development of video reasoning technology and providing good scalability.

The technical principles of Open-o3 Video

• Spatiotemporal evidence integrationBy explicitly introducing key timestamps and bounding boxes as spatiotemporal evidence, the video reasoning process is closely integrated with specific visual observations, making the model's reasoning more interpretable and reliable.
• Two-stage training strategyA two-stage training method combining cold start initialization and reinforcement learning is adopted. The cold start stage provides the model with basic spatiotemporal reasoning capabilities through supervised learning; the reinforcement learning stage further optimizes the model's reasoning performance through various reward mechanisms, improving the accuracy, temporal alignment, and spatial precision of the answers.
• Dataset planningWe have carefully curated two high-quality datasets, STGR-CoT-30k and STGR-RL-36k, to provide rich spatiotemporal annotations and inference traces for model training, thus solving the problem of the lack of unified spatiotemporal supervision in existing datasets.
• Non-proxy framework designThe model built on a non-proxy framework can efficiently handle complex spatiotemporal relationships, avoid the problems of information loss and low reasoning efficiency that may be caused by proxy models, and improve the overall efficiency and accuracy of video reasoning.

Open-o3 Video project address

• Project official website: https://marinero4972.github.io/projects/Open-o3-Video/
• Github repositoryhttps://github.com/marinero4972/Open-o3-Video
• HuggingFace model library: https://huggingface.co/marinero4972/Open-o3-Video/tree/main
• arXiv technical paper: https://arxiv.org/pdf/2510.20579

Application scenarios of Open-o3 Video

• Video content comprehensionIt can accurately understand and analyze key events and objects in videos, and provide detailed reasoning and explanations for video content through spatiotemporal evidence, helping users better understand the core information of videos.
• Video Q&A SystemIt can serve as a core component of a video question-and-answer system, quickly locating relevant spatiotemporal segments in a video based on the user's question, generating accurate and explanatory answers, and improving the user experience.
• Video editing and creationIt provides assistance for video editing and creation, helping creators quickly find key elements and exciting moments in videos, and perform editing, special effects additions and other creative activities more efficiently.
• Intelligent monitoring and analysisIn the field of intelligent surveillance, it can analyze surveillance videos in real time, quickly identify abnormal events and key objects, provide detailed spatiotemporal evidence, and help upgrade security surveillance to intelligent levels.
• Education and TrainingIn education and training, it can be used to analyze teaching videos, helping teachers and students better understand the teaching content, while providing students with more targeted learning suggestions and feedback.
• Entertainment and InteractionIn the entertainment field, such as short video platforms and live streaming, it can provide users with richer interactive experiences. For example, it can generate interesting questions and answers or challenges through video reasoning, thereby enhancing user engagement.