LiveCC - A real-time video narration model open-sourced by ByteDance and the National University of Singapore.
LiveCC is a real-time video narration model developed by the Show Lab team at the National University of Singapore in collaboration with ByteDance. It is based on large-scale training using Automatic Speech Recognition (ASR) captioning. LiveCC analyzes video content quickly, much like a professional narrator...
What is LiveCC?
LiveCC is a real-time video commentary model developed by the Show Lab team at the National University of Singapore in collaboration with ByteDance. It is trained on a large scale using Automatic Speech Recognition (ASR) captioning. Like a professional commentator, LiveCC quickly analyzes video content and simultaneously generates natural and fluent audio or text commentary. LiveCC offers the Live-CC-5M dataset for pre-training and the Live-WhisperX-526K dataset for high-quality supervised fine-tuning. LiveCC also features the LiveSports-3K benchmark to evaluate the model's real-time video commentary capabilities. Experiments show that LiveCC performs exceptionally well on real-time video commentary and video question-answering tasks, demonstrating low latency and high-quality generation capabilities.
LiveCC's main functions
- Real-time video commentaryIt generates continuous, human-like real-time commentary based on video content, suitable for various scenarios such as sports events, news broadcasts, and educational videos.
- Video Q&AAnswer questions related to the video content to help users better understand the events and details in the video.
- Low latency processingIt processes video streams with extremely low latency (less than 0.5 seconds per frame), supporting real-time applications.
- Multi-scenario adaptationSuitable for various video types, including sports, news, education, entertainment, etc.
LiveCC Technical Principles
- Streaming training methodThis approach involves densely interleaving words from Automatic Speech Recognition (ASR) with video frames according to their timestamps, allowing the model to learn time-aligned visual-language relationships. It simulates the real-time perception process of humans watching videos, enabling the model to generate comments closely related to the video content.
- Large-scale datasetsTwo datasets were constructed from ASR captions extracted from YouTube videos: Live-CC-5M (used for pre-training) and Live-WhisperX-526K (used for high-quality supervised fine-tuning). The datasets provided rich training material for the model.
- Model ArchitectureBased on the Qwen2-VL model architecture, this system combines a visual encoder and a language model to process video frames and text information. The model predicts text tokens using an autoregressive approach, treating video tokens as non-predictable input.
- Real-time inferenceDuring the inference phase, the LiveCC model processes the input video frame by frame, generating real-time commentary. To improve efficiency, the model caches previous cues, visual frames, and generated text, accelerating language decoding.
- Evaluation methodsThe real-time commenting capabilities of the model are evaluated based on the LiveSports-3K benchmark test, and the quality of comments generated by different models is compared using the LLM-as-a-judge framework.
LiveCC's project address
- Project official website:https://showlab.github.io/livecc/
- GitHub repository:https://github.com/showlab/livecc
- HuggingFace model library:https://huggingface.co/collections/chenjoya/livecc
- arXiv technical paper:https://arxiv.org/pdf/2504.16030
- Experience the demo online:https://huggingface.co/spaces/chenjoya/LiveCC
LiveCC application scenarios
- Sports eventsProvides real-time commentary and event analysis to enhance the viewer experience.
- News reportIt assists in real-time news interpretation, enhancing the depth and professionalism of reporting.
- Education: To generate explanations for instructional videos and assist in skills training.
- Entertainment MediaProvides real-time plot analysis for film and television content, increasing interactivity.
- Smart AssistantIt combines video content to provide real-time information and enhance the interactive experience.