StoryTeller - A fully automated long video description generation and consistency system jointly launched by ByteDance, Shanghai Jiao Tong University, and Peking University.
StoryTeller is a system jointly developed by ByteDance, Shanghai Jiao Tong University, and Peking University. It improves the quality and consistency of long video descriptions based on audio-visual character recognition technology. The system combines low-level visual concepts with high-level narrative...
What is StoryTeller?
StoryTeller, a system jointly developed by ByteDance, Shanghai Jiao Tong University, and Peking University, improves the quality and consistency of long video descriptions based on audio-visual character recognition technology. The system combines low-level visual concepts with high-level narrative information to generate detailed and coherent video descriptions. StoryTeller consists of three main modules: video segmentation, audio-visual character recognition, and description generation. It can effectively handle videos several minutes long and demonstrates higher accuracy than existing models on the MovieQA task, achieving 9.5% higher accuracy than the strongest baseline, Gemini-1.5-pro.
The main functions of StoryTeller
- Video SegmentationCut long videos into multiple short segments, maintaining the independence and integrity of each segment.
- Audio-visual character recognitionBy combining audio and visual information, identify the characters corresponding to the dialogue in the video.
- Description generationIt generates detailed descriptions for each video segment, integrating them into a coherent narrative for the entire long video.
- Dataset ConstructionCreate and use the MovieStory101 dataset to provide training and testing data for long video descriptions.
- Automatic evaluationBased on MovieQA, GPT-4 is used to automatically evaluate the accuracy and quality of video descriptions.
- Model training and fine-tuningTrain a large-scale multimodal language model to improve the accuracy of character recognition and video description.
- Global Decoding: Ensure consistent recognition results for the same character across different video clips.
The technical principles of StoryTeller
- Multimodal fusionIt integrates visual (video frames), audio (dialogue), and textual (subtitles and descriptions) information to fully understand video content.
- Audio separation and role ID assignmentEach dialogue is embedded using an audio embedding model, and a global ID is assigned based on a clustering algorithm. Similar audio embeddings are assigned the same ID to represent the same role.
- Audio-visual character recognition modelUsing a large language model (such as Tarsier-7B) combined with OpenAI's Whisper-large-v2 audio encoder, each audio ID is mapped to a specific role.
- Global decoding algorithmDuring reasoning, ensure that the global ID of the same character in different segments is mapped to a consistent character name to improve the accuracy of character identification.
- Video description generationUsing the recognition results as input, a detailed description of each segment is generated based on a large language model and integrated into a complete video description.
StoryTeller's project address
- GitHub repository:https://github.com/hyc2026/StoryTeller
- arXiv technical paper:https://arxiv.org/pdf/2411.07076
Application scenarios of StoryTeller
- Film and video content productionAutomatically generates descriptions for movie trailers or clips, helping directors and screenwriters quickly understand the video content. It also assists video editing by quickly locating key segments within the video based on the descriptions.
- Video content analysisIn the field of video analytics, key information such as characters, plot, and actions in video content is extracted for in-depth content analysis.
- Assisting visually impaired peopleProvides audio descriptions of video content for visually impaired individuals, enabling them to better understand the video content and storyline.
- Education and trainingIn education, it provides students with detailed descriptions of video teaching materials to enhance their learning experience. In vocational training, it generates detailed step-by-step descriptions of video tutorials to improve training efficiency.
- Video search and indexingImprove the accuracy of video search by quickly retrieving relevant segments from videos based on video descriptions.