Deep Video Discovery - Microsoft's intelligent agent for deep video exploration

What is Deep Video Discovery?

Deep Video Discovery (DVD) is a deep video exploration agent launched by Microsoft, designed specifically for understanding and analyzing long videos. Deep Video Discovery segments long videos into multiple shorter segments and autonomously plans and selects appropriate tools and parameters to collect information based on the advanced reasoning capabilities of a Large Language Model (LLM). Deep Video Discovery is equipped with a suite of search center tools, including global browsing, segment search, and frame inspection, enabling it to collect information at different levels and gradually build an understanding of the video content based on iterative reasoning. In multiple long video understanding benchmarks, Deep Video Discovery has achieved state-of-the-art performance, significantly improving the accuracy and efficiency of long video understanding.

Main functions of Deep Video Discovery

• Multi-granularity video understandingIt provides comprehensive video understanding capabilities by understanding and analyzing video content at three levels: global, segment, and frame.
• Autonomous Search and ReasoningIt autonomously plans and executes search strategies, dynamically selects appropriate tools and parameters based on user queries, and gradually collects information and builds an understanding of video content.
• Efficient Information RetrievalBased on tools such as global browsing, segment search, and frame inspection, it can quickly locate and extract video segments and details related to user queries.
• Long video comprehensionIt excels at handling information-intensive videos that can last for several hours and effectively solves the problems of temporal and spatial complexity in long video understanding.
• Flexible use of toolsDepending on the specific task requirements, various tools can be flexibly combined and used to achieve efficient video content analysis and question answering.

The technical principles of Deep Video Discovery

• Multi-granularity video database constructionThe long video is evenly divided into multiple shorter clips, each approximately 5 seconds long. Video information is extracted from three levels: global, clip, and frame. The global layer provides a summary of the main content and events of the video, the clip layer provides text descriptions (captions), and the frame layer retains the original pixel information. A structured database containing decoded frames, text descriptions, and corresponding embedding vectors is constructed to support fast retrieval and detailed analysis.
• Autonomous Search and Answer Generation:
  • Global BrowseProvides a global summary of the video, helping the agent quickly understand the main content and key events of the video.
  • Clip SearchBased on text embedding matching, quickly retrieve video clips related to the user's query.
  • Frame inspectionPerform fine-grained visual question answering (VQA) within a specific time frame to extract detailed information at the frame level.
  • Independent agency designThe agent uses an iterative observation-reasoning-action loop, leveraging the reasoning capabilities of LLM to dynamically select and use tools, gradually gathering information and building an understanding of the video content.
  • Iterative ReasoningThe agent selects appropriate tools and parameters based on the current observation status and inference results, gradually refines the query, and finally generates an accurate answer.
• LLM-driven inferenceLLM, as the core, is responsible for reasoning and planning. Based on the current dialogue history and observations, LLM selects appropriate tools and parameters, dynamically adjusting its inference strategy. Depending on task requirements, it flexibly combines and uses different tools to construct a multi-step tool usage chain to solve complex query tasks.

The project address for Deep Video Discovery

• arXiv technical paperhttps://arxiv.org/pdf/2505.18079

Application scenarios of Deep Video Discovery

• EducationOnline education platforms analyze long video courses, allowing students to quickly locate specific knowledge points or chapters within the course.
• Sports event analysis: Analyze sports events and match videos to quickly extract key events.
• Video surveillanceIn security monitoring systems, surveillance videos are analyzed in real time to quickly identify abnormal behaviors or events.
• Film and television productionThe film and television post-production team analyzes the shooting footage and quickly finds the required shots.
• Corporate meeting minutes: Analyze meeting videos to quickly extract key points and crucial decisions.