AVD2 - A framework for understanding and generating videos of autonomous driving accidents, jointly developed by Tsinghua University, Fudan University, and other institutions.
AVD2 (Accident Video Diffusion for Accident Video Description) is a technology jointly developed by Tsinghua University, Hong Kong University of Science and Technology, Jilin University, Nanjing University of Science and Technology, Beijing Institute of Technology, Fudan University, and other institutions...
What is AVD2?
AVD2 (Accident Video Diffusion for Accident Video Description) is an innovative framework developed by Tsinghua University in collaboration with Hong Kong University of Science and Technology, Jilin University, Nanjing University of Science and Technology, Beijing Institute of Technology, Fudan University, and other institutions. It is used for understanding accident videos in autonomous driving. Based on generating accident videos aligned with detailed natural language descriptions and reasoning, it significantly improves the ability to understand complex accident scenarios. AVD2 combines video generation and accident analysis systems to generate high-quality video content including accident descriptions, causal analysis, and preventative measures. Based on AVD2, researchers created the EMM-AU (Enhanced Multi-Modal Accident Video Understanding) dataset, providing robust data support for accident analysis and prevention. Experimental results show that AVD2 performs excellently in both automatic and manual evaluation metrics, setting a new benchmark for the safety and reliability of autonomous driving.
AVD2's main functions
- Accident video generationBased on advanced video generation technology, it generates high-quality accident videos that are aligned with accident descriptions, cause analyses, and preventative measures.
- Accident Cause AnalysisProvides detailed causes of the accident, helping to understand its complexity.
- Recommendations for preventive measuresBased on accident analysis, we propose effective preventive measures to reduce the occurrence of similar accidents.
- Dataset AugmentationBased on generating new accident videos, expanding and enriching accident video datasets (such as EMM-AU), we can provide stronger data support for safety research on autonomous driving.
- Video understanding and reasoningBy combining natural language processing and computer vision technologies, descriptions and inferences related to accident videos can be generated, thereby improving the interpretability of accident scenes.
AVD2 technical principles
- Video generation technologyUtilizing advanced text-to-video generation models such as Open-Sora 1.2, we fine-tuned the pre-trained model to generate high-quality videos aligned with the incident descriptions. We then combined this with super-resolution techniques (such as Real-ESRGAN) to enhance video quality, ensuring the generated videos possess high definition and detail.
- Video understanding and description generationBased on the ADAPT (Action-aware Driving Caption Transformer) framework, combined with the Swin Transformer and BERT architecture, this system achieves deep fusion of visual and textual features from accident videos. Self-critical sequence training (SCST) optimizes the description generation process, and based on reinforcement learning mechanisms, it makes the generated descriptions more in line with human evaluation quality standards.
- Accident Analysis and ReasoningBased on natural language processing technology, the visual content of accident videos is transformed into detailed natural language descriptions, including the causes of the accident and preventive measures. By combining the accident videos and text descriptions, inference results aligned with the accident scene are generated, helping autonomous driving systems understand and respond to complex accident scenarios.
- Dataset Augmentation and EvaluationThe generated accident videos are used to expand the dataset, providing richer training data for autonomous driving accident analysis. The quality of the generated videos and descriptions is validated based on automated evaluation metrics (such as BLEU, METEOR, and CIDEr) and human evaluation.
AVD2 project address
- Project official website:https://an-answer-tree.github.io/
- GitHub repository:https://github.com/An-Answer-tree/AVD2
- arXiv technical paper:https://arxiv.org/pdf/2502.14801
Application scenarios of AVD2
- Autonomous driving R&D engineerUsed for developing and optimizing autonomous driving systems, analyzing accident scenarios, and improving algorithms and models.
- Traffic management departmentIt helps in developing traffic rules and safety policies, optimizing road design, and preventing accidents.
- car manufacturersIt is used in the design and testing of vehicle safety systems to improve vehicle safety performance.
- Researchers and scholarsUsed in research on autonomous driving and traffic safety to explore new technologies and methods.
- autonomous driving testersTo test the accident handling capabilities of the autonomous driving system and verify the system's reliability and safety.