DrivingDojo - An interactive driving world model dataset jointly launched by the Chinese Academy of Sciences and Meituan.
DrivingDojo is a dataset developed by the Institute of Automation, Chinese Academy of Sciences, in collaboration with Meituan's autonomous vehicle team. It's used to train and research complex interactive world models for autonomous driving. The dataset contains 18,000 video clips, covering a wide range of topics...
What is DrivingDojo?
DrivingDojo is a dataset jointly developed by the Institute of Automation, Chinese Academy of Sciences, and Meituan's autonomous vehicle team. It's used to train and research complex interactive world models for autonomous driving. The dataset contains 18,000 video clips, covering complete driving operations, multi-agent interactions, and rich open-world driving knowledge, providing a solid foundation for developing next-generation autonomous driving models. DrivingDojo defines an Action Instruction Following (AIF) benchmark to evaluate the world model's ability to predict the future in executing action control.
DrivingDojo's main functions
- Interactive World Model TrainingDrivingDojo provides a platform for training interactive world models that understand and simulate complex driving dynamics.
- Action Instruction Follow (AIF)The dataset defines the AIF benchmark to evaluate the world model’s ability to generate future predictions in accordance with action instructions.
- Diverse driving operationsBased on the DrivingDojo-Action subset, the model learns diverse driving operations such as acceleration, deceleration, emergency braking, and lane changing.
- Multi-agent interactionUsing the DrivingDojo-Interplay subset, the model can understand and predict complex interactions between vehicles and other road users.
- Open World KnowledgeUsing a subset of DrivingDojo-Open, the model learns to handle rare events and long-tailed scenes in open worlds.
- Video generation and predictionThe model generates future videos based on initial frames and action commands to simulate and predict driving scenarios.
DrivingDojo's technical principles
- Data collection: Collect video data using Meituan's autonomous vehicle team's autonomous driving vehicle platform.
- Data planningVideo clips containing rich driving operations, multi-agent interactions, and open-world knowledge were selected from a large amount of collected data.
- Action codingEncode driving actions (such as acceleration, braking, and steering) into a format that the model can understand, making it easy to use in video generation.
- Video generation modelBased on technologies such as Stable Video Diffusion, we develop models that generate video from initial frames and motion instructions.
- Action Instruction Follow (AIF)The model's prediction accuracy is evaluated by comparing the consistency between the generated video and the input action commands.
- Evaluation indicatorsThe visual quality of the generated video is evaluated using metrics such as FID (Frechet Inception Distance) and FVD (Frechet Video Distance), and the ability to follow motion instructions is evaluated using AIF error.
DrivingDojo's project address
- Project official website:drivingdojo.github.io
- GitHub repository:https://github.com/Robertwyq/Drivingdojo
- HuggingFace model library:https://huggingface.co/datasets/Yuqi1997/DrivingDojo
- arXiv technical paper:https://arxiv.org/pdf/2410.10738
Application scenarios of DrivingDojo
- Autonomous driving algorithm developmentResearchers are developing and testing perception, prediction, and decision-making algorithms for autonomous vehicles.
- World Model TrainingThe dataset is used to train an interactive world model that simulates real-world driving environments. The model can predict future states and possible events.
- Multi-agent interaction simulationSimulate the interaction between vehicles and other road users (such as pedestrians, bicycles, and other vehicles) to optimize the interaction strategy of autonomous vehicles.
- Rare events and long-tail scenario handlingTraining models to recognize and respond to rare events, such as animals suddenly crossing the road or obstacles on the road, improves the safety of autonomous driving systems.
- Virtual testing and verificationTesting autonomous driving systems in a virtual environment reduces the cost and risk of real-vehicle testing and accelerates the development process.