HoloDrive - A 2D-3D multimodal street scene generation framework launched by SenseTime in collaboration with Shanghai AI Lab and other institutions.

What is HoloDrive?

HoloDrive is a holistic 2D-3D multimodal street scene generation framework for autonomous driving, proposed by SenseTime and the Shanghai Artificial Intelligence Laboratory, among other institutions. The framework jointly generates camera images and LiDAR point clouds, filling the gap in 2D-3D multimodal joint generation for autonomous driving. HoloDrive employs BEV-to-Camera and Camera-to-BEV conversion modules between heterogeneous generation models, introducing a depth prediction branch into the 2D generation model to eliminate projection ambiguity from image space to BEV space.

HoloDrive's main functions

• Jointly generate camera images and LiDAR point cloudsHoloDrive can simultaneously generate multi-view camera images and LiDAR point clouds, filling the gap in 2D-3D multimodal joint generation in autonomous driving.
• cross-modal structureHoloDrive achieves efficient alignment and information exchange between 2D and 3D spaces through BEV-to-Camera and Camera-to-BEV conversion modules, as well as a depth prediction branch in the 2D generative model, enabling end-to-end training of the entire model.
• Time structure and progressive trainingHoloDrive extends this method to predict the future by incorporating temporal structures and carefully designed progressive training, making it suitable for both single-frame generation and video generation tasks.
• High-performance generationExperimental results on the NuScenes dataset show that HoloDrive achieves state-of-the-art performance in generating single-frame and sequential data of multi-view camera images and LiDAR point clouds, significantly outperforming existing state-of-the-art methods (SOTA).

HoloDrive's technical principles

• BEV-to-Camera and Camera-to-BEV conversion modules: Use BEV-to-Camera and Camera-to-BEV conversion modules between heterogeneous generative models to align 3D and 2D spaces.
• Deep prediction branchA depth prediction branch is introduced into the 2D generative model, and natural supervision is derived from 3D LiDAR, eliminating projection ambiguity from image space to BEV space.
• Time structureBy incorporating a temporal structure, HoloDrive extends the method to predict the future, making it suitable for both single-frame generation and video generation tasks.
• Progressive trainingBased on a progressive training strategy, combined with additional multi-task learning in the video domain, a smooth transition in the training phase is achieved.

HoloDrive project address

• arXiv technical paper:https://arxiv.org/pdf/2412.01407

HoloDrive application scenarios

• Generate realistic street scenesHoloDrive can jointly generate multi-view camera images and LiDAR point clouds to create realistic street scenes, reducing the need for expensive manual modeling of the real world.
• 2D-3D joint generationHoloDrive achieves efficient alignment and information exchange between 2D and 3D spaces through BEV-to-Camera and Camera-to-BEV conversion modules, as well as a depth prediction branch in the 2D generative model.
• Time structure and progressive trainingHoloDrive extends this method to predict the future by incorporating temporal structures and carefully designed progressive training.