DynamicCity - A 4D dynamic scene generation framework launched by Shanghai AI Lab

What is DynamicCity?

DynamicCity is a 4D generation framework for large-scale dynamic scene generation developed by the Shanghai AI Lab. DynamicCity focuses on generating dynamic LiDAR scenes with semantic information, capable of handling large-scale spatial (80×80×6.4 m³) and long sequences (up to 128 frames) of data. DynamicCity encodes 4D scenes into compact HexPlane representations based on the VAE model and reconstructs dynamic scenes using a Diffusion-based (DiT) generator. The DynamicCity framework supports various applications, such as trajectory guidance, instruction-driven generation, and dynamic scene inpainting. DynamicCity performs exceptionally well on the CarlaSC and Occ3D-Waymo datasets, significantly outperforming existing methods and demonstrating its powerful capabilities in generating high-quality dynamic scenes, providing strong support for autonomous driving and robotics technologies.

Main functions of DynamicCity

• High-quality 4D scene generationGenerates large-scale, high-quality dynamic LiDAR scenes, capturing the dynamic spatiotemporal evolution of real-world environments. Supports the generation of long sequences up to 128 frames, capable of simulating complex dynamic environments.
• Diverse downstream applications:
  • Trajectory-guided generationBased on a specific input trajectory, control the movement of objects in the scene.
  • Instruction-driven generation: Control the movement of the vehicle or scene based on commands (such as "turn left", "turn right", "go forward").
  • Dynamic scene restoration (Inpainting)Repair missing or damaged scenes to generate complete dynamic scenes.
  • Layout condition generation: Control the placement of vehicles and other objects based on a bird's-eye view layout.

The technical principles of DynamicCity

• VAE model:
  • Encoding stageEncode 4D LiDAR scenes into compact HexPlane representations. Features are extracted using a 3D convolutional neural network, and the 4D features are compressed into six 2D feature maps using a Projection Module. This improves the fitting quality of the HexPlane (up to 12.56% improvement in mIoU).
  • Decoding stageBased on the Expansion & Squeeze Strategy (ESS), HexPlane is decoded in parallel to reconstruct 3D feature volumes. Compared with point-by-point query methods, ESS improves fitting quality (up to 7.05% improvement in mIoU), speeds up training (up to 2.06 times improvement), and reduces memory usage (up to 70.84% reduction).
• DiT model:
  • HexPlane generationBased on the encoded HexPlane, the DiT model is used to generate new HexPlanes, enabling the generation of 4D LiDAR scenes. To make HexPlane suitable for DiT generation, the Padded Rollout Operation (PRO) is proposed, which reorganizes the six feature planes into a square 2D feature map, efficiently modeling the spatial and temporal relationships in the feature sequence.
  • Conditional generationDiT supports diverse 4D scene generation applications based on conditional injection (such as trajectories, instructions, layouts, etc.). Classifier-Free Guidance (CFG) allows the model to learn both conditional and unconditional generation during training, enabling finer control during the generation process.

DynamicCity project address

• Project official website:https://dynamic-city.github.io/
• GitHub repository:https://github.com/3DTopia/DynamicCity
• arXiv technical paper:https://arxiv.org/pdf/2410.18084

Application scenarios of DynamicCity

• Autonomous driving simulationGenerate complex dynamic scenarios for the development and testing of autonomous driving algorithms, thereby improving system safety.
• Virtual RealityCreate realistic virtual environments that support VR and AR applications, such as virtual driving and city planning demonstrations.
• Robot NavigationSimulates a three-dimensional dynamic environment to help robots perform path planning and obstacle detection, enhancing their adaptability.
• Traffic flow analysisModeling and analyzing traffic flow, predicting congestion, and optimizing traffic signals and road planning.
• Smart city planningGenerate city-level dynamic scenes to assist in evaluating urban layout and public facility planning.