OpenEMMA - An open-source end-to-end multimodal model for autonomous driving developed in collaboration with universities in Texas, Toronto, and other institutions.

What is OpenEMMA?

OpenEMMA is an open-source end-to-end multimodal model framework for autonomous driving, jointly developed by Texas A&M University, the University of Michigan, and the University of Toronto. It leverages pre-trained large multimodal language models (MLLMs) to process visual data and reason in complex driving scenarios. The framework utilizes a chain-like reasoning process, significantly improving model performance in trajectory planning and perception tasks. It also integrates a specially optimized YOLO model to enhance the accuracy of 3D bounding box prediction. OpenEMMA provides a platform to facilitate broader research and development, driving advancements in autonomous driving technology.

Main functions of OpenEMMA

• End-to-end trajectory planningIt learns driving actions directly from sensor input, achieving end-to-end optimization from perception to decision-making without the need for symbolic interfaces.
• Multimodal data processingThe framework processes forward-facing camera images and text history, along with vehicle status as input, framing the driving task as a visual question-answering (VQA) problem.
• Chain thinking reasoningIt employs a chain-thinking reasoning process to guide the model in generating detailed descriptions, behavioral insights, and meta-driving decisions for key objects.
• 3D object detectionWith its integrated and finely tuned YOLO model, OpenEMMA can accurately detect objects on 3D roads, improving the accuracy of object detection.
• Human-readable outputBased on the pre-existing world knowledge of MLLM, OpenEMMA can produce interpretable, human-readable output for perception tasks such as scene understanding.

OpenEMMA's technical principles

• Pre-trained MLLMsBased on pre-trained MLLMs, it processes complex visual data and infers driving scenarios.
• Chain-like reasoning processBased on chain-like reasoning, the model can generate velocity vectors and curvature vectors, which are used to calculate the future trajectory of the vehicle.
• Velocity and curvature vectorsGiven velocity and curvature vectors, the model first integrates the heading angle at each time step, then calculates the x and y components of the velocity, and finally calculates the final trajectory using the integrated velocity components.
• Enhanced object detectionTo overcome the limitations of MLLMs in spatial reasoning, a YOLO model specifically optimized for 3D bounding box prediction is integrated.
• End-to-end planning and reasoningOpenEMMA follows an instruction-based approach, prompting MLLMs to generate human-interpretable knowledge by breaking down the trajectory generation task into human-interpretable components that reflect the driving process.

OpenEMMA project address

• GitHub repository:https://github.com/taco-group/OpenEMMA
• arXiv technical paper:https://arxiv.org/pdf/2412.15208

Application scenarios of OpenEMMA

• City drivingIn complex urban traffic environments, it handles changing traffic signals, pedestrians, bicycles, and other vehicles, providing real-time decision-making and trajectory planning for autonomous vehicles.
• Highway drivingOn highways, it provides decision support for handling vehicles traveling at high speeds, including lane keeping, overtaking, and obstacle avoidance.
• Parking and low-speed drivingIn parking lots or low-speed environments, it helps autonomous vehicles perform precise parking maneuvers, avoid obstacles, and navigate in confined spaces.
• Night drivingOpenEMMA can operate in low-light conditions, providing decision support for nighttime driving, including object detection and trajectory planning.
• Complex weather conditionsIn complex weather conditions such as rain and fog, it assists autonomous vehicles in maintaining safe driving and reduces the impact of weather on driving.