EMMA - Waymo's end-to-end multimodal autonomous driving model

What is EMMA?

EMMA, Waymo's end-to-end multimodal autonomous driving model based on the Gemini model, directly maps raw camera sensor data to driving-specific outputs, such as planned trajectories, perceived objects, and road map elements. EMMA represents non-sensor inputs and outputs as natural language text, leveraging the world knowledge of a pre-trained large language model to jointly process multiple driving tasks in a unified language space. EMMA demonstrates state-of-the-art performance on nuScenes motion planning and the Waymo Open Dataset, but it has limitations, such as a limited number of image frames processed, a lack of integration of precise 3D sensing methods, and high computational cost. The model can drive the development of autonomous driving model architectures and improve the generalization and inference capabilities of autonomous driving systems in complex scenarios.

EMMA's main functions

• End-to-end motion planning:
  • The future trajectory of an autonomous vehicle is generated directly from raw camera sensor data.
  • The trajectory is translated into vehicle-specific control actions, such as acceleration and steering.
• 3D object detectionIt uses cameras as the primary sensor to detect and identify surrounding objects, such as vehicles, pedestrians, and cyclists.
• Road map element recognitionIdentify and construct road maps, including key road elements such as lane lines and traffic signs.
• Scene understanding: Understand the context of the entire scenario, including temporary road blockages and other situations that affect driving.
• MultitaskingIt enables the joint processing of multiple driving tasks within a unified language space, generating output using task-specific prompts.
• Chain thinking reasoning:based onChain reasoning enhances the decision-making ability and interpretability of models, enabling them to explain their decision-making rationale when predicting future trajectories.

EMMA's technical principles

• Multimodal Large Language Models (MLLMs)Pre-trained MLLMs, such as Gemini, are models trained on a wide range of internet-scale data and possess rich "world knowledge".
• Natural Language RepresentationAll non-sensor inputs and outputs (such as navigation commands, vehicle status, trajectory, and 3D position) are represented as natural language text.
• Visual Question Answering (VQA)The driving task is reimagined as a VQA problem, based on Gemini's pre-training capabilities, preserving a wide range of world knowledge.
• Autoregressive modelThe autoregressive Gemini model is used to process interleaved text and visual inputs and generate text output.
• End-to-end trainingBased on end-to-end training, driving actions are generated directly from sensor data, eliminating the need for symbolic interfaces between modules.

EMMA's project address

• arXiv technical paper:https://arxiv.org/pdf/2410.23262

Application scenarios of EMMA

• City and suburban drivingEMMA can handle complex urban traffic environments and suburban road conditions, providing real-time driving decisions and trajectory planning.
• Traffic congestion and complex intersectionsIn traffic congestion or multi-intersection scenarios, EMMA can perform effective route planning and decision-making to ensure safe and efficient navigation.
• Special weather and lighting conditionsEMMA can adapt to different weather and lighting conditions, such as rain, fog, or night driving, and maintain stable driving performance.
• Construction area and temporary roads closedBased on its scene understanding capabilities, EMMA can identify construction areas and temporary road closures, and make corresponding driving adjustments.
• Emergency ResponseIn the event of an emergency, such as a sudden obstacle or animal, EMMA can react quickly and take measures such as avoidance or slowing down.