AB
AB
AiBoss

All Categories

project

RDT - Tsinghua University's open-source diffusion model for dual-arm robots

RDT (Robotics Diffusion Transformer) is the world's largest foundational model for diffusing maneuvers in dual-arm robots, developed by the TSAIL team at Tsinghua University's AI Research Institute. RDT boasts one billion parameters and can perform maneuvers without human intervention...

What is RDT?

RDT (Robotics Diffusion Transformer) is the world's largest foundational model for diffusing maneuvering tasks in dual-arm robotics, developed by the TSAIL team at Tsinghua University's Institute for AI. With one billion parameters, RDT can autonomously complete complex tasks such as bartending and dog walking without human intervention. Based on imitation learning of human movements, RDT demonstrates powerful generalization ability and operational accuracy, and can handle unseen objects and scenes. The Tsinghua team has open-sourced RDT's code, model, and training dataset to promote the development and application of robotics technology.

Main functions of RDT

  • Coordinated operation of both armsTo direct the robot's two arms to work together to complete complex physical tasks, such as bartending and walking a dog.
  • Autonomous task executionIt can autonomously complete entirely new tasks that have never been seen before, without direct human control.
  • High-precision operationRDT can perform precise operations, such as controlling a robot dog to move in a straight line, which requires extremely high operational precision.
  • Language instruction comprehension: To understand and follow human natural language instructions and perform corresponding actions.
  • Few-shot learningRDT has a strong learning ability and can learn new skills with only a few demonstrations.

RDT Technical Principles

  • Multimodal input codingRDT combines language, vision, and action modalities, and processes input based on different encoding methods.
    • Action coding: Multilayer perceptron (MLP) with Fourier characteristics.
    • Image encodingBased on aligned SigLIP.
    • Language encodingUse the T5-XXL language model.
  • Transformer backbone networkRDT uses Transformer as its backbone network and makes key modifications for robot operation.
    • QKNorm and RMSnormTo mitigate the problem of extreme values caused by sensor malfunction.
    • Nonlinear MLP Decoder: Enhances the ability to approximate nonlinear dynamics.
    • Alternating injectionBalance image and text modalities to prevent information overload.
  • Pre-training and fine-tuningRDT is pre-trained on a large-scale embodied dataset to achieve generalization, and then fine-tuned on a high-quality dual-arm fine-tuning dataset to enhance dual-arm maneuverability.
  • Unified Action Space: Construct a unified motion space to unify the format of data from different robots, allowing the model to learn shared physical laws from different data.
  • Generalization and operational accuracy testingDesign challenging tasks to evaluate the generalization ability and operational accuracy of the RDT and ensure its effectiveness in practical applications.

RDT's project address

Application scenarios of RDT

  • Food and beverage servicesIt can be used to automate tasks such as bartending, cooking, and serving, thereby improving the efficiency and innovation of the food service industry.
  • Home AssistantIn a home environment, it can perform household chores such as cleaning, tidying, and laundry, and can also take care of pets, such as walking the dog.
  • Medical assistance: Assist medical staff in performing routine nursing tasks, such as distributing medications and moving medical equipment.
  • Industrial AutomationIn manufacturing, it is used for precision assembly, quality inspection, and material handling.
  • Disaster reliefTo carry out search and rescue missions at disaster sites, especially in environments that are difficult for humans to reach or that are dangerous.
Back