UniAct - An embodied foundational model framework jointly developed by Tsinghua University, SenseTime, Peking University, and Shanghai AI Lab.

What is UniAct?

UniAct is a novel embodied foundational model framework that addresses the problem of behavioral heterogeneity among different robots. By learning common behaviors, it captures shared atomic behavioral features across different robots, eliminating behavioral heterogeneity caused by differences in physical form and control interface. UniAct's architecture includes a common behavior extractor, a common behavior space, and a heterogeneous decoder. The common behavior extractor, based on a visual language model, extracts common behaviors through observation and task objectives; the common behavior space is implemented as a vector-quantized codebook, with each vector representing an atomic behavior; and the heterogeneous decoder translates the common behaviors into robot-specific control signals.

Main functions of UniAct

• General Action CodingUniAct quantizes the atomic behaviors of different robots (such as "moving to the target location" or "avoiding obstacles") into a universal codebook through vectorization. Each token represents a common skill that can be shared across platforms.
• Lightweight architecture and high performanceThe UniAct-0.5B model requires only 50 million parameters, and in real and simulated task tests, it has outperformed the OpenVLA model with 1.4 billion parameters.
• Quickly adapt to new environments and robotsUniAct requires only 50 dedicated teaching data points to fine-tune a model in a new environment, enabling rapid adaptation to new robots and control interfaces. It can be easily extended to new robot platforms by adding new lightweight decoders.
• Cross-domain data utilizationUniAct, through its universal behavior space, can better utilize cross-domain data for training, achieving more efficient generalization across different robots and environments.
• Consistent behavioral patternsAcross different deployment scenarios and robot types, the same general action can exhibit consistent behavioral patterns. This provides a new and convenient way to control embodied agents; operators can simply select the corresponding general action from the codebook to command different types of robots to complete tasks.

UniAct's technical principles

• Universal Action SpaceUniAct constructs a discrete, universal action space through vector quantization. This space is designed as a vectorized codebook, where each vector embedding represents a common atomic behavior. These atomic behaviors are basic behavioral patterns shared by different robots in different contexts, such as "moving to a target location" or "avoiding obstacles." In this way, UniAct can uniformly represent the actions of different robots, eliminating the heterogeneity of the action space.
• Universal Action ExtractorUniAct uses a general action extractor based on a Visual Language Model (VLM) to identify and extract common actions. Given an observation and a task objective, the extractor outputs the probability of selecting a common action. In this way, UniAct can extract common actions directly related to task progress from complex visual and linguistic inputs, going beyond simply recognizing observed changes. This extraction method avoids interference from external factors (such as environmental changes or human intervention), ensuring the purity and consistency of the action space.
• Heterogeneous DecodersTo translate general actions into executable commands specific to a particular robot, UniAct introduces heterogeneous decoders. These decoders are designed for different robot platforms and can convert general actions into specific control signals based on the robot's specific characteristics (such as joint torques or camera views). In this way, UniAct can flexibly adapt to different types of robots, achieving efficient cross-platform control.
• Lightweight architecture and efficient trainingUniAct employs a lightweight model architecture, maintaining good performance even with low parameter settings. UniAct is trained using behavior cloning loss and selects an appropriate loss function based on the nature of the action labels (such as cross-entropy for discrete actions and mean squared error for continuous actions). This efficient training method allows UniAct to quickly adapt to new environments and new robots.

UniAct's project address

• Project official website:https://2toinf.github.io/UniAct/
• Github repository:https://github.com/2toinf/UniAct
• arXiv technical paper:https://arxiv.org/pdf/2501.10105

Application scenarios of UniAct

• Autonomous driving and intelligent transportationThe technical principles and architecture of UniAct are also applicable to the field of autonomous driving. By learning common driving behavior patterns, UniAct can provide more efficient motion planning and control for autonomous driving systems.
• medical robotsUniAct can be applied to the field of medical robotics, such as assistive rehabilitation robots or surgical robots. Through its universal motion space, medical robots can more flexibly adapt to different patient needs and surgical scenarios.
• Industrial AutomationIn the field of industrial automation, UniAct can be used to control various industrial robots, enabling efficient production process optimization. By quickly adapting to different robot platforms and task requirements, UniAct can significantly improve the flexibility and efficiency of industrial production.
• Smart Home and Service RobotsUniAct can be applied to the fields of smart homes and home service robots. Through a universal motion space, service robots can interact with humans more naturally and complete various household tasks.