AB
AB
AiBoss

All Categories

project

TinyVLA - East China Normal University and Shanghai University launch VLA model for robot control

TinyVLA is a vision-language-action (VLA) model for robot manipulation, developed by a team from East China Normal University and Shanghai University. It addresses the shortcomings of existing VLA models, such as slow inference speed and the need for large amounts of pre-trained data...

What is TinyVLA?

TinyVLA is a vision-language-action (VLA) model for robot manipulation, developed by a team from East China Normal University and Shanghai University. Addressing the shortcomings of existing VLA models, such as slow inference speed and the need for large amounts of pre-trained data, TinyVLA proposes a solution. Based on a lightweight multimodal model and a diffusion policy decoder, TinyVLA significantly improves inference speed and reduces reliance on large-scale datasets. The model has been extensively tested on simulated and real robot platforms, demonstrating superior performance compared to the state-of-the-art OpenVLA model in terms of speed, data efficiency, and multi-task learning and generalization capabilities. The introduction of TinyVLA provides a fast and efficient new approach for robot control, with the potential for rapid deployment and application in resource-constrained environments.

TinyVLA's main functions

  • Quick reasoning: TinyVLA can process visual data and language commands in a very short time to generate corresponding robot actions, significantly improving the speed of decision-making and execution.
  • Data efficiency: The model reduces its reliance on large-scale robotic datasets, eliminates the need for expensive pre-training processes, and lowers training costs and resource consumption.
  • Multitasking: It can handle a variety of different tasks, such as grasping, placing, and stacking, demonstrating good multi-task learning ability.
  • Strong generalization ability: TinyVLA maintains a high execution success rate even in unfamiliar environments, with new objects, and with new instructions, demonstrating strong generalization capabilities.

TinyVLA's technical principles

  • Multimodal model initialization: TinyVLA is based on a pre-trained multimodal model (such as Pythia) as the foundation of the policy network. The model has been trained on a large amount of visual-language data and has good semantic understanding capabilities.
  • Strategy backbone optimization: TinyVLA reduces the number of model parameters while maintaining model performance by training a lightweight multimodal model (VLM).
  • Diffusion strategy decoder: During the fine-tuning phase, the integrated diffusion strategy decoder directly outputs the robot's actions, replacing the traditional token-based prediction method and improving the continuity and accuracy of action prediction.
  • LoRA fine-tuning technology: The low-rank adaptation (LoRA) technique is used to fine-tune the pre-trained VLM part, updating only a small portion of the parameters in the model, thereby preserving the model's original knowledge and improving training efficiency.
  • One-time multi-step prediction: TinyVLA can predict a series of future actions at once, rather than making predictions one step at a time, thus improving the coherence and overall efficiency of action prediction.

TinyVLA's project address

Application scenarios of TinyVLA

  • Home automation: In a smart home environment, TinyVLA enables robots to understand natural language commands and perform household tasks, such as tidying up, turning lights on or off.
  • Industrial Automation: On manufacturing and packaging production lines, TinyVLA can control robots to perform precise assembly, sorting, and quality inspection.
  • Service robots: In the food and healthcare industries, TinyVLA helps robots understand customer needs and provide services such as ordering, food delivery, or medical guidance.
  • Disaster Response: In disaster relief scenarios, TinyVLA guides robots to conduct search and rescue missions in complex environments.
  • Education and training: TinyVLA serves as the brain of educational robots, assisting children's education or professional skills training through interactive learning.
  • Laboratory Assistant: In the scientific laboratory, TinyVLA guides robots to perform repetitive tasks such as sample processing and chemical analysis.
Back