SmolVLA - Hugging Face's open-source lightweight robot model

What is SmolVLA?

SmolVLA is a lightweight, open-source vision-language-action (VLA) model from Hugging Face, designed for cost-effective robotics. With 450 million parameters, the model is small enough to run on a CPU, be trained on a single consumer-grade GPU, and deploy on a MacBook. SmolVLA is trained entirely on an open-source dataset labeled "lerobot".

Main functions of SmolVLA

• Multimodal input processingSmolVLA can handle various inputs, including multiple images, language commands, and robot state information. It extracts image features through a visual encoder, tokenizes the language commands, and inputs them into a decoder. The sensed motion state is then projected onto a label through a linear layer, aligning with the label dimension of the language model.
• Action sequence generationThe model includes a motion expert module, a lightweight Transformer that generates action sequence blocks for future robots based on the output of a vision-language model (VLM). It is trained using flow matching technology, learning motion generation by guiding noisy samples back to the distribution of real data, thus achieving high-precision real-time control.
• Efficient reasoning and asynchronous executionSmolVLA introduces an asynchronous inference stack, separating action execution from perception and prediction, enabling faster and more sensitive control. This allows robots to respond more quickly in rapidly changing environments, improving response speed and task throughput.

SmolVLA Technical Principles

• Visual-Language Model (VLM)SmolVLA uses SmolVLM2 as its VLM backbone. The model is optimized to handle multiple image inputs. It includes a SigLIP visual encoder and a SmolLM2 language decoder. Image tags are extracted by the visual encoder, language commands are tokenized and directly input into the decoder, and sensed motion states are projected onto a tag through a linear layer, aligned with the tag dimension of the language model. The decoder layer processes the concatenated image, language, and state tags, and the resulting features are then passed to the action expert.
• Action ExpertMotion Expert is a lightweight Transformer (approximately 100 million parameters) that generates motion sequence blocks for future robots based on the output of the VLM. Motion Expert is trained using flow matching technology, learning motion generation by guiding noisy samples back to the distribution of real data, thus achieving high-precision real-time control.
• Visual token reductionTo improve efficiency, SmolVLA limits the number of visual tokens per frame to 64, which greatly reduces processing costs.
• Layer jump accelerated reasoningSmolVLA skips half of the layers in a VLM for computation, effectively halving the computational cost while maintaining good performance.
• Interleaved attention layersUnlike traditional VLA architectures, SmolVLA alternates between cross-attention (CA) and self-attention (SA) layers. This improves the efficiency of multimodal information integration and accelerates inference.
• Asynchronous reasoningSmolVLA introduces an asynchronous reasoning strategy, allowing the robot's "hands" and "eyes" to work independently. Under this strategy, the robot can perform the current action while simultaneously processing new observations and predicting the next set of actions, eliminating reasoning delays and increasing control frequency.

SmolVLA project address

• HuggingFace model library:https://huggingface.co/lerobot/smolvla_base
• arXiv technical paper:https://arxiv.org/pdf/2506.01844

Application scenarios of SmolVLA

• Object grasping and placementSmolVLA can control robotic arms to perform complex grasping and placement tasks. For example, on an industrial production line, robots need to accurately grasp parts and place them in designated locations based on visual input and verbal instructions.
• Household choresSmolVLA can be applied to home service robots to help complete various household chores. For example, the robot can identify and clean up clutter in a room or place items in designated locations based on natural language commands.
• Cargo handlingIn logistics warehouses, SmolVLA can control robots to complete cargo handling tasks. The robot can identify the position and shape of goods based on visual input, and combine this with verbal commands to generate optimal handling paths and action sequences, improving the efficiency and accuracy of cargo handling.
• Robotics EducationSmolVLA can serve as a tool for robotics education, helping students and researchers better understand and develop robotics technologies.