VLN-R1 - An embodied intelligence framework jointly developed by the University of Hong Kong and the Shanghai AI Lab

What is VLN-R1?

VLN-R1 is a novel embodied intelligence framework jointly developed by the University of Hong Kong and the Shanghai Artificial Intelligence Laboratory. Based on a large visual language model (LVLM), it directly converts first-person video streams into continuous navigation actions. The framework builds upon the VLN-Ego dataset using the Habitat 3D simulator and employs a long short-term memory sampling strategy to balance historical and current observations. Training is divided into two phases: Supervised Fine-Tuning (SFT) aligns the model's action sequence text predictions with expert demonstrations, while Reinforcement Fine-Tuning (RFT) optimizes multi-step future actions based on a time-decaying reward (TDR) mechanism. VLN-R1 demonstrates strong performance on the VLN-CE benchmark, proving the effectiveness of LVLM in embodied navigation, enhancing task-specific reasoning capabilities, and exhibiting high data efficiency.

Main functions of VLN-R1

• Continuous Environment NavigationIt directly processes first-person video streams, allowing intelligent agents to move freely in a continuous 3D environment, not just limited to predefined nodes.
• Action generationGenerates four basic action commands (FORWARD, TURN-LEFT, TURN-RIGHT, STOP) to achieve precise navigation control.
• Data-efficient trainingBased on supervised fine-tuning (SFT) and reinforcement fine-tuning (RFT), efficient model training is achieved with limited data, thereby improving navigation performance.
• Cross-domain adaptationBased on reinforcement fine-tuning (RFT), the model can quickly adapt to new navigation tasks and environments, even with only a small amount of data.
• Task-Specific ReasoningBased on the Time Decay Reward (TDR) mechanism, the prediction of multi-step future actions is optimized, thereby enhancing long-term navigation performance.

VLN-R1 Technical Principles

• Dataset ConstructionThe VLN-Ego dataset was generated based on the Habitat 3D simulator and contains first-person video streams and corresponding future action predictions, providing rich data for model training.
• Long Short-Term Memory SamplingWhen processing video input, a long short-term memory sampling strategy is used to dynamically balance the importance of historical frames with the sensitivity of real-time input, ensuring that the model considers both short-term relevance and long-term contextual information during navigation.
• Supervisory fine-tuning (SFT)By minimizing the cross-entropy loss between the model's predicted text and the expert's demonstration text, the model's action sequence predictions are aligned with the actual actions, ensuring that the model can accurately understand language instructions and generate the corresponding actions.
• Enhanced Fine-tuning (RFT)A reinforcement learning method based on group relative policy optimization (GRPO) is proposed, which uses time decay reward (TDR) mechanism to evaluate and optimize the prediction of multi-step future actions, thereby enhancing the performance of the model in long-term navigation tasks.
• Large Visual Language Model (LVLM)Based on advanced LVLM (such as Qwen2-VL) processing of visual and language input, it realizes direct mapping from first-person video stream to navigation actions, improving the model's generalization ability and adaptability.

VLN-R1 Project Address

• Project official websitehttps://vlnr1.github.io/
• GitHub repository: https://github.com/Qi-Zhangyang/GPT4Scene-and-VLN-R1
• arXiv technical paper: https://arxiv.org/pdf/2506.17221

Application scenarios of VLN-R1

• Home service robotsHome service robots can navigate freely around the home based on the owner's natural language commands, completing tasks such as cleaning and retrieving items, thus improving the convenience of life.
• Industrial AutomationIn factory workshops, robots can navigate flexibly according to operator instructions to complete material handling and equipment maintenance, thereby improving production efficiency.
• Smart WarehousingThe goal is to enable warehouse robots to navigate precisely between shelves according to instructions, efficiently complete the storage and retrieval of goods, and optimize warehouse management.
• healthcareIt supports robots in hospitals or nursing homes to navigate according to the instructions of medical staff or patients to complete tasks such as delivering medicine and meals, thereby reducing the burden on medical staff.
• Intelligent TransportationIt helps autonomous vehicles navigate in complex urban environments according to traffic signals and instructions, enhancing driving safety and flexibility.