AB
AB
AiBoss

All Categories

project

OpenR - University College London, in collaboration with several other universities, has launched a framework to enhance the inference capabilities of large models.

OpenR is a full-chain training framework jointly open-sourced by University College London (UCL), Shanghai Jiao Tong University, University of Liverpool, Hong Kong University of Science and Technology (Guangzhou), and Westlake University. It aims to improve the performance of large language models (LLMs)...

What is OpenR?

OpenR is an open-source framework jointly developed by University College London (UCL), Shanghai Jiao Tong University, University of Liverpool, Hong Kong University of Science and Technology (Guangzhou), and Westlake University. It combines search, reinforcement learning, and procedural supervision to enhance the reasoning capabilities of large language models (LLMs). Inspired by OpenAI's o1 model, which significantly improves reasoning ability by integrating reinforcement learning during the reasoning process, OpenR is the first framework to provide an open-source implementation of ensemble techniques, supporting LLMs to achieve advanced reasoning based on efficient data acquisition, training, and inference paths. OpenR supports online reinforcement learning training and various search strategies, adhering to the test-time scaling rule, allowing models to provide refined outputs during testing through generation or search. OpenR provides an automated data pipeline that extracts reasoning steps from result labels, reducing manual annotation workload while ensuring the collection of valuable reasoning information.

Main functions of OpenR

  • Integrated Training and InferenceIt integrates data acquisition, reinforcement learning training (online and offline), and non-autoregressive decoding into a unified platform.
  • Process Reward Model (PRM)During training, the LLM policy is improved based on policy optimization techniques, and the search process of the LLM is guided during the decoding phase.
  • Enhance the learning environmentThe mathematical problem is modeled as a Markov decision process (MDP), and the model strategy is optimized based on reinforcement learning methods.
  • Multi-strategy search and decodingIt supports multiple search algorithms, such as Beam Search and Best-of-N, combined with guided search and rating based on PRM.
  • Data augmentation and automated annotation: Generate synthetic samples based on automated methods, reduce reliance on manual annotation, and improve data collection efficiency.

OpenR's technical principles

  • Process Reward Model (PRM)PRM is used to evaluate the correctness of solution steps. Based on supervised learning training, it uses correct or incorrect judgments as classification labels to predict subsequent labels for each step.
  • Strategy IterationDuring training, PRM improves the LLM policy based on policy optimization techniques such as policy iteration. In the decoding phase, PRM guides the LLM search process, allowing inference to develop towards more effective results.
  • Markov Decision Process (MDP)The mathematical problem is transformed into an MDP, which consists of states, actions, and rewards. The model generates reasoning steps as actions, and determines the next state based on the current state and actions.
  • reinforcement learningOnline reinforcement learning training is performed using algorithms such as Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO) to optimize the language output generated by the model.
  • Search AlgorithmDuring the decoding phase, the accuracy of each solution step is evaluated using PRM, and guided searches and multiple generated scores or votes are performed in conjunction with the language model.

OpenR project address

Application scenarios of OpenR

  • Solving mathematical problemsOpenR helps solve math problems by generating and evaluating reasoning steps to find the correct solution path.
  • Code generation and debuggingIn software development, OpenR helps generate code snippets or find and correct errors in the code during debugging.
  • Natural Language Processing (NLP) TasksOpenR is used in NLP tasks that require in-depth understanding of text and logical reasoning, such as machine reading comprehension, question answering systems, and text summarization.
  • Educational SupportIn the field of education, OpenR serves as an auxiliary tool to help students understand complex concepts and problem-solving steps, providing personalized learning paths.
  • Automated customer serviceIn the field of customer service, OpenR provides accurate answers and solutions based on reasoning about users' questions and needs.
Back