rStar-Math - Microsoft's innovative technology for small-model complex inference and self-evolving SLMs

What is rStar-Math?

rStar-Math is an innovative research project launched by Microsoft Research Asia. Based on Monte Carlo Tree Search (MCTS) driven deep thinking, it enables small language models (SLMs) to reach or even surpass the performance of large OpenAI models in mathematical reasoning. rStar-Math does not rely on data distillation from higher-level models; instead, it improves model performance through self-evolving deep thinking. rStar-Math introduces three innovative methods: code-enhanced stepwise verification reasoning trajectory synthesis, a Q-value-based process preference model (PPM) training method, and a four-round self-evolving training strategy. In the MATH benchmark, rStar-Math improved the accuracy of Qwen2.5-Math-7B from 58.8% to 90.0%, and in the AIME 2024 test, it solved an average of 53.3% of the problems, surpassing OpenAI's o1-preview model. rStar-Math demonstrates the model's inherent self-reflection ability, identifying and correcting erroneous steps during the reasoning process.

Main functions of rStar-Math

• Generate high-quality mathematical reasoning trajectoriesThe system generates step-by-step verification inference trajectories based on Monte Carlo Tree Search (MCTS), ensuring the correctness and high quality of each step.
• Self-evolution: Use four rounds of self-evolution to gradually improve the performance of the strategy model and process preference model (PPM) to handle more complex mathematical problems.
• Improve model accuracySignificantly improves model accuracy across multiple mathematical benchmarks, for example, increasing the accuracy of Qwen2.5-Math-7B from 58.8% to 90.0% on the MATH benchmark.
• Self-reflection abilityThe model can identify and correct erroneous steps during the reasoning process, demonstrating its ability to self-reflect.

The technical principles of rStar-Math

• Code-enhanced stepwise verification inference trajectory synthesis
  • Deep Thinking Driven by MCTSThe complex mathematical problem is broken down into multiple single-step generation tasks, and a search tree is built step by step based on MCTS to generate inference trajectories.
  • Code execution verificationThe policy model generates natural language (NL) inference steps and corresponding Python code.
  • Q value labelingBased on two methods, terminal-guided annotation and PPM-enhanced annotation, a Q-value is automatically assigned to each step to guide MCTS nodes in selecting and identifying high-quality steps.
• Process Preference Model (PPM) Training Method
  • Avoid using Q value directlyTraditional Q-values as reward labels suffer from noise and inaccuracy. rStar-Math improves the reliability of labels by constructing step-level positive and negative preference pairs and using pairwise ranking loss to train the PPM.
  • Preferences for buildingFor each step, the two steps with the highest Q-values are selected as positive examples, and the two steps with the lowest Q-values are selected as negative examples. PPM is trained using these preference pairs to predict the reward label for each step.
• Four-wheel self-evolution
  • Initial strong policy modelIn the first round, DeepSeek-Coder-V2-Instruct was used as the initial policy model, and MCTS rollout was performed to generate training data.
  • Reliable PPM TrainingIn the second round, the updated policy model is used for more reliable Q-value labeling to train the first reliable PPM.
  • PPM enhances MCTSThe third round uses a reliable PPM for MCTS, generating higher quality inference trajectories that cover more mathematical and competition-level problems.
  • Solving challenging problemsThe fourth round increases the number of MCTS rollouts and uses different random seeds to improve coverage of competitive problems.

rStar-Math project address

• arXiv technical paper:https://arxiv.org/pdf/2501.04519

Application scenarios of rStar-Math

• Educational guidanceIt provides students with personalized math learning tutoring, gradually solving complex math problems and improving their problem-solving and comprehension skills.
• Research supportIt assists mathematicians and scientists in exploring complex mathematical problems, generating preliminary problem-solving ideas and verification steps, and accelerating the research process.
• FintechIn financial risk assessment and quantitative trading, market risks are predicted and trading strategies are optimized based on accurate mathematical models and reasoning.
• Engineering DesignIn engineering design and system optimization, mathematical reasoning is used to optimize system parameters and improve system performance and reliability.
• Data AnalysisIn enterprise data analysis, valuable information is extracted from large amounts of data based on mathematical models and reasoning to support market forecasting and business decision-making.