SCoRe - A multi-round reinforcement learning method introduced by Google DeepMind

What is SCoRe?

SCoRe (Self-Correction via Reinforcement Learning) is an innovative multi-round reinforcement learning method developed by Google DeepMind, designed to improve the self-correction capabilities of large language models (LLMs). By training on model-generated data, the model can self-correct incorrect answers without external guidance. SCoRe training consists of two phases: the first phase initializes the model with appropriate regularization constraints to prevent mode collapse during training; the second phase uses a reward mechanism to encourage effective self-correction in the second attempt. Experimental results show that SCoRe improves self-correction capabilities by 15.6% and 9.1% on mathematical problems and programming tasks, respectively, outperforming traditional supervised learning methods. The success of SCoRe demonstrates the potential of reinforcement learning to improve the performance of large models, especially in applications requiring high accuracy.

SCoRe's main functions

• Self-correctionSCoRe enables large language models to identify and correct their own errors without external feedback.
• Self-generated data trainingIt is trained based on data generated by the model itself, without relying on external annotations or teacher models.
• Performance improvementImprove the self-correction ability of models in mathematical and programming tasks.
• Multi-round learningThe answer is gradually improved through multiple rounds of attempts to achieve the best response.
• Highly adaptableIt can adapt to the differences in data distribution between training and inference.

SCoRe's technical principles

• Multi-round reinforcement learningSCoRe is based on a multi-round RL framework, which allows the model to learn how to improve its behavior through multiple consecutive attempts.
• Regularization constraintsIn the first attempt at the model, use regularization techniques, such as KL divergence, to maintain the stability of the output.
• Reward ShapingBy designing a reward function, we can encourage the model to perform effective self-correction in subsequent attempts.
• Policy initializationIn the first stage of training, a specific strategy is used to initialize and improve the model's self-correction ability.
• Avoid distribution mismatchSCoRe avoids the mismatch between training data and the actual response distribution of the model by training on self-generated data.
• Incremental learningThe model attempts to improve upon previous outputs in each trial, achieving incremental learning.

SCoRe's project address

• arXiv technical paper:https://arxiv.org/pdf/2409.12917

Application scenarios of SCoRe

• Solving mathematical problemsIn mathematics, models need to perform complex calculations and logical reasoning. SCoRe helps models self-correct after providing incorrect answers, improving the accuracy of problem-solving.
• Programming and code generationIn programming tasks, code correctness is paramount. SCoRe can guide models to correct errors in the code, improving its reliability.
• Legal document analysisDocument analysis in the legal field requires extremely high accuracy. SCoRe helps the model self-correct when interpreting legal provisions and cases.
• Financial report generationErrors in financial reports can have serious consequences. SCoRe ensures the accuracy of the model when generating reports.
• Medical diagnostic aidIn the medical field, the self-correcting ability of models helps improve diagnostic accuracy and reduce the risk of misdiagnosis.