AReaL-boba - A reinforcement learning training framework jointly open-sourced by Ant Financial and Tsinghua University.
AReaL-boba is an open-source reinforcement learning training framework jointly developed by Ant Research Institute and Tsinghua University. AReaL-boba is an upgraded version of AReaL, lowering the barrier to entry for reinforcement learning training, allowing users to easily train inference models.
What is AReaL-boba?
AReaL-boba is an open-source reinforcement learning training framework jointly launched by Ant Research Institute and Tsinghua University. An upgraded version of AReaL, AReaL-boba lowers the barrier to entry for reinforcement learning training, allowing users to easily train inference models. The framework boasts fast training speed, supports various computing resources, and significantly improves training throughput through innovative optimizations. Its 7B model demonstrates outstanding performance in mathematical inference, setting a new AIME score record. AReaL-boba provides open-source training data, scripts, and models, replicating the inference performance of QwQ-32B with only 200 data points and a cost of $200 on a 32B model, thus promoting the democratization of reinforcement learning technology.
Main functions of AReaL-boba
- High-efficiency trainingBased on the optimized and adapted SGLang inference framework, it significantly improves training throughput and supports distributed training from small to large scale.
- Improved reasoning abilityIt performs exceptionally well on tasks such as mathematical reasoning, with the 7B model setting a new score record for models of the same size in the AIME benchmark test.
- Low-resource trainingBased on innovative data distillation technology, the inference performance of QwQ-32B can be reproduced with 200 data points, reducing training costs.
- Fully open sourceIt provides complete code, datasets, training scripts, and evaluation scripts to ensure reproducibility and facilitate developer use and improvement.
AReaL-boba's technical principles
- reinforcement learningReinforcement learning optimizes model behavior based on reward signals and learns optimal strategies through interactions with the environment. In language models, reinforcement learning is used to optimize the model's generative capabilities, resulting in better performance on specific tasks.
- SGLang Inference Framework IntegrationAReaL-boba is the first open-source training system to fully integrate the SGLang inference framework. SGLang provides efficient inference capabilities and optimizes computational efficiency during the training process.
- Engineering optimizationMultiple engineering optimizations were performed on the training process, including parallel computing and memory management, to improve training throughput. Significant training speed improvements were achieved across different model sizes.
- Data distillation technologyBased on an innovative data distillation method, key information is extracted from large amounts of data, and training data is streamlined.
AReaL-boba project address
- GitHub repository:https://github.com/inclusionAI/AReaL
- HuggingFace model library:https://huggingface.co/collections/inclusionAI/areal-boba
Application scenarios of AReaL-boba
- Mathematical Reasoning and EducationDevelop intelligent educational tools to help students solve complex mathematical problems.
- Natural Language Processing TasksImprove the performance of text generation, question answering systems, machine translation, etc.
- Intelligent agent developmentUsed for training intelligent agents in fields such as games and robot control.
- Low-resource model trainingSuitable for environments with limited data resources, enabling efficient model training.
- Academic research and community collaborationAs a research tool, it promotes academic exchange and technology sharing.