Math24o - SuperCLUE: An open-source reasoning assessment benchmark for high school Olympiad mathematics competitions.
Math24o is an open-source Chinese large-scale model evaluation benchmark, similar to SuperCLUE, designed for high school Olympiad-level mathematical reasoning assessment. It is primarily used to evaluate the mathematical reasoning capabilities of large language models. It utilizes the 2024 national standard...
What is Math24o?
Math24o is an open-source Chinese large-scale model evaluation benchmark, part of SuperCLUE, designed for high school Olympiad-level mathematical reasoning. It primarily assesses the mathematical reasoning capabilities of large language models. Using questions from the preliminary rounds of the 2024 National High School Mathematics Competition, it includes 21 challenging problems with unique, integer or decimal answers. The evaluation automatically compares the model's answer with the reference answer to objectively assess the model's accuracy. This benchmark effectively measures the performance of language models in solving complex mathematical problems, providing a powerful tool for related research and development.
Main functions of Math24o
- Assessment of Advanced Mathematical ProblemsMath24o uses preliminary questions from the 2024 National High School Mathematics Competition, including 21 challenging problem-solving questions covering multiple mathematical fields such as functions, sequences, and geometry. It can comprehensively evaluate the model's reasoning ability in high school mathematics competitions.
- Uniqueness of answer and objective evaluationAll assessment questions have unique final answers, which must be either integers or decimals, ensuring the fairness and reliability of the assessment. The program automatically determines whether the model's answers match the reference answers, objectively evaluating the model's accuracy.
- Automated evaluation processMath24o provides an automated evaluation tool that allows users to save the model's answers to a specified file and automatically obtain the model's average score and detailed evaluation results for each question by running a Python script.
- Provide a reference for model developmentThe evaluation benchmark provides a reference for the development of future models, helps developers understand the performance of models in complex mathematical tasks, and promotes further improvement of models in mathematical reasoning ability.
Technical Implementation of Math24o
- Technical ImplementationMath24o is technically based on the Python programming language, using specific scripts to automate the evaluation process. This makes the entire evaluation process more efficient and standardized, reducing the possibility of human intervention.
Math24o's project address
- Github repository:https://github.com/CLUEbenchmark/Math24o
Math24o Model Performance Evaluation and Comparison
The test results show that o3-mini(high) performed the best, scoring the highest at 85.71 points. Other models, such as QwQ-32B and DeepSeek-R1, scored relatively lower, at 66.67 and 57.14 points respectively, indicating that there is still room for improvement in the performance of large models in the field of high school math olympiads.
| Ranking | Model | mechanism | Total Score | How to use | release date |
|---|---|---|---|---|---|
| 1 | o3-mini(high) | OpenAI | 85.71 | API | 2025.03.12 |
| 2 | Gemini-2.0-Flash-Thinking-Exp-01-21 | 71.43 | API | 2025.03.12 | |
| 3 | QwQ-Max-Preview | Alibaba Cloud | 66.67 | Official website | 2025.03.12 |
| 3 | QwQ-32B | Alibaba Cloud | 66.67 | Model | 2025.03.12 |
| 3 | o1 | OpenAI | 66.67 | API | 2025.03.12 |
| 4 | DeepSeek-R1 | In-depth exploration | 57.14 | API | 2025.03.12 |
| 4 | Claude 3.7 Sonnet | Anthropic | 57.14 | POE | 2025.03.12 |
Application scenarios of Math24o
- EducationMath24o provides tools for the education sector to assess and improve students' mathematical abilities. By using high school math Olympiad-level problems, it helps teachers and educational researchers understand students' reasoning abilities and problem-solving skills in complex mathematical problems.
- academic researchIn academic research, Math24o can serve as a benchmark for evaluating and comparing the mathematical reasoning capabilities of different large language models (LLMs). Researchers can use this benchmark to measure the model's performance in solving complex mathematical problems, thereby driving improvements and optimizations in model performance.
- Model Development and OptimizationMath24o provides an important reference for developing and optimizing large-scale language models. By testing the model's performance on high school math Olympiad problems, developers can better understand the model's strengths and weaknesses, adjust the model's architecture and training strategies, and improve the model's performance in mathematical reasoning tasks.
- Smart Campus ConstructionMath24o can be used as part of the construction of a smart campus to help schools assess and improve students' math skills.
- Educational resource integrationMath24o's questions and assessment tools can be integrated into educational resource platforms, providing students and teachers with abundant learning and teaching resources.