OlympicArena - A multidisciplinary cognitive reasoning benchmark framework launched by Shanghai Jiao Tong University in collaboration with AI Lab and others.
OlympicArena is a multidisciplinary cognitive reasoning benchmark framework jointly developed by Shanghai Jiao Tong University, Shanghai AI Lab, Soochow University, and the Generative Artificial Intelligence Laboratory (GAIR Lab) of Shanghai Jiao Tong University. OlympicArena includes...
What is OlympicArena?
OlympicArena is a multidisciplinary cognitive reasoning benchmark framework jointly launched by Shanghai Jiao Tong University, Shanghai AI Lab, Soochow University, and the Generative Artificial Intelligence Laboratory (GAIR Lab) of Shanghai Jiao Tong University. OlympicArena contains 11,163 bilingual problems from international Olympiad competitions, covering seven major fields: mathematics, physics, chemistry, biology, geography, astronomy, and computer science. OlympicArena comprehensively evaluates the advanced cognitive reasoning abilities of AI models, particularly logical and visual reasoning. Based on fine-grained evaluation at both the answer and process levels, OlympicArena reveals the limitations of AI models in solving complex problems, driving AI technology towards superintelligence.
Main functions of OlympicArena
- Full coverageIt covers seven core disciplines, including mathematics, physics, chemistry, biology, geography, astronomy and computer science, with a total of 34 sub-fields, comprehensively evaluating the cognitive reasoning ability of AI models in multiple disciplines.
- Bilingual supportThe benchmark tests are available in both Chinese and English, enhancing their international applicability.
- Answer-level assessmentTo accurately evaluate the answers provided by AI models.
- Process-level assessmentEvaluate each step in the problem-solving process to ensure that the AI model's reasoning is logical and correct.
- Multimodal supportIt supports problems involving interwoven text and images, and evaluates the ability of AI models to process multimodal information.
OlympicArena's technical principles
- Data collection and labelingQuestions were collected from 62 international Olympiad competitions to ensure high quality and diversity. A professional team extracted and labeled the questions, including categorizing them, labeling answer types, and annotating solution steps. A multi-step verification mechanism ensured the accuracy and consistency of the labeled data.
- Evaluation methodsFor problems with fixed answers, the correctness of the model output is verified based on rule matching; for problems requiring code generation, the correctness of the code is verified using test cases. The solution steps generated by the model are compared with the standard solution steps to evaluate the correctness of each step. For problems that are difficult to evaluate using rule matching, a high-performance model (such as GPT-4V) is used as an evaluator to determine the correctness of the model output.
- Multimodal processingFor problems involving images, key information is extracted from the images using image recognition technology and combined with text information to evaluate the multimodal processing capabilities of the AI model. Descriptive text is generated for the images to help the AI model better understand the image content.
- Data leak detectionThis involves using N-gram prediction technology to check if the model has seen the benchmark questions before, ensuring the fairness of the benchmark tests. Instance-level testing is performed for each question to verify whether the model correctly predicts the key information within the question.
OlympicArena's project address
- Project official website:https://gair-nlp.github.io/OlympicArena/
- GitHub repository:https://github.com/GAIR-NLP/OlympicArena
- HuggingFace model library:https://huggingface.co/datasets/GAIR/OlympicArena
- arXiv technical paper:https://arxiv.org/pdf/2406.12753
Application scenarios of OlympicArena
- AI Model Performance Evaluation: To test the cognitive reasoning ability of AI models in multidisciplinary fields.
- Model training and optimizationIt helps identify model weaknesses and guides improvements to training strategies.
- Education and learning supportProvides Olympic-level learning resources to support teaching.
- Scientific research and discoveryTo promote the application of AI in scientific research and contribute to scientific discovery.
- Technological Competition and ChallengesAs an AI technology competition platform, it aims to inspire innovation and promote technological development.