AutoCodeBench - Tencent Hunyuan open-source dataset for evaluating the code capabilities of large models.

 What is AutoCodeBench?

AutoCodeBench is a benchmark dataset launched by Tencent Hunyuan specifically for evaluating the coding capabilities of large-scale models. It contains 3920 questions, evenly distributed across 20 programming languages. The dataset is highly challenging, practical, and diverse, capable of measuring the performance of large models in multi-language programming tasks. The benchmark dataset generates data through an automated workflow, ensuring high quality and coverage, and provides a simplified version (AutoCodeBench-Lite) and a version for evaluating the underlying model (AutoCodeBench-Complete).

AutoCodeBench's main functions

• Multilingual coding ability assessmentIt provides 3,920 questions covering 20 programming languages to comprehensively measure the multilingual code generation capabilities of large models.
• High-difficulty benchmark testsIt supports designing highly challenging problems and effectively identifies the shortcomings of large models in complex programming tasks.
• Performance differences magnifiedAutoCodeBench-Lite is built based on the selection problem to amplify the performance differences between different models and facilitate comparative analysis.
• Basic model evaluation: Use 3-shot hints to build AutoCodeBench-Complete, specifically to evaluate the code generation performance of the base model.
• Automated code data generation: Generate test input based on LLM and obtain output with the help of sandbox, and synthesize high-quality multi-language code generation data.
• Multilingual code execution verificationIt provides MultiLanguageSandbox service, which supports the compilation and execution of more than 30 programming languages and verifies the correctness of the generated code.

The technical principles of AutoCodeBench

• Automated data generationAutoCodeGen generates test inputs using a Large Language Model (LLM) and passes them to a sandbox environment. The sandbox environment executes the code and returns test outputs, constructing high-quality test functions. It constructs programming problems in reverse order, ensuring the generated problems are highly challenging and diverse. Multiple strategies are used to filter the generated data, further guaranteeing high quality, high difficulty, and practicality.
• Multilingual supportThe 3920 problems in AutoCodeBench are evenly distributed across 20 programming languages, ensuring that each language has enough problems for evaluation and avoiding uneven distribution. MultiLanguageSandbox supports compilation and execution of more than 30 programming languages, verifying the correctness and performance of generated code in different language environments, and ensuring that the model's performance across multiple languages can be accurately evaluated.
• High difficulty and practicalityBased on reverse-order problem construction and policy filtering, the generated problems are highly challenging and can effectively evaluate the model's performance in complex programming tasks. The generated problems are difficult, have practical application value, and reflect complex problems in real-world programming scenarios, helping the model to better function in practical applications.

AutoCodeBench project address

• Project official websitehttps://autocodebench.github.io/
• GitHub repositoryhttps://github.com/Tencent-Hunyuan/AutoCodeBenchmark
• HuggingFace model libraryhttps://huggingface.co/datasets/tencent/AutoCodeBenchmark
• arXiv technical paper: https://arxiv.org/pdf/2508.09101

Application scenarios of AutoCodeBench

• Model performance evaluationUsed to comprehensively measure the code generation capabilities of large models in multilingual programming tasks, helping to identify the strengths and weaknesses of the models.
• Dataset Construction and OptimizationGenerates high-quality, challenging code generation datasets, supports custom dataset construction, and improves model training performance.
• Multilingual proficiency verificationTo verify the performance of large models in different programming languages (including low-resource languages) and promote research on multilingual programming capabilities.
• Model training and validationAs a supplement to training data, it improves the model's performance on complex programming tasks and periodically verifies the training effect.
• Academic and industrial applicationsIt provides standardized benchmarks for academic research and supports the development and optimization of code generation tools in industrial scenarios.