FullStack Bench - A new code evaluation benchmark open-sourced by ByteDance and the M-A-P community.

What is FullStack Bench?

FullStack Bench is a new code evaluation benchmark jointly launched by ByteDance's Doubao Large Model Team and the M-A-P community, focusing on assessing full-stack programming and multi-language programming capabilities. FullStack Bench covers more than 11 real-world programming scenarios, includes 3374 questions, and involves 16 programming languages, enabling a more effective measurement of large models' real-world code development capabilities. Based on simulating real-world programming applications, FullStack Bench provides a comprehensive, multi-domain evaluation platform, contributing to the advancement of code intelligence technology.

Main functions of FullStack Bench

• Comprehensive assessmentFullStack Bench can evaluate the capabilities of large language models (LLMs) in a variety of real-world programming scenarios, including basic programming, data science, machine learning, and other fields.
• Multilingual supportIt covers 16 widely used programming languages, making the evaluation results more universal and practical.
• Real-world application scenario simulationBased on questions extracted from communities such as Stack Overflow, we simulate real-world programming problems to ensure the relevance and practical application value of the evaluation.
• Code quality controlEach problem includes a problem description, a reference solution, and unit test cases to ensure the accuracy and reliability of the evaluation.

The technical principles of FullStack Bench

• Dataset ConstructionWe analyzed the distribution of questions in technical communities such as Stack Overflow, extracted common real-world programming application areas, and constructed a dataset containing multiple questions.
• Manual annotation and verificationBased on manual annotation and verification processes, we ensure the quality of each issue, including the accuracy of the issue description and the correctness of the reference solution.
• Unit test casesDesign unit test cases for each problem to automatically verify whether the code generated by the model meets expectations.
• Multilingual programming ability assessmentBased on real-world design instructions and corresponding unit test cases, evaluate the model's performance in different programming languages.
• Sandbox execution environmentSandboxFusion, a sandbox execution tool, supports multiple programming languages and packages, providing a safe and isolated environment for code execution.

FullStack Bench project address

• GitHub repository:https://github.com/bytedance/FullStackBench
• HuggingFace model library:https://huggingface.co/datasets/ByteDance/FullStackBench
• arXiv technical paper:https://arxiv.org/pdf/2412.00535

Application scenarios of FullStack Bench

• Code intelligent evaluationUse FullStack Bench to test and evaluate the performance of large language models on code generation, understanding, and debugging tasks.
• Education and trainingUsing real-world programming problems from datasets as teaching cases helps students understand programming concepts and practice programming skills.
• Research and developmentIt provides researchers with a standardized testing platform to experiment with and improve AI technologies related to code.
• Software Development TestingUse FullStack Bench to automate code quality testing during the software development cycle and identify potential defects in advance.
• Multilingual programming ability assessmentAssessing a developer's proficiency in different programming languages as part of technical recruitment and career development.