Multi-SWE-bench - ByteDance's open-source multilingual code fix benchmark

What is Multi-SWE-bench?

Multi-SWE-bench is the first multi-language code fixing benchmark open-sourced by ByteDance's Doubao Big Model team. Building upon SWE-bench, it covers seven major programming languages besides Python for the first time, including Java, TypeScript, JavaScript, Go, Rust, C, and C++, making it a true benchmark for "full-stack engineering." The dataset contains 1632 real-world fixing tasks, all from GitHub issues. These tasks have undergone rigorous screening and manual verification to ensure that each sample has a clear problem description, a correct fix patch, and a reproducible test environment. A task difficulty grading mechanism is introduced, dividing problems into three categories: easy, medium, and hard, covering development challenges ranging from single-line modifications to multi-file, multi-step, and multi-semantic dependency issues.

The main functions of Multi-SWE-bench

• Multilingual code fix assessmentAs the industry's first multi-language code repair benchmark dataset, Multi-SWE-bench covers seven major programming languages besides Python, including Java, TypeScript, JavaScript, Go, Rust, C, and C++. This allows the dataset to more comprehensively evaluate the automatic code repair capabilities of large models in different programming language environments.
• Task difficulty levelThe dataset introduces a task difficulty grading mechanism, dividing problems into three categories: Easy, Medium, and Hard. This grading method covers development challenges ranging from a single-line modification to multiple files, multiple steps, and multiple semantic dependencies, and can more systematically measure the performance of large models at different capability levels.
• Real data supportThe 1632 instances of Multi-SWE-bench all originate from real open-source repositories (GitHub issues) and have undergone unified testing standards and review by professional developers. Each sample has a clear problem description, correct fixes, and a reproducible test environment, ensuring the quality and usability of the dataset.

The technical principle of Multi-SWE-bench

• Data sources and quality controlThe 1632 instances in the dataset all originated from real open-source repositories (GitHub issues) and underwent rigorous testing and review by professional developers. During the build process, the team employed a strict five-stage data construction process:
  • Open source repository screeningBased on GitHub public repositories, we filter high-quality project repositories from multiple dimensions.
  • Pull Request CrawlingCollect pull requests (PRs) related to the issue and extract key information.
  • Docker environment setup: Build a corresponding Docker container for each PR to ensure that each task in the dataset is fully runnable.
  • PR filtering and validation: Identify effective fixes through a three-state testing process (original state, applying only test patches, and applying both test and fix patches).
  • Manual verificationWe introduced a manual double-labeling process to ensure the reliability and accuracy of the data.
• Enhanced learning supportTo support the application of reinforcement learning (RL) in code repair tasks, the team open-sourced Multi-SWE-RL. This community provides 4723 structured training samples, each equipped with a reproducible Docker environment, supporting one-click startup, automatic evaluation, and rapid integration with RL training frameworks. This dual-engine approach of "evaluation + training" provides strong support for the continuous optimization of large models.

The project address for Multi-SWE-bench

• Project official website:https://multi-swe-bench.github.io/#/
• Github repository:https://github.com/multi-swe-bench/multi-swe-bench
• HuggingFace dataset:https://huggingface.co/datasets/ByteDance-Seed/Multi-SWE-bench
• arXiv technical paper:https://arxiv.org/pdf/2504.02605

Application scenarios of Multi-SWE-bench

• Code Fix AutomationDevelopers can use models trained with Multi-SWE-bench to automatically identify and fix bugs in their code, reducing the time and workload of manual debugging.
• Model performance evaluation and improvementThe dataset provides a systematic benchmark for evaluating large models, helping developers and researchers assess the model's performance across different programming languages and task difficulties.
• Comparative Study of Programming LanguagesBy comparing the bug-fixing capabilities of different programming languages, researchers can gain a deeper understanding of the strengths and limitations of each language.
• Intelligent Learning and EducationFor developers and learners, Multi-SWE-bench is a platform for learning and improvement. By studying and using this dataset, developers can better understand common errors and fixes in different programming languages, thereby improving their programming and problem-solving skills.