project
WorkBuddy Bench - Tencent's open-source coding intelligence evaluation suite
WorkBuddy Bench is an open-source coding intelligence evaluation suite from Tencent, covering four real-world scenarios: code, web development, office work, and security. Tasks are reverse-engineered from real commits, pull requests, and business scenarios, and rewritten into conversational requests...
What is WorkBuddy Bench?
WorkBuddy Bench is an open-source coding intelligence evaluation suite from Tencent, covering four real-world scenarios: code, web development, office work, and security. Tasks are reverse-engineered from real commits, pull requests, and business scenarios, rewritten into conversational requests to combat data corruption. 260 tasks undergo multi-dimensional scoring in an isolated sandbox using hidden testing, rule checks, and LLM evaluation, and are cross-validated under a dual framework, achieving open-source, reproducible, and auditable end-to-end evaluation.
WorkBuddy Bench's main functions
- CodeThere are 80 warehouse-level software engineering tasks, covering 18 categories such as bug fixing, feature development, interface adjustment, and algorithm implementation. Requests are initiated by five roles: developers, algorithm engineers, product managers, QA, and operations and maintenance.
- Web (webpage)There are 70 front-end tasks, covering 7 categories including page implementation, interaction design, data visualization, visual design, code testing, and document conversion. The requirement is to deliver a working front-end artifact rather than dialogue text.
- OfficeThe task involves 50 multi-file business processes. The intelligent agent needs to process mixed-format files such as spreadsheets, documents, PDFs, and JSONs, and complete handover tasks such as data verification, report generation, and status updates.
- SecurityThe test consisted of 60 security tasks for red and blue teams, including vulnerability discovery and exploitation, malware analysis, security operations, and security assessment of AI agents. It employed deterministic scoring scripts and did not involve an LLM evaluator.
The technical principles of WorkBuddy Bench
- Reverse engineering constructionEach task is reverse engineered from real code commits, PRs, CVEs, or business scenarios, and rewritten into short, conversational role-playing requests so that the prompts cannot be retrieved by search engines.
- Intentional lack of informationThe request only states the intent and constraints, omitting details such as target files, precise interfaces, and boundary handling. The agent must restore the context from the workspace on its own to test the disambiguation and implementation capabilities of the requirements.
- Post-round assessment isolationDuring the problem-solving process, the agent can see the task instructions and workspace. The scoring assets (hidden tests, scoring rules, etc.) are only introduced into the sandbox after completion to avoid the leakage of scoring information.
- Dual-frame cross-validationAll tasks are run under two agent frameworks, CodeBuddy Code and Claude Code, to eliminate the systematic bias of a single framework.
- Anti-pollution mechanismIt relies on a closed, searchable path for task construction and regular updates to the dataset version management system, replacing traditional confidential anti-pollution solutions.
Follow us on WeChat and reply with "open source",join inAI open source project discussion group
How to use WorkBuddy Bench
- Get task setAccess GitHub to download the standardized task directory, which includes instruction.md, task.toml, the environment/ environment image, and tests/ scoring assets.
- Operational evaluationThe agent is connected to a Harbor-style task directory and executed in an isolated Docker sandbox. The agent reads natural language requests, explores the workspace, and generates artifacts.
- View resultsThe scorer runs automatically upon completion of the task. The code subset is scored using hidden unit tests, the web page subset is scored using a rule-based LLM/VLM + agent-based scorer, the office subset is scored using a rule-based check and evidence-driven LLM scorer, and the security subset is scored using a deterministic scoring.py file.
- Independent AuditAny third party can rerun a single task offline and directly inspect its contents, achieving end-to-end reproducibility and auditability.
WorkBuddy Bench's core advantages
- Actual job distributionThe task categories, modes, and difficulties are all matched to the internal classification system used in practice, and are not constructed based on public question banks or tutorials.
- Fully open source and auditableThe task directory, environment image, evaluation code, scoring test, and reference answers are all publicly available, unlike closed vendor benchmarks such as CursorBench.
- Multi-dimensional scoring systemThe code uses hidden testing, the web page uses a triple evaluation of rules, semantics, and interaction, the office uses rule-based and evidence-driven LLM evaluation, and security uses deterministic scripts. Each subset of the scoring tools is designed independently.
- Diversity of characters and scenesThe code subset introduces five requester roles, and the security subset covers the full spectrum of red and blue teams, avoiding evaluation bias caused by a single task type.
- Decoupling efficiency and capabilityThe leaderboard also reports token consumption and round efficiency. GPT-5.5 achieved the top score with the minimum output budget, proving that a high score does not equal high consumption.
WorkBuddy Bench's project address
- Project official website：https://workbuddybench.com/
- GitHub repository：https://github.com/Tencent/workbuddy-bench
- HuggingFace model library：https://huggingface.co/datasets/tencent/workbuddy-bench
- Technical Papers：https://workbuddybench.com/report/main.pdf
WorkBuddy Bench Comparison
| Dimension | WorkBuddy Bench | SWE-bench |
|---|---|---|
| Task Source | Real commit/PR/reverse engineering of business scenarios | Public GitHub issue |
| Anti-pollution methods | Build-level closed search path + version management | Depends on novelty at the time of release |
| Coverage Area | Four areas: Code / Web / Office / Security | Code Defect Fixes Only |
| Open source level | Tasks, mirror images, ratings, and answers are all publicly available. | Public task set and testing |
| Request style | Colloquialism, role-playing, and insufficient information | Detailed technical issue description |
| Evaluation Framework | Dual-framework (CodeBuddy + Claude Code) | Single framework |
Application scenarios of WorkBuddy Bench
- Intelligent Agent R&D BenchmarkIt provides standardized capability benchmarks for coded intelligent agents, front-end AI tools, office automation agents, and security AI.
- Model Selection ReferenceThe cross-model leaderboard covers mainstream models such as Claude Opus 4.8, GPT-5.5, GLM-5.2, and DeepSeek-V4, assisting enterprises in model selection.
- Academic Research and ComparisonIt features a fully open-source task set and scoring protocol, supporting third-party reproduction, auditing, and improvement, and driving the iteration of domain benchmarks.
- Product iteration and optimizationBy analyzing subset scores and token efficiency data, we can pinpoint the weaknesses of intelligent agents in specific areas such as code navigation, front-end state management, and cross-file consistency.
- Security capability assessmentA separate security subset provides a quantifiable attack and defense testing environment for the capabilities of AI red and blue teams.