PawBench - A general-purpose intelligent agent evaluation benchmark launched by Alibaba Tongyi

What is PawBench?

PawBench is a general-purpose intelligent agent evaluation benchmark launched by Tongyi Labs, targeting personal assistant and agent scenarios. It incorporates the base model and runtime framework (Harness) into the same evaluation system. PawBench v1.0 has built an evaluation set of 150 real tasks and 4050 test units. Through cross-evaluation of 9 models × 3 Harnesses, it can evaluate the best combination of model + Harness, helping Harness developers accurately locate problems and verify optimizations.

PawBench's main functions

• Joint evaluationThe base model and the operating framework (Harness) are incorporated into the same evaluation system to assess the combined effect of LLM × Harness.
• Real Mission SetWe constructed 150 real-world tasks and 4050 test units, which were extracted and aggregated from 6 high-quality Agent evaluation sets.
• Five-dimensional labeling systemEach question is structured and labeled according to its application scenario, atomic capabilities, complexity (L1-L3), input modality, and runtime environment.
• Hybrid scoring mechanismIt combines an automatic scorer (rule assertion) with LLM-as-judge (semantic quality assessment), with scores ranging from 0 to 1.
• Multi-slice rankingIt supports freely switching between three dimensions for viewing: Overall (150 questions), Text (124 questions), and Multimodal (26 questions).
• Full trajectory retentionAll tasks run in a Docker sandbox, and execution traces, grader artifacts, and environment snapshots are fully preserved.
• In-depth diagnostic analysisIt supports slicing by dimensions such as model size, modality, task type, and skill domain to pinpoint specific behavioral defects in Harness.

PawBench's technical principles

• Cross-evaluation matrixWe constructed a three-dimensional evaluation space consisting of 9 models × 3 Harnesses × 150 tasks, and used the controlled variable method to isolate the contributions of model capabilities and framework capabilities.
• Five-dimensional labeling systemThe task is structured and labeled from five dimensions: application scenario, atomic capability, complexity, input modality, and runtime environment, supporting multi-dimensional slice drill-down analysis.
• Hybrid scoring mechanismIt combines deterministic rule assertions (file existence, field matching, exit code) with LLM-as-judge semantic evaluation to balance objectivity and flexibility.
• Docker sandbox isolationEach test unit runs in an independent container, capturing execution traces, workspace artifacts, and environment snapshots to ensure reproducible results and traceable issues.
• Product-level hard verificationBy checking substantive indicators such as whether the file has been written to disk, whether the diff has been generated, and whether the test has passed, we can avoid the model being "falsely completed".

How to use PawBench

• Cloning source codeAccess the GitHub repository agentscope-ai/PawBench Cloning evaluation framework.
• Configure environmentWrite the API key of the model under test and configure the Harness runtime environment.
• Select task: Select a specific slice (Text/Multimodal) or start the full task from 150 tasks.
• Operational evaluationWhen a task is executed in a Docker sandbox, the system automatically saves the execution path and artifacts.
• Get ratingThe final score is calculated using an automatic scorer and LLM-as-judge with mixed weights.
• Submit the list:access agentscope-ai.github.io/PawBench Submit your results and view your ranking.

PawBench's core advantages

• Diagnosis Harness Through the analysis of 4050 test units, we accurately identified behavioral defects in the framework in areas such as skill loading, path awareness, and tool configuration.
• True reproducibilityIt runs on a Docker sandbox, preserving the complete execution trajectory and environment snapshots, allowing for step-by-step backtracking and review of failure cases.
• Zero-barrier unboxing review: Restore the default experience after the developer's first clone, without pursuing the "theoretical upper limit of having all API keys".
• Continuous regression validationHarness allows for reslicing and re-scoring after each fix to verify whether the optimization truly addresses the problem dimension.
• Harness Differential QuantizationThis demonstrates for the first time that the Harness gap (maximum 6.4 points) is comparable to a major model upgrade, providing a path for smaller models to overtake the competition..

PawBench project address

• Project official websitehttps://agentscope-ai.github.io/PawBench/
• GitHub repositoryhttps://github.com/agentscope-ai/PawBench

PawBench's Competitive Comparison

Application scenarios of PawBench

• Harness Developer Optimization: By analyzing the slices, we can pinpoint the specific deficiencies of the framework in areas such as skill loading, path awareness, and default availability of web tools.
• Model Selection ReferenceIt helps users select the best model + Harness combination based on task types such as plain text, multimodal, skill, and web search.
• Benchmark Regression TestAfter Harness iteratively fixed the issue, re-run the benchmark to verify whether the optimization truly resolved the corresponding slice problem.
• academic researchIt provides a reproducible cross-evaluation benchmark for general intelligent agent research, and promotes the co-evolution of models and frameworks.