HarnessEval - a benchmarking system jointly launched by MirroS, Tsinghua University, NVIDIA, and others.

What is HarnessEval?

HarnessEval is an evaluation system developed by MirroS in collaboration with institutions such as Tsinghua University, Peking University, Berkeley, MIT, and NVIDIA. HarnessEval agentizes the evaluation process. For each case, the system dynamically plans the evaluation path, selects applicable skills from the Skill Library, breaks them down into verifiable sub-problems, calls upon tools to collect evidence, and finally outputs a traceable evidence tree. HarnessEval breaks through the limitations of traditional static benchmarks, transforming evaluation from a black-box scoring system into an interpretable and reproducible intelligent workflow. The first implemented project, HarnessEval-W, has been used for evaluating interactive world models.

HarnessEval's main functions

• Dynamic evaluation planningA customized evaluation plan is automatically generated for each case, taking into account its context and objectives.
• Skill routingInstead of running all the fixed metrics, select applicable skills from the Skill Library as needed.
• Problem BreakdownBreak down abstract evaluation problems into measurable and verifiable subproblems.
• Evidence collectionIt invokes sub-agents and diagnostic tools to collect visual and logical evidence from different perspectives.
• Evidence verificationThe main agent verifies the quality and logical relationships of the evidence in each branch to ensure the reliability of the conclusion.
• Evidence Tree OutputOutput a complete evidence tree, recording test content, tool calls, inference chain, and final score.
• Extensible Skill LibraryIt has 9 built-in core skill modules and supports the dynamic addition of new skills as the model evolves.
• Automatic generation of evaluation casesThe system automatically generates the initial world, actions, and verification based on scene taxonomy to ensure the quality of the evaluation.

The technical principles of HarnessEval

• Four-layer Agentic workflowHarnessEval employs a four-layer workflow: Plan-Router-Decompose-Verify. The Plan phase understands the case context and generates a customized evaluation plan; the Route phase dynamically selects applicable skills from the Skill Library; the Decompose phase breaks down high-level problems into sub-problems and delegates them to sub-agents; and the Verify phase has the main agent verify the quality of evidence and logical relationships to form a final conclusion.
• Hierarchical intelligent agent architectureThe system adopts a main intelligent agent for overall planning and verification, with each Skill Agent responsible for specific evaluations. Each Skill is further broken down into multiple sub-agents to handle specific sub-tasks, forming a hierarchical evaluation execution structure.
• Extensible Skill LibraryThe framework has 9 built-in core skill modules, covering dimensions such as rendering quality, physical rationality, state change verification, drift analysis, revisit consistency, and off-screen evolution. It also supports the dynamic addition of new skills as the model's capabilities evolve, enabling continuous expansion of evaluation capabilities.
• Evidence Tree Output MechanismThe evaluation results are presented in a hierarchical evidence tree format, which fully records the test content, tool calls, visual evidence and reasoning chain, making the evaluation conclusions verifiable, reproducible and accountable, rather than just outputting a single scalar score.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use HarnessEval

• Cloning repositoryPull the HarnessEval-W codebase from GitHub.
• Creating an environmentCreate separately harnesseval-main、harnesseval-metrics and harnesseval-pavrm Three conda environments.
• Configuration credentials:copy config/example.env for harnesseval.env And load environment variables.
• Prepare dataPlace the video results generated by the model to be evaluated into the specified directory and prepare them. manifest.json List file.
• Execution Evaluation:run harnesseval eval The command specifies the path to the model results, the evaluation plan, and the output directory.
• Verification results:run harnesseval verify run Check whether the evaluation output is complete and whether the score has been generated.
• View reportView in the output directory summary.json、leaderboard_latest.csv and LEADERBOARD.md Get detailed ratings and rankings.

HarnessEval's core advantages

• Dynamic evaluation planningInstead of applying a fixed process, it automatically generates customized evaluation plans based on the context of each case.
• Intelligent Skill RoutingSelect applicable skills from the Skill Library as needed to avoid running irrelevant metrics for all cases.
• Explainable chain of evidenceOutputs a complete Evidence Tree, tracing the tool calls, visual evidence, and reasoning logic behind each score.
• Scalable skill librarySkill Library supports dynamically adding new skills, continuously expanding the evaluation dimensions as the model's capabilities evolve.
• Hierarchical agent collaborationThe main intelligent agent coordinates planning and verification, while sub-agents handle sub-problems, enabling the refined decomposition of complex evaluation tasks.
• Evaluation data is generated automatically.Based on scene taxonomy, the system automatically constructs the initial world, actions, and verification to ensure the quality and diversity of evaluation cases.
• The evaluation process is transparent and auditable.A complete chain of reasoning and evidence records make the evaluation conclusions verifiable, reproducible, and accountable.
• Human-like evaluation methodsIt simulates the evaluation workflow of human experts, which involves "understanding, planning, evidence gathering, and reasoning," rather than mechanical scoring.

HarnessEval's project address

• Project official website：https://mirros.ai/blog/harnesseval
• GitHub repository：https://github.com/mirros-lab/harnesseval-w

Comparison of HarnessEval with similar products

Application scenarios of HarnessEval

• Interactive World Model EvaluationTo evaluate the correctness, physical plausibility, and world continuity of the video generation model under action conditions.
• Embodied Intelligence and Robot SimulationThe evaluation aims to assess a robot's maneuverability in a virtual environment, as well as the causal nature of its physical interactions and its long-term state consistency.
• Autonomous driving world model: Verify the camera trajectory adherence, object permanence, off-screen evolution, and physical dynamics rationality in the generated scene.
• Game and Virtual World Generation: Evaluate scene consistency, NPC behavior causality, revisit consistency, and visual quality in open-world games.
• Physical simulation and scientific computing: To examine the physical laws and causal fidelity of the generated video.
• AI video creation platformProvides interpretable quality diagnostics for models such as Seedance, Kling, and Sora, and identifies specific failure modes (such as additional events, target drift, and physical violations).