One-Eval - An open-source automated large model evaluation framework from Peking University

What is One-Eval?

One-Eval is an open-source, automated agent-driven large model evaluation framework developed by the OpenDCAI team at Peking University. Built on DataFlow and LangGraph, it primarily focuses on NL2Eval (Natural Language to Evaluation) capabilities. Users only need to describe the evaluation objective in natural language, and the system can automatically complete benchmark recommendation, data download, inference execution, metric matching, scoring, and multi-dimensional report generation, achieving end-to-end automation from requirement to report.

The main functions of One-Eval

• NL2Eval Intelligent ParsingThe user inputs their natural language requirements, and the agent automatically parses the intent and plans the evaluation path.
• Bench GalleryBuilt-in metadata management for mainstream evaluation benchmarks such as GSM8K, MATH, MMLU, C-Eval, IFEval, and HumanEval.
• End-to-end automatic executionIt automatically completes data download, model reasoning, answer scoring, statistical analysis, and report generation.
• Human-machine collaborative interventionIt supports interrupting, reviewing, editing, and rerunning at key points, and adjusting the evaluation strategy in real time.
• Unified interface for heterogeneous dataStandardize the format and column mapping of different datasets through the DataFlow engine.

The technical principle of One-Eval

• Agent Graph Orchestration ArchitectureBased on LangGraph, a state machine workflow is built, and the evaluation is broken down into three major stages: NL2Bench → BenchResolve → Metrics & Reporting.
• DataFlow operator systemThe underlying layer uses the DataFlow engine to handle data preparation and streaming computation, enabling unified access and transformation of heterogeneous datasets.
• Local-first + HuggingFace dual-source parsingPrioritize loading local pre-configured settings; if missing, automatically call the HuggingFace tool to search, download, and structure the dataset metadata.
• Traceable state managementThe execution state of each step is persisted, supporting breakpoint recovery, backtracking and rerunning, and failure data analysis.

How to use One-Eval

• Environmental preparationCreate and execute virtual environments using Conda or uv. pip install -e . Install dependencies.
• Start the backend:run uvicorn one_eval.server.app:app --host 0.0.0.0 --port 8000 Start the FastAPI service.
• Start the front end:Enter one-eval-web Directory Execution npm install && npm run dev,access localhost:5173.
• Configuration parametersSet the API Key, target model, and HuggingFace Token in the web interface, and then you can start interactive evaluation.
• Initiate evaluationDescribe your needs in natural language in the input box, such as "Evaluate the model's performance on mathematical reasoning tasks," and the Agent will automatically execute and generate a report.
• Developer modeRun directly python -m one_eval.graph.workflow_all "评测目标" Trigger the complete workflow via command line.

Key information and usage requirements for One-Eval

• Development TeamOpenDCAI Lab, Peking University.
• Open source licenseIt is completely open source, and the code is hosted on GitHub and can be obtained directly.
• technology stackThe backend is based on Python 3.11 and FastAPI, the frontend uses React + Vite, the core workflow relies on LangGraph for graph orchestration, and the underlying data processing is driven by the DataFlow operator system.
• Support ModelIt is compatible with any service provider that offers OpenAI standard interfaces, including OpenRouter, Silicon Flow, Volcano Engine, and self-deployed vLLM.
• Hardware RequirementsLocal operation only requires a standard development machine configuration; the computation and token consumption during the actual inference phase depend entirely on the external model API selected by the user.
• Usage thresholdBefore use, a valid API Key needs to be configured for model invocation, and a HuggingFace Token is required to download and load part of the evaluation dataset.

One-Eval's core advantages

• Zero-script evaluationSay goodbye to manually downloading data and writing configurations; natural language can initiate end-to-end evaluations.
• Scalable architectureBased on DataFlow operators and LangGraph state management, private datasets and custom metrics can be easily integrated.
• Human-machine closed loopKey nodes support manual review and intervention, balancing automation and controllability.
• Multi-dimensional reportAutomatically generates score statistics, model rankings, failure case analysis, and actionable suggestions.
• End-to-end tracingThe evaluation process is fully traceable, and the inputs, outputs, and state changes of each node are auditable.

One-Eval project address

• GitHub repositoryhttps://github.com/OpenDCAI/One-Eval
• arXiv technical paper: https://arxiv.org/pdf/2603.09821

Comparison of One-Eval with similar competing products

Comparison Dimensions	One-Eval	OpenCompass	EleutherAI LM Harness
Development Team	Peking University OpenDCAI Lab	Shanghai Artificial Intelligence Laboratory	EleutherAI Community
Core positioning	Agent-driven, NL2Eval automated evaluation	Mainstream Chinese community and configuration driver evaluation	Established overseas scripted evaluation tool
How to use	Natural Language Description + Web UI Interaction + Code Calling	YAML configuration file + command-line script	Python scripts + CLI command line
Entry threshold	Low cost; a review can be initiated with just one sentence.	In this case, a configuration file needs to be written.	In this process, code and scripts need to be written.
Human-machine collaboration	Support; key nodes can be interrupted, reviewed, and intervened upon.	Not supported, purely automated execution	Not supported, purely automated execution
Front-end interface	Built-in React + Vite visual workflow	There is a results display page, but no interactive front-end.	No front-end interface
Chinese Ecology	Native support for C-Eval, CMMLU, etc.	Extremely powerful, with comprehensive coverage of Chinese benchmarks.	It's relatively weak; you need to configure the Chinese dataset yourself.
Heterogeneous data support	Unified access to DataFlow operator systems	Relatively complete	It's relatively complete, but the configuration complexity is high.
Agent/Sandbox Review	In the planning stages (future support for SWE-bench, etc.)	Partial support	Not supported
Applicable Scenarios	Rapid selection, project acceptance, academic experiments	Deep customization, large-scale batch evaluation	English benchmark research, flexible code-level customization

Application scenarios of One-Eval

• Model selection preliminary screeningQuickly compare the performance of multiple candidate models in dimensions such as mathematics, reasoning, code, and instruction compliance.
• Privatization Model AcceptanceStandardized capability acceptance and regression testing are performed on self-deployed or fine-tuned models.
• BenchmarkingQuickly search and configure evaluation sets suitable for specific task types using Bench Gallery.
• academic researchIt provides a reproducible and traceable automated evaluation pipeline for thesis experiments.
• Agent Capability Assessment: Evaluate the performance of LLM in tool invocation, planning, and complex task execution (future support for scenarios such as SWE-bench).