E-Bench - A benchmark for evaluating intelligent agents launched by Tencent Hunyuan and others.

E-BenchWhat is it?

E-Bench is a multi-step tool usage evaluation benchmark developed by Tencent's Hunyuan team in collaboration with Tsinghua University's AIR and Southeast University. Based on three real-world products—Honor of Kings, QQ Music, and Tencent Meeting—it constructs a fully synthetic virtual environment containing 323 state-change tasks, 41 database tables, and over 76,000 data entries. Through a dual asymmetric design of the information gap and the tool gap, it forces the agent to actively collect hidden information and orchestrate multi-step tool calls. Ultimately, a deterministic database state diff score is used to reveal the current capabilities of AI agents in real-world product scenarios.

E-BenchMain functions

• Construction of fully synthetic product environmentBased on graph-guided database population technology, we generate virtual worlds for three major products that are free of orphaned records and consistent across tables.
• Double Gap Mission GenerationThe question-generating model has full access to the database, while the tested model can only interact through limited business tools, creating a dual asymmetry of information and tools.
• Deterministic state evaluationThe system uses the diff of the database state before and after the operation as the sole criterion for evaluation. Only accurate matching is considered passing, thus eliminating partial scores and judges' variance.
• Cost-performance joint analysisSynchronously record the API overhead for each task, plot the Pareto front of accuracy versus cost, and evaluate the model's cost-effectiveness.
• Code execution extensionE-Bench Code version released exec_code Tools can separate the two types of difficulty: the arrangement mechanism and the information acquisition.

E-BenchTechnical principles

• Graph guides database populationStarting from a relational schema, a table-level dependency graph is constructed, and the root table and downstream tables are populated in topological order. A strong LLM is used as a constrained synthesizer, and data is written only through the insertion tool. Downstream tables must query candidate entities from the current database. Finally, a deterministic script is used to verify and correct semantic errors such as time order and aggregation count, ensuring that the environment is complete and self-consistent.
• Generator-solver asymmetryThe question-setting agent possesses... query_sql and exec_code Privileges allow the user to view the entire database and perform complex calculations; the tested Agent can only call business-level tools, creating an information gap (global state hiding) and a tool gap (removal of internal computing tools), forcing the model to actively explore the environment through multi-step parallel tool calls.
• Intent rewriting and rule replacementReplace the specific value derived from the database with the rule that generated it, such as "add all songs that have been taken down from the collection" instead of directly giving the song ID, to ensure that the task cannot be completed independently of the environment.
• Cross-validation task generationEach task is generated through a productized cycle of reviewing data, defining objectives, modifying status, and summarizing intent. It is cross-validated by three different strong agent models, and only tasks that are consistently reproduced by at least two models are adopted, ensuring task solvability and labeling accuracy.

How to use E-Bench

• Environmental preparationThe E-Bench fully synthesized virtual environment was obtained from the paper's supporting resources, and database templates covering three major product domains—Honor of Kings, QQ Music, and Tencent Meeting—and 323 evaluation tasks were loaded.
• Version SelectionChoose either the basic version of E-Bench or E-Bench-Code based on your research objectives.
• Model AccessThe LLM Agent under test is integrated into the evaluation framework and configured to interact with the environment indirectly only through restricted domain tools (such as searching for songs, creating meetings, adding friends, etc.), and is prohibited from directly querying the underlying database.
• Task ConfigurationSet the number of times each task can run independently and the conditions for starting a new database copy. Enable API call cost tracking to ensure that the results are reproducible.
• Execution Evaluation: Initiate automated evaluation. After receiving natural language instructions, the agent interacts with the virtual environment and modifies its status through multi-step tool calls. The system automatically records the complete operation trajectory and token consumption.
• Results AnalysisAfter the evaluation, the system outputs the accuracy of Pass@1 and Pass³ based on the deterministic database state diff, and generates a cost-performance joint analysis report to help evaluate the model's cost-effectiveness.

E-Bench's core advantages

• Real-scale interference is sufficientEach library is filled with 76,000+ rows and 600,000+ data units. The target entity is surrounded by a large number of approximate records, and the model cannot rely on environmental sparsity to guess.
• Environment and task decoupling and reuseA self-consistent product environment can support the generation of approximately one hundred tasks with extremely low marginal costs, and the environment layer is controllable while the task layer is scalable.
• The test results are stable and reproducible.The fully synthetic environment is unaffected by the evolution of online services, and the deterministic diff scoring eliminates the subjective bias of semantic judgment.
• Clear Difficulty LevelsThe basic version tests multi-step orchestration and information gathering, while the code version additionally tests the ability to unload code for computationally intensive tasks.

E-Bench project address

• arXiv technical paper：https://arxiv.org/pdf/2607.23722

E-Bench's Competitive Product Comparison

Application scenarios of E-Bench

• AI agent capability assessmentProvides standardized, quantifiable, multi-step tool usage capability assessment for LLM Agents, covering core dimensions such as information retrieval, state change, and cross-entity orchestration.
• Model selection and productization decisionBy using cost-performance Pareto charts, we help businesses weigh accuracy against API overhead and identify cost-effective models.
• Agent training data synthesisAs a controllable and scalable source of training data, it is used to continuously improve the Agent's ability to invoke multi-step tools and its reliability.
• Code Enhancement Agent ResearchE-Bench-Code supports researching the payoff frontier of code execution for different types of tasks (computationally intensive vs. inference-based decision-making).
• Cross-domain benchmark extensionThe methodology can be migrated to more product backends and CLI scenarios, promoting the establishment of general intelligent agent evaluation standards.