PAST-Bench - A performance attribution benchmark for personal agents launched by Princeton.

What is PAST-Bench?

PAST-Bench is a benchmark test developed by Mengdi Wang's team at Princeton University to examine the recursive self-improvement capabilities of personal AI agents. PAST-Bench compares task performance with and without memory conditions to determine whether the agent's saved cross-session experiences truly improve subsequent behavior. PAST-Bench covers four capabilities: memory, process reuse, information gathering, and updating, encompassing 26 scenarios and 204 task rounds.

PAST-Bench's main functions

• Evaluation of recursive self-improvementTo examine whether the cross-session experiences (memories, skills, task history) stored by a personal AI agent truly improve subsequent behavior.
• The effect of accumulating isolation experienceDistinguish between "score improvement from experience accumulation" and other factors such as "strengthening of the basic model/changes in the prompt/task difficulty".
• Four-dimensional ability diagnosisIt covers four dimensions: memory, process reuse, information collection, and status update, to pinpoint which specific capabilities of the Agent are deficient.
• Mechanism Attribution AnalysisIt not only looks at the final score, but also tracks the complete path of "storage → retrieval → application" to determine whether the improvement is supported by a real path.

PAST-Bench Technical Principles

• Task family sequence designThe agent executes multiple rounds of sessions within the same task family sequentially. Early sessions create opportunities to save experiences, while later sessions verify whether these experiences have been used correctly.
• Matched control experimentA control version is designed for each later session, with persistence disabled, while all other conditions remain identical. The self-evolutionary difference Δ is obtained by subtracting the score with memory from the score without memory.
• Mechanism Evidence TrackingAt runtime, it records telemetry data such as memory writes, skill calls, session retrieval, and status updates, calculates the Mechanism-Evidence Score, and verifies whether the improvements follow the expected path.
• Persistent product auditExamine the actual content saved by the Agent (memory entries, skill files, session indexes), and analyze its structure, timeliness, and reusability.

How to use PAST-Bench

• Environmental preparationClone the PAST-Bench repository from GitHub and install the core dependencies and Agent adapter.
• Model ConfigurationConfigure the API Key for the model being used in the environment variables.
• Sandbox building:implement past-bench build-image --kind sandbox Build the Docker sandbox image required for the evaluation.
• Quick Verification:use past-bench evolve Run a single task family and add --compare-no-persistence Perform a Smoke Test on the parameters.
• Full assessmentThe evaluation process iterates through all 26 task families, with each task family automatically executing two sets of control experiments: one with persistence and one without.
• Results Analysis:exist --trace-dir View in directory sequence_comparison.json, and obtain the Δ value and mechanism evidence score.
• Custom AccessIf you need to evaluate your own agents, in agents/ Implement the adapter in the directory and ensure support. --compare-no-persistence Simply turn it on or off.

PAST-Bench's core advantages

• True attribution abilityPAST-Bench can accurately distinguish whether improvements come from accumulated experience, a stronger model, changes in the prompt, or a simplified task.
• Matched control experimental designEach task family is equipped with a control version with persistence disabled, all other conditions are exactly the same, to achieve a direct causal comparison between memory and no memory.
• Mechanism-level diagnosisThe system proposes a Mechanism-Evidence Score to determine whether the Agent answers correctly. It also tracks the complete path of "storage → retrieval → application" and identifies the difference between "accidental correct answer" and "genuine reuse of experience".
• Four-dimensional capability breakdownThe vague concept of self-improvement is broken down into four specific capabilities: memory, process reuse, information gathering, and status updating, so as to accurately identify shortcomings.
• Diagnostic-driven improvementsThe runtime failures exposed by the benchmark directly led to the development of the Hermes+ framework, proving that it is not only an evaluation tool, but also a diagnostic engine for agent architecture optimization.

PAST-Bench project address

• GitHub repository：https://github.com/Gen-Verse/PAST-Bench
• arXiv technical paper：https://arxiv.org/pdf/2608.04003

Comparison of PAST-Bench products with similar competitors

Application scenarios of PAST-Bench

• academic researchIt provides a standardized, reproducible, and quantitative evaluation environment for Agent recursive self-improvement, supporting objective comparisons of different architectures.
• Framework developmentBy breaking down the capabilities of the four dimensions, we can accurately identify the shortcomings of the Agent framework and guide the optimization of the architecture (such as the creation of Hermes+).
• Model selection: Compare the cross-session experience reuse performance of different base models under a fixed framework to identify the capability bias of each model.
• Persistent verificationTo test the actual effects of different memory structures and retrieval strategies, and to avoid ineffective persistence such as "stored but not found" or "stored but not used".
• Product iterationDistinguish whether the score improvement in the new version comes from "accumulated cross-session experience" or "strengthened basic model" to ensure that the persistence function truly generates value.