PlanningBench - A large-scale planning capability evaluation framework from open-source platforms such as Tencent's Hunyuan. - AiBoss

What is PlanningBench?

PlanningBench is an open-source framework developed by Tencent's Hunyuan team in collaboration with institutions such as the Gaoling School of Artificial Intelligence at Renmin University of China. It focuses on evaluating and training the planning capabilities of large language models. Starting from real-world scenarios, the framework constructs a verifiable data generation system covering six categories and more than 30 planning tasks. Through a constraint-driven closed-loop synthesis process, it enables model planning results to go from appearing reasonable to truly executable.

PlanningBench's main functions

Real-world scenario coverageIt covers more than 30 specific planning tasks in six major categories: schedule arrangement, resource allocation, manpower scheduling, route scheduling, production operation, and emergency services.
Three-layer constraint systemThe planning difficulty is broken down into basic constraints, medium constraints, and difficult constraints, enabling refined control of task difficulty.
Automated verification mechanismEach instance comes with an executable checklist that supports both Avg-pass and All-pass evaluation metrics.
Closed-loop data generationThe system automatically increases the difficulty of the data through a closed-loop process: Generator generates questions, Responder attempts to answer them, and Critic verifies the answers.
Training signal outputVerifiable samples can be directly used for reinforcement learning training, providing stable optimization signals for the model's planning capabilities.

PlanningBench's technical principles

Construction of Real-world Scene Abstraction and Constraint SystemPlanningBench will be developed through collaboration between experienced annotators and algorithm researchers. It will abstract reusable problem structures from real-world scenarios such as personal commuting, corporate production scheduling, and medical emergencies, ultimately forming six major planning task families and more than 30 specific task types. Constraints will be divided into three levels: basic constraints (time window, resource boundary), medium constraints (multi-objective optimization, dependency relationships), and difficult constraints (anomaly recovery, objective conflict). This will enable data generation to combine tasks and constraints in a structured design space.
Constraint-driven closed-loop synthesis processThe core of the framework is a closed-loop system with three collaborative roles: Generator, Responder, and Critic. The Generator generates a self-contained planning problem and a corresponding verification list based on the sampled tasks and constraints. The Responder model attempts to generate a solution. The Critic checks each item on the checklist to see if the answer meets all the requirements. If it does, the difficulty is increased; otherwise, it is retained as a challenging sample, making PlanningBench a dynamic data generation system that can proactively find the boundaries of the model's capabilities.
Three-tiered difficulty constraint systemPlanningBench breaks down planning difficulty into factors such as task structure, constraint hierarchy, resource stress, goal conflict, dependencies, and anomaly handling. It adjusts the difficulty of generated data by controlling the sampling distribution of the constraint pool. Basic constraints ensure the integrity and executability of the solution; medium constraints introduce fairness and load balancing requirements; and difficult constraints test the model's capabilities in infeasibility identification, minimum change replanning, and robust design, ensuring that increased difficulty truly impacts the depth of planning reasoning.

How to use PlanningBench

Obtain resourcesAccess the GitHub repository to clone the code, or download the dataset from HuggingFace.
Operational evaluationLoad the PlanningBench evaluation set, call the model to generate planning schemes, and use the built-in checklist to automatically calculate Avg-pass and All-pass indicators.
Generate customized dataConfigure task types and constraint levels, run closed-loop generation processes, and produce planning and training data of specific difficulty and domain as needed.
Training ModelThe generated verifiable samples are used as reward signals to train the target model through reinforcement learning, and then transferred to downstream planning tasks.

PlanningBench's core advantages

Verifiability: Accurately determine whether a solution is globally executable by using an automatic checklist.
Difficulty is controllableThe difficulty of the task is adjusted based on the constraint hierarchy rather than simply increasing the length of the prompts, which is closer to the complexity of real planning.
Accurate diagnosisThe Avg-pass and All-pass dual indicators can identify model outputs that are "partially correct but globally unexecutable".
Training transferOnly 300 samples are needed to train the model to improve its performance on external planning benchmarks and general instruction compliance tasks.
Realistic sceneThe tasks are derived from real-world scenarios such as commuting, production scheduling, and medical dispatch, to avoid overfitting the model to a fixed question bank.

PlanningBench project address

GitHub repositoryhttps://github.com/Tencent-Hunyuan/PlanningBench
HuggingFace model libraryhttps://huggingface.co/datasets/tencent/PlanningBench
arXiv technical paperhttps://arxiv.org/pdf/2605.20873

Comparison of PlanningBench with similar products

Dimension	PlanningBench	AgentBench
Core positioning	Focus on large modelsPlanning abilityThe evaluation and training framework emphasizes moving from "seemingly reasonable" to "truly feasible".	Comprehensive large modelIntelligent agent capabilitiesThe evaluation framework covers multiple dimensions, including reasoning, decision-making, tool use, and environmental interaction.
Task Coverage	It covers more than 30 real-world planning tasks across 6 major categories, including schedule arrangement, resource allocation, manpower scheduling, route dispatching, production operations, and emergency services.	It covers eight interactive environments, including operating systems, databases, knowledge graphs, digital cards, lateral thinking, family management, online shopping, and web browsing.
Verification mechanism	Each instance is accompanied by a structured checklist, which automatically verifies the constraint satisfaction item by item and accurately determines whether the solution is globally executable.	Task success is determined by matching environmental execution feedback with results; different environments have independent verification logic.
Evaluation metrics	We use two metrics, Avg-pass (local constraint satisfaction rate) and All-pass (global executability rate), to identify outputs that are "locally correct but globally unexecutable".	The success rate is primarily used, and statistics are collected separately for 8 environments to measure the final completion rate of the task.
Difficulty control	Through a three-layer constraint system (basic/medium/hard) and a closed-loop difficulty enhancement mechanism, the constraint coupling degree and resource conflict intensity are actively adjusted.	The difficulty is increased through multiple rounds of interaction (5-50 rounds) and progressively increasing environmental complexity, focusing more on the depth of interaction than the density of constraints.
Environmental Interaction	For solving static programming problems, the model generates a complete solution under explicit input without needing to interact with the external environment in real time.	The model is required to interact with the simulation environment (such as a bash terminal, browser, or database) in real time and dynamically obtain feedback.
Training support	It natively supports training, and verifiable data can be directly used as reward signals for reinforcement learning such as GRPO, and can be transferred to external benchmarks.	Primarily used for evaluation, the framework itself does not directly provide scalable training data generation capabilities.

Application scenarios of PlanningBench

Model capability diagnosis: Accurately identify the shortcomings of large models in core planning capabilities such as constraint reasoning, resource allocation, and state tracking.
Agent trainingProvide AI agents with verifiable planning and training data to improve their reliability in tasks such as scheduling and logistics.
academic researchAs a benchmark tool for planning capability research, it supports reproducible and scalable model comparison experiments.
Enterprise scheduling optimizationThe framework generates evaluation data for specific business scenarios (such as nurse scheduling and production scheduling) to assess the feasibility of model implementation.
Education Grouping and Resource AllocationThe model's fairness and optimality under multiple constraints were verified using tasks such as student grouping and material distribution.