MineExplorer - Meituan's benchmark for evaluating open-world, minute-level long-duration tasks. - AiBoss

What is MineExplorer?

MineExplorer, developed by the Meituan LongCat team, is the first benchmark for evaluating open-world, minute-level long-duration tasks based on Minecraft. MineExplorer includes 813 manually verified instances with 1-4 hops. Tasks contain hidden preconditions, and instructions do not enumerate dependency graphs, requiring the model to autonomously infer subtasks. MineExplorer actively removes game-specific knowledge, testing only general dynamic exploration capabilities. It is accompanied by a five-agent collaborative data synthesis and rule-based milestone automatic evaluation framework; the code and dataset are now open source.

MineExplorer's main functions

Minute-long open-world reviewBased on Minecraft's real-time 3D sandbox, each task runs 1800 environment steps, and the model needs to continuously adjust its strategy according to the dynamic visuals.
Hidden multi-jump tasksIt provides 813 1-4 hop manual verification examples. The instructions only give the final goal and hide the preceding subtasks. The model needs to infer the dependency graph on its own.
Knowledge decoupling assessmentActively strip away Minecraft-specific mechanics, retaining only quests that rely on common world knowledge.
Rule-based automated evaluationTransform subtasks into automated milestone checkers, enabling automatic scoring without manual annotation.
Multi-agent data synthesisFive agents collaborate to automatically generate tasks, including the entire process of task selection, scenario design, milestone design, expert review and verification.

MineExplorer's technical principles

Dynamic environment interaction architectureThe evaluation environment is a real-time running physical sandbox, not a static screenshot question-and-answer environment; the world state is updated in real time for each action the model performs, and continuous decision-making is made based on the latest observations.
Implicit DAG task modelingEach composite task is defined as a quadruple τ=(q,s₀,Gτ,Mτ), where Gτ is the task dependency graph. The key design is that instruction q does not enumerate all nodes in the DAG, and the agent must infer the hidden preceding task from the environment.
Multi-agent collaborative problem-solving processThe system is controlled by an orchestrator. Five professional agents (task selector, scene designer, milestone designer, Minecraft expert, and validator) collaborate in a group chat, generating and revising a first draft in two phases: initialization and debate. This improves efficiency by about 30% compared to a single agent.
Knowledge filtering mechanismThe LLM judge performs domain knowledge assessments on each atomic task, distinguishing between general world knowledge and Minecraft-specific mechanics, and filtering out high-difficulty game-specific tasks that rely on the latter.
ReAct Capability Decomposition FrameworkDrawing inspiration from the ReAct paradigm, the open-world exploration capability system is broken down into three dimensions—perception, reasoning, and action—comprising 14 fine-grained indicators.

How to use MineExplorer

Environmental preparationClone the GitHub repository to your local machine, ensuring that Python 3.9+ and the necessary Minecraft runtime environment are installed.
Start the testing environmentConfigure and start the Minecraft sandbox server to ensure that the evaluation environment can run in real time and receive model action commands.
Operating benchmark evaluationLoad the 813 verification instances provided by MineExplorer, connect the multimodal model to be tested to the environment, and the model outputs action sequences based on the real-time screen.
Automatic ratingAfter the evaluation is completed, the system automatically calls the rule-based milestone checker to calculate the TSR and MSR scores based on quantitative indicators such as backpack items and coordinate areas.
Custom task generationRun the multi-agent data synthesis script, configure the task hop count, and have five collaborating agents automatically generate new task instances with hidden preconditions.
Training and IterationThe generated tasks are used as training data input into the model, and the exploration-feedback loop is repeatedly executed in the Minecraft sandbox to improve the long-term planning ability of the open world.
View resultsThe evaluation report automatically outputs scores for each dimension and supports tiered comparisons by model and difficulty.

MineExplorer's core advantages

Pioneering minute-level open-world evaluationBased on a real-time running Minecraft 3D sandbox, the model needs to make continuous decisions during 3 minutes of dynamic interaction.
Hidden multi-jump tasksThe instructions only give the final goal and do not enumerate the dependency graph, forcing the model to infer the hidden subtasks autonomously.
Knowledge decoupling designActively strip away Minecraft-specific mechanics while retaining common-sense world quests.
Efficient Problem Formulation by Multi-AgentFive agents collaborate to generate tasks, improving efficiency by about 30% compared to a single agent, and supporting 1-12 hops of custom expansion.
Rule-based automated evaluationIt transforms subtasks into automated milestone checkers that can be quantified and scored without manual annotation, and the results are reproducible.

MineExplorer project address

GitHub repository：https://github.com/meituan-longcat/MineExplorer
arXiv technical paper：https://arxiv.org/pdf/2605.30931

Comparison of MineExplorer and similar competing products

Comparison Dimensions	MineExplorer	MineDojo
Evaluation Objectives	Test general open-world exploration capabilities, stripping away game-specific knowledge.	The ability of an embodied intelligent agent to complete tasks and utilize internet knowledge in Minecraft was measured.
Task Design	Jumps 1-4 hide preceding multi-jump tasks; instructions do not enumerate the dependency graph and require independent deduction.	Thousands of procedural and creative tasks, with instructions that clearly state the goals and steps.
Source of knowledge	It relies solely on common world knowledge and actively filters the Minecraft Wiki mechanism.	Deeply integrates large-scale game knowledge bases such as YouTube videos, Wiki, and Reddit.
Data Synthesis	Five agents collaborate to automatically generate tasks, improving efficiency by approximately 30%.	Automatically construct task and reward functions based on multimodal data from the Internet.
Evaluation Method	The rule-based milestone automatic checker covers 14 indicators for perception, reasoning, and action.	For procedural tasks, simulator status is used for determination; for creative tasks, MINECLIP video model is used for scoring.
Open source focus	Evaluation benchmarks and an scalable training environment support 1-12 jump custom tasks.	The training framework and assessment suite emphasize learning generalizable skills from internet knowledge.

Application scenarios of MineExplorer

Multimodal large model capability evaluationIt provides a standardized testing environment for dynamic open-world long-range planning and reasoning for top MLLM programs such as Claude, GPT, and Gemini.
Agent training environmentAs a training ground for the Minecraft sandbox, it supports models to improve their perception-reasoning-action closed-loop capabilities through continuous interaction.
Model Selection Reference: Provides industry with a leaderboard for cross-sectional comparison of open-world exploration capabilities, assisting in selection and capability boundary assessment.
Academic research platformPromote research in cutting-edge areas such as long-range reasoning, implicit task decomposition, visual grounding, and action alignment.
Embodied Intelligence Pre-research and Verification: Provides a low-cost capability verification sandbox for real-world applications such as robotics and autonomous driving.