LoHoSearch - Meituan's next-generation search intelligence benchmark

What is LoHoSearch?

LoHoSearch is a next-generation search agent benchmark launched by Meituan's LongCat team. It automatically generates questions based on a knowledge graph covering 7.62 million Wikipedia entities and 265 million relationship edges, replacing the traditional manual question generation method. The benchmark includes 544 high-quality, human-verified questions covering 11 domains, including music, film, and geography, creating challenges by controlling both search space and structural complexity. The current top-performing model, GPT-5.5, has an accuracy of only 34.74%, far lower than its performance of over 90% on BrowseComp, providing a more discriminative evaluation standard for long-range search reasoning and context management.

LoHoSearch's main functions

• Automated knowledge graph constructionUsing the complete English Wikipedia as the data source, a massive knowledge graph with 7.62 million entity nodes and 265 million directed edges was constructed. Each entity was labeled with Wikidata type and in-degree popularity, providing a foundation for topic selection from a global perspective.
• Dual-dimensional difficulty controlThe system regulates the difficulty of questions from two dimensions: search space and structural complexity, breaking through the cognitive ceiling of manual question generation.
• Tree structure and graph structure samplingTree structures increase the difficulty by expanding the search space, while graph structures introduce circular dependencies and cross constraints on top of the huge search space, further increasing the structural complexity.
• Automated QA generation and verificationThe wiki descriptions are extracted from the subgraph and rewritten into natural language questions. The questions undergo three layers of automatic filtering: relation extraction, subgraph coverage check, and answer satisfaction verification. These are then supplemented by manual review to ensure quality.
• Multi-model performance evaluationIt supports standardized evaluation of mainstream search agents, outputting core indicators such as accuracy, number of tool calls, and calibration error, which intuitively reflect the model's long-range inference capabilities.

LoHoSearch's technical principles

• Automated knowledge graph constructionUsing the complete English Wikipedia as the data source, 7.62 million entity pages were extracted as nodes, and 265 million hyperlinks in the text were extracted as directed edges. Each entity was labeled with Wikidata P31 type and in-degree popularity to construct a global knowledge network.
• Two-dimensional difficulty definitionThe difficulty of questions is quantified from two orthogonal dimensions: "search space" and "structural complexity," breaking through the cognitive limitations of manually generated questions.
• Tree structure subgraph samplingBy expanding the search space to create difficulty, multiple independent branches are expanded downward from a single answer node. Each branch corresponds to a constraint condition of a high candidate space, which makes the elimination cost increase exponentially.
• Graph structure subgraph samplingBased on the huge search space, circular dependencies and cross constraints are introduced. Multiple condition nodes are interconnected with each other. The solution needs to satisfy multiple sets of interlocking relationships at the same time, which adds to the structural complexity and creates a double challenge.
• Relation extraction and natural language generationExtract Wikipedia descriptions of each entity from the sampled subgraph, rewrite them into coherent natural language questions using a large model, and ensure that each clue fully preserves the relational information in the original subgraph.

How to use LoHoSearch

• Loading data:pass datasets Direct library loading meituan-longcat/LoHoSearchGet 544 questions and standard answers.
• View structureThe dataset contains the question text, the corresponding subgraph structure, the domain label, and the unique answer verified by the knowledge graph.
• Access EvaluationInput the problem into the search agent to be tested, and record its reasoning trajectory and final output.
• Automatic scoringCompare the model output with the standard answers provided in the dataset to verify whether all constraints are met and calculate the accuracy.
• Comparative analysisUsing the subgraph complexity and domain labels attached to the dataset, we analyze the model's performance under different difficulties and topics from multiple dimensions.

LoHoSearch's core advantages

• Breaking through the ceiling of manual question generationTraditional manual benchmarks are limited by the scope of entities known to the annotator. LoHoSearch, relying on a global knowledge graph, can systematically identify "real problems" with high search space and high structural complexity.
• The difficulty is significantly higher than existing benchmarks.The same model achieved an accuracy of 58.84% on BrowseComp but only 10.02% on LoHoSearch. The average number of tool calls increased from 35 to 61, and the challenge intensity increased by 74%.
• Stronger differentiationThe existing context management strategy improves performance by 14.03% on BrowseComp but only by 6.8% on LoHoSearch, thus more accurately reflecting the marginal benefits of the strategy in long-term, complex scenarios.
• Structural complexity is independent and controllableThe accuracy rate for graph structure problems (8.01%) was significantly lower than that for tree structure problems (11.89%), proving that structural complexity is an additional source of difficulty independent of the search space and can be quantified separately.

LoHoSearch's project address

• HuggingFace model library：https://huggingface.co/datasets/meituan-longcat/LoHoSearch
• arXiv technical paper：https://arxiv.org/pdf/2606.12837

LoHoSearch's Competitive Comparison

Dimension	LoHoSearch	BrowseComp
Question format	Automated knowledge graph generation + manual verification	Artificial design
Number of questions	544	Hundreds of
Coverage Area	11 fields (music, film and television, geography, etc.)	Undefined field
Difficulty control	Search space + structural complexity (two dimensions)	Human subjective control
The highest model accuracy	GPT-5.5：34.74%	GPT-5.5 Pro：90.1%
Median of tool calls	59 times	26 times
Contextual policy promotion	+6.8%	+14.03%
saturation state	Far from saturated, with high differentiation	It is close to saturation.
Applicable Scenarios	Long-range search reasoning and context management evaluation	Basic search ability assessment

Application scenarios of LoHoSearch

• Search agent capability evaluationIt provides highly challenging standardized tests for search agents with long-range reasoning capabilities, accurately identifying the model's weaknesses in complex information retrieval chains.
• Context management strategy verificationEvaluate the real gains of context compression and verification strategies such as Summary, Discard-all, and Verify in scenarios with extremely long interactions, and avoid overestimating the effectiveness of these strategies on simple benchmarks.
• Model Iteration BenchmarkingIt provides research teams with unsaturated evaluation criteria to support capability boundary detection and performance comparison during model version iteration.
• Research on Search AlgorithmsIt provides datasets and evaluation frameworks with controllable difficulty for research on complex search algorithms that require multi-step reasoning and cross-validation.
• Smart Agent Product SelectionLoHoSearch helps companies differentiate the actual performance of different models in real-world, complex query scenarios when selecting search agent solutions.