PerceptionBench - An open-source visual perception diagnostic benchmark from the Dark Side of the Moon

What is PerceptionBench?

PerceptionBench is an open-source visual perception diagnostic benchmark from Dark Side of the Moon. From failure cases of cutting-edge models in 42 existing benchmark sets, PerceptionBench identifies 10 atomic visual perception capabilities and constructs 3,000 isolated verification questions, each testing only a single perceptual capability without requiring reasoning or external knowledge. Tests show that none of the current 16 mainstream multimodal models achieve 60% accuracy, revealing that atomic-level visual perception remains an unsolved problem in the industry.

The main functions of PerceptionBench

• Atomic ability diagnosticsVisual perception is broken down into 10 atomic capabilities, including visual relationships, counting, attributes, depth, localization, combination, fine-grained recognition, context, OCR, and illusion, to accurately locate model breakpoints.
• Isolated evaluationEach question tests only a single atom's ability, avoiding interference from reasoning and knowledge, and ensuring that the score reflects the true level of perception.
• Failure case attributionBased on failure cases from 42 existing benchmarks, 1,800 atomic sub-problems were decomposed, and evaluation dimensions were extracted from real errors.
• Comprehensive ability profileThe Leaderboard displays detailed scores for the model across 10 capabilities, revealing starkly different capability distributions despite similar total scores.
• Open source and reproducibleProvides complete code, the Hugging Face dataset, and the paper, supporting community reproduction and expansion.

The technical principle of PerceptionBench

• Bottom-up error classificationBy diagnosing the earliest failure point of the frontier MLLM in 42 existing benchmarks, an error classification tree was constructed, and 10 atomic capabilities of the perception branch were extracted.
• Difficulty Stratification and Ability BalanceFrom an internal pool of 17,000+ validation samples, 3,000 questions were sampled in a balanced and stratified manner based on ability and difficulty. 60% of these questions were sub-questions derived from failed cases, and 40% were newly created questions based on supplementary images.
• Short answer unambiguous designAll questions require short and clear answers to reduce assessment uncertainty and make automated evaluation more reliable.
• Perception-Inference DecouplingThe question design ensures that the challenge comes from visual perception itself, rather than from reasoning chains or domain knowledge, thus isolating the boundaries of measurement perception.

Follow us on WeChat and reply with "open source",join inAI open source project discussion group

How to use PerceptionBench

• Cloning repositoryPull from GitHub MoonshotAI/PerceptionBench Open source code.
• Download dataset: Obtain 3,000 validation questions and labeled data from Hugging Face.
• Access ModelThe multimodal model to be evaluated is connected to the evaluation framework, and the answer is output in a standard format.
• Operational evaluationExecute the evaluation script to automatically calculate the model's accuracy in 10 atomic capabilities and overall.
• View Image: By comparing the model's capabilities with the Leaderboard, we can identify its shortcomings and guide the direction of subsequent training and optimization.

PerceptionBench's core advantages

• Precise positioning rather than general scoringUnlike traditional benchmarks that conflate perception, reasoning, and knowledge, PerceptionBench can accurately diagnose which perception stage of the model is malfunctioning.
• Growing from real failuresThe evaluation is based on real-world failure cases from 42 mainstream benchmarks, ensuring that the assessment closely reflects the weaknesses of actual models.
• Exposing hidden capability gapsModels with similar total scores may have vastly different atomic capabilities, and PerceptionBench's capability profile can reveal these gaps hidden by the total score.
• Promote the development of perception first: Shift the industry focus away from score boosting, retention, and perception fundamentals, and promote a more faithful and consistent visual understanding ability.