MME-CoT - The Chinese University of Hong Kong and other institutions have launched a benchmark framework for assessing visual reasoning ability.

What is MME-CoT?

MME-CoT is a benchmark framework jointly developed by the Chinese University of Hong Kong, Shenzhen, ByteDance, Nanjing University, Shanghai Artificial Intelligence Laboratory, University of Pennsylvania, Tsinghua University, and other institutions. It is used to evaluate the chain-of-thought (CoT) reasoning capabilities of large multimodal models (LMMs). Covering six domains—mathematics, science, OCR, logic, spatiotemporal, and general scenarios—it contains 1,130 questions, each annotated with key reasoning steps and reference image descriptions. The MME-CoT benchmark comprehensively evaluates the model's reasoning capabilities based on three novel evaluation metrics: reasoning quality (logical rationality), robustness (interference with perceptual tasks), and efficiency (relevance of reasoning steps). Experimental results reveal some key issues with current multimodal models in CoT reasoning, such as the inefficiency of reflection mechanisms and their negative impact on perceptual tasks.

Main functions of MME-CoT

• Multi-domain reasoning ability assessmentIt covers six major domains (mathematics, science, OCR, logic, spatiotemporal, and general scenarios) and comprehensively evaluates the model's reasoning ability in different scenarios.
• Fine-grained inference quality assessmentBased on the annotation of key reasoning steps and reference image descriptions, evaluate the logical rationality (quality), robustness (interference with the perceptual task), and efficiency (relevance of reasoning steps) of the model's reasoning.
• Revealing the problem of model reasoningThis study reveals the problems of current multimodal models in CoT inference, such as the inefficiency of the reflection mechanism and its interference with perception tasks.
• Provide a reference for model optimizationThe evaluation results and analyses provided offer important references for the design and optimization of multimodal models, helping researchers improve the reasoning capabilities of their models.

MME-CoT Technical Principles

• Multimodal dataset constructionA high-quality multimodal dataset was constructed, containing 1,130 questions covering six domains and 17 subcategories. Each question is annotated with key reasoning steps and reference image descriptions, used to evaluate the model's reasoning process.
• Fine-grained evaluation metrics:
  • Inference quality: Evaluate the logical rationality and accuracy of reasoning steps based on recall and precision.
  • Reasoning robustnessThe impact of CoT on perception and reasoning tasks is evaluated based on stability and effectiveness.
  • Reasoning efficiencyThe relevance of reasoning steps and the effectiveness of reflection are assessed based on the relevance rate and reflection quality.
• Reasoning Steps Analysis and EvaluationThe model output is parsed into logical reasoning, image description and background information using models such as GPT-4o, and each step is evaluated.

MME-CoT project address

• Project official website:https://mmecot.github.io/
• GitHub repository:https://github.com/CaraJ7/MME-CoT
• HuggingFace model library:https://huggingface.co/datasets/CaraJ/MME-CoT
• arXiv technical paper:https://arxiv.org/pdf/2502.09621

Application scenarios of MME-CoT

• Model Evaluation and ComparisonAs a standardized benchmark, it is used to evaluate and compare the performance of different multimodal models in terms of inference quality, robustness, and efficiency.
• Model optimizationBased on fine-grained evaluation metrics, we can reveal problems in the model's inference process and provide direction for model optimization.
• Multimodal researchIt provides tools for multimodal reasoning research, helping to explore new model architectures and training methods.
• Education and TrainingUsed in the field of education to help students and researchers understand the reasoning logic of multimodal models.
• Industry ApplicationsIn fields such as smart education, autonomous driving, and medical imaging, evaluate and improve the practical application performance of the model.