UNO-Bench - Meituan LongCat's full-modality large model evaluation benchmark

What is UNO-Bench?

UNO-Bench is a multimodal large model evaluation benchmark launched by the Meituan LongCat team. Addressing the shortcomings of existing evaluation systems, UNO-Bench accurately measures the single-modal and multimodal capabilities of models through high-quality, diverse data construction. The benchmark is the first to verify the "combinatorial law" of multimodal large models, revealing the complex relationship between single-modal and multimodal capabilities. UNO-Bench's innovative multi-step open-ended questions and efficient data compression algorithms improve the discrimination and efficiency of the evaluation, providing a scientific evaluation tool to promote the development of multimodal large models.

Main functions of UNO-Bench

• Accurate assessment model capabilities: Measure the performance of models on both unimodal and multimodal tasks, including images, audio, video, and text, using high-quality, diverse datasets.
• Revealing the laws of ability combinationThis study is the first to verify the "combination law" of a full-modal large model, revealing the complex relationship between single-modal and full-modal capabilities and providing theoretical support for model optimization.
• Innovative evaluation methodsIntroducing multi-step open-ended questions (MOs) can effectively assess the degradation of a model's capabilities in complex reasoning tasks and accurately distinguish the model's reasoning depth.
• Efficient data managementBy using cluster-guided stratified sampling, the evaluation cost is significantly reduced while maintaining a high degree of consistency in model rankings.
• Support multimodal fusion researchIt provides researchers with a unified evaluation framework, promotes the development of multimodal large models, and reserves space for the emergence of stronger models in the future.

The technical principle of UNO-Bench

• Unified capability systemThe model's capabilities are broken down into a perception layer and an inference layer. The perception layer covers basic recognition and cross-modal alignment capabilities, while the inference layer includes higher-order tasks such as spatial reasoning and temporal reasoning. This two-dimensional framework provides a clear blueprint for data construction and model evaluation.
• High-quality data construction:
  • Data collection and labelingThrough manual annotation and multiple rounds of quality checks, we ensure the high quality and diversity of our data. Over 90% of the data is proprietary and original, preventing data contamination.
  • Cross-modal solvabilityThrough modal ablation experiments, we ensure that more than 98% of the problems can only be solved by relying on multimodal information, thus avoiding the redundancy of single-modal information.
  • Audiovisual separation and recombinationIndependently design audio content and manually combine it with visual materials to break information redundancy and force the model to perform true cross-modal fusion.
  • Data optimization and compressionThe method employs cluster-guided stratified sampling to select representative samples from large-scale data, reducing evaluation costs while maintaining consistency in model rankings.
• Innovative evaluation methodsThis approach breaks down complex reasoning tasks into multiple sub-questions, requiring the model to provide open-ended text answers. Expert-weighted scoring accurately evaluates the model's reasoning ability. Through subdividing question types and multiple rounds of annotation iterations, it achieves automatic scoring for various question types with an accuracy rate of up to 95%.
• Verification of the law of combinationThrough regression analysis and ablation experiments, it is revealed that the full-modal performance is not a simple linear superposition of single-modal capabilities, but follows a power-law synergistic principle. This nonlinear relationship provides a new analytical paradigm for evaluating model fusion efficiency.

UNO-Bench project address

• Project official website: https://meituan-longcat.github.io/UNO-Bench/
• GitHub repository: https://github.com/meituan-longcat/UNO-Bench
• HuggingFace model libraryhttps://huggingface.co/datasets/meituan-longcat/UNO-Bench
• arXiv technical paperhttps://arxiv.org/pdf/2510.18915

Application scenarios of UNO-Bench

• Model Development and OptimizationIt provides developers with standardized evaluation tools to help optimize model architecture and improve multimodal fusion capabilities.
• Industry application assessmentIt can be used in fields such as intelligent customer service and autonomous driving to evaluate the performance of models in multimodal interaction scenarios and optimize user experience.
• Academic research and competitionAs a unified academic evaluation benchmark, it supports model performance comparison and multimodal competition, and promotes technological breakthroughs.
• Product Development and Market EvaluationIt helps companies evaluate product functionality and market competitiveness, and provides a scientific basis for the development of multimodal products.
• Cross-modal application developmentIt supports multimedia content creation and intelligent security, improving the performance and reliability of multimodal applications.