Meeseeks - Meituan's open-source model instruction compliance assessment set

What is Meeseeks?

Meeseeks is an open-source large-scale model evaluation dataset from Meituan's M17 team, used to assess a model's ability to follow instructions. Meeseeks uses a three-level evaluation framework to comprehensively measure whether a model can strictly generate answers according to user instructions, from macro to micro levels, without evaluating the knowledge correctness of the answers. Meeseeks introduces a multi-round error correction mode, allowing the model to correct itself after receiving feedback, evaluating its self-correction ability. Meeseeks uses objective evaluation standards, rejecting ambiguous instructions to ensure consistency and accuracy of results. Meeseeks' data design is more challenging, effectively differentiating between different models and providing optimization directions for model developers.

Meeseeks' main functions

• Instruction compliance assessment:
  • Level 1 capabilityThe evaluation criteria include whether the model correctly understands the user's core task intent, whether the overall structure of the response meets the instructions, and whether each individual unit in the response conforms to the details of the instructions.
  • Level 2 abilityFocus on how well the model performs on various specific constraints, such as content constraints (topic, style, language, word count, etc.) and format constraints (template compliance, number of units, etc.).
  • Level 3 AbilityThe evaluation model follows fine-grained rules, such as rhyming, keyword avoidance, prohibition of repetition, and symbol usage.
• Multi-round error correction modeIf the model's first-round response fails to fully satisfy all instructions, the evaluation framework automatically generates explicit feedback, indicating which specific instruction was not satisfied, and requires the model to correct its answer based on the feedback.
• Objective evaluation criteria: Abandoning vague instructions, all evaluation items are objective and verifiable standards to ensure the consistency and accuracy of evaluation results.
• High-difficulty data designTest case design is more challenging and can effectively differentiate between different models, providing model developers with clear optimization directions.

Meeseeks's technical principles

• Three-level evaluation framework:
  • Level 1 capabilityThis involves parsing user instructions using Natural Language Processing (NLP) techniques to extract the core task intent and structural requirements. For example, an intent recognition algorithm can be used to determine whether the model understands the task of "generating flower names".
  • Level 2 abilityThis involves performing content and format constraints checks on the model-generated responses. For example, text analysis algorithms can be used to check whether the generated comments meet word limits or use a specified writing style.
  • Level 3 AbilityPerform fine-grained rule checks on the model-generated answers. For example, use regular expressions to check whether the generated comments contain prohibited words or conform to specific writing styles.

Meeseeks' project address

• GitHub repositoryhttps://github.com/ADoublLEN/Meeseeks
• HuggingFace model libraryhttps://huggingface.co/datasets/meituan/Meeseeks

Application scenarios of Meeseeks

• Model Evaluation and OptimizationIt provides a standardized assessment of the instruction-following ability of large models, helping developers to identify and optimize the model's shortcomings in understanding and executing instructions.
• Model training and fine-tuningMeeseeks' evaluation dataset and multi-round error correction feedback serve as supplementary training methods to guide model fine-tuning and improve performance in real-world applications.
• Model Deployment and ApplicationThe evaluation model assesses whether it can strictly follow user instructions and generate high-quality, compliant content in scenarios such as content generation, intelligent customer service, and education.
• Model Research and AnalysisAs a standardized evaluation benchmark, it supports academic research and industry analysis, helps to deeply analyze model performance differences, and explore improvement methods.
• Model safety and complianceTo assess the compliance of the content generated by the model, help ensure that the model output complies with laws, regulations and ethical standards, and protect data privacy.