CL-bench - A benchmark for evaluating model learning capabilities developed by Tencent's Yao Shunyu team.

What is CL-bench?

CL-bench, jointly launched by Tencent Hunyuan and Fudan University, is a benchmark for evaluating context learning capabilities. It measures the ability of large language models to learn and apply knowledge from entirely new contextual information in real time. The CL-bench benchmark includes 500 complex scenarios meticulously constructed by experts and 1899 tasks, covering four main categories: domain knowledge reasoning, rule system application, procedural task execution, and experience discovery and simulation. Tests show that the most powerful model, GPT-5.1, only achieved a 23.7% solution rate, revealing the core bottleneck of current AI's inability to learn in real-time and pointing to new directions for model optimization.

CL-bench's main functions

• Real-time learning ability assessmentIt supports evaluating the ability of large language models to learn and apply knowledge from new contextual information in real time.
• Building a large-scale test setThe test set includes 500 complex scenarios, 1899 tasks, and 31607 verification criteria, covering four types of real-world scenarios: domain knowledge reasoning, rule system application, procedural task execution, and experience discovery and simulation.
• Pollution-free data designThe design employs a pollution-free approach to ensure data novelty, and prevents models from relying on memory-based non-learning methods to solve problems through fictional creation, modification of existing content, and integration of niche and emerging content.
• Serial dependency verificationThe study validated the model's ability to perform multi-round inference in sequence-dependent tasks, with 51.1% of the tasks requiring subsequent inference based on previous interaction results.
• Multi-dimensional evaluation systemIt provides a multi-dimensional evaluation system, with an average of 16.6 evaluation criteria per task, comprehensively examining the model's understanding and application accuracy of the context from multiple perspectives.

CL-bench technical principles

• Self-contained ContextThe core technology of CL-bench is to build a completely self-contained context environment, ensuring that all the information needed to solve the task is explicitly provided within the context itself, without requiring external retrieval and without supporting hidden hypotheses. It forces the model to learn from the new information in the current input, rather than calling upon internal knowledge stored during the pre-training stage, thus truly reflecting the model's context learning ability rather than its memory ability.
• Triple Pollution-Free StrategyTo achieve pollution-free assessment, CL-bench employs a three-pronged technical strategy: experts create entirely fictional content, such as designing complete legal systems for fictional countries or creating new programming languages with unique syntax; systematically modify real-world content to create variants, including altering historical events, changing scientific definitions, or revising technical documents; and incorporate niche or recently emerging content that is not highly representative of the pre-trained dataset, such as cutting-edge research findings and newly released product manuals.
• Complexity and Verifiable DesignIn terms of task design, CL-bench emphasizes high complexity and sequential dependencies. 51.1% of the tasks involve multi-round interaction mechanisms, with solutions for subsequent tasks depending on the results of previous interactions, significantly increasing task difficulty and simulating real-world work scenarios. Simultaneously, each task has fully verifiable evaluation criteria, with an average of 63.2 validation criteria associated with each context. This multi-faceted evaluation ensures a comprehensive assessment of model performance and avoids evaluation bias caused by a single metric.

CL-bench project address

• Project official websitehttps://www.clbench.com/
• GitHub repositoryhttps://github.com/Tencent-Hunyuan/CL-bench
• HuggingFace model libraryhttps://huggingface.co/datasets/tencent/CL-bench

Application scenarios of CL-bench

• AI Model Capability AssessmentIt provides research institutions and enterprises with a standardized tool for evaluating context learning capabilities, accurately identifying the shortcomings of models in real-world applications, and guiding the direction of model optimization.
• New model development and verificationAs a core testing component in the development of large language models, it verifies whether the new version of the model truly possesses the breakthrough of learning from dynamic information, rather than merely improving parameter memorization capabilities.
• Industry Solution SelectionIt helps enterprise users evaluate the context learning performance of different commercial models in specific business scenarios, providing an objective basis for selecting AI solutions that suit their needs.
• Education and training sectorAs a teaching case and experimental platform, it helps AI practitioners understand the essential difference between context learning and parameter learning, and cultivates the ability to design and optimize models for real-world application scenarios.
• Academic research benchmarksTo provide the academic community with a unified research benchmark for context learning, promote the formation of comparable and reproducible research results in related fields, and accelerate the overall progress of context learning theory and technology.