LIMO - A highly efficient inference method developed by Shanghai Jiao Tong University, requiring only 817 training samples.

What is LIMO?

LIMO (Less Is More for Reasoning) is an efficient reasoning method proposed by a research team at Shanghai Jiao Tong University. It activates the complex reasoning capabilities of Large Language Models (LLMs) using a very small number of high-quality training samples. The core assumption is that complex reasoning capabilities can be effectively activated in models with a rich knowledge base from the pre-training stage using a small number of carefully designed training samples. LIMO achieves significant performance improvements on multiple mathematical reasoning benchmarks using only 817 training samples.

LIMO's main functions

• Efficient Reasoning ActivationLIMO, using only 817 carefully curated training samples, achieved significant performance improvements across multiple mathematical reasoning benchmarks. In the AIME benchmark, LIMO achieved an accuracy of 57.1%, and in the MATH benchmark, it achieved 94.8%, representing improvements of 50.6 and 35.6 percentage points respectively compared to previous models.
• Excellent generalization abilityLIMO demonstrated superior out-of-distribution generalization across 10 different benchmarks, achieving an average accuracy of 72.8%, and representing an absolute performance improvement of 40.5% compared to models trained with 100 times more data.
• Verifying the "less is more" hypothesisThe LIMO research proposed the "Less is More Reasoning Hypothesis," which states that in the base model, when domain knowledge has been fully encoded in the pre-training phase, complex reasoning abilities can be demonstrated through a minimal but precisely orchestrated set of cognitive processes.
• Data efficiencyLIMO uses only 1% of the training data required by previous methods, challenging the conventional assumption that complex inference tasks require a large amount of training data.

LIMO technical principles

• Core assumptionsThe core assumption of LIMO is the "Less Is More" hypothesis: in the base model, when domain knowledge is fully encoded in pre-training, it can emerge through examples of minimal but precisely organized cognitive processes. This hypothesis posits that the threshold for eliciting complex reasoning is not inherently limited by the complexity of the target reasoning task, but is determined by two key factors:
  • The completeness of the knowledge base encoded by the model during pre-trainingModern foundational models integrate an unprecedented amount of mathematical content during pre-training. For example, Llama 2 and Llama 3 have 1.8T and 3.7T tokens of training data for mathematical reasoning, respectively.
  • Effectiveness of trained examplesThese examples serve as “cognitive templates” demonstrating how models can effectively utilize existing knowledge bases to solve complex reasoning tasks.
• Problem DefinitionLIMO focuses on reasoning tasks with verifiable answers, with the goal of generating an answer and a chain of reasoning.
• Question SelectionChoose challenging problems that promote complex reasoning chains, diverse thinking processes, and knowledge integration.
• Inference chain constructionIt collects official solutions, supplements them with solutions written by human and AI experts, and generates diverse solutions based on state-of-the-art inference models.
• Training Protocol: Fine-tuning a large language model on the LIMO dataset using supervised fine-tuning, with full parameter fine-tuning, DeepSpeed ZeRO-3 optimization and FlashAttention-2.

LIMO project address

• Github repository:https://github.com/GAIR-NLP/LIMO
• HuggingFace model library:https://huggingface.co/GAIR/LIMO
• arXiv technical paper:https://arxiv.org/pdf/2502.03387

LIMO application scenarios

• EducationLIMO enhances students' logical thinking and complex problem-solving abilities. Through a small number of high-quality reasoning training samples, LIMO helps students better understand and master complex mathematical and logical reasoning processes.
• Scientific researchIn scientific research, LIMO can be used for modeling and analyzing complex problems. In mathematics and physics, LIMO can help researchers quickly validate and optimize complex theoretical models.
• Industrial applicationsLIMO can be used to optimize production processes and quality control. LIMO can help companies quickly identify and resolve complex problems in production, improving production efficiency and product quality.
• medical fieldLIMO can be used to assist in the optimization of diagnosis and treatment plans. Through a small number of high-quality medical cases, LIMO can help doctors quickly identify complex disease patterns and provide more accurate diagnostic recommendations.