OLMo - A fully open-source framework for large language models developed by the Allen Institute for AI.

What is OLMo?

OLMo (Open Language Model) is a fully open-source large-scale language model (LLM) framework developed by Allen AI (AI2, Allen Institute for Artificial Intelligence). It was designed to promote collaborative research on language models among academia and researchers through open research. The OLMo framework provides a range of resources, including data, training code, model weights, and evaluation tools, enabling researchers to gain a deeper understanding and improve language models.

OLMo's official website entrance

• Official project homepage:https://allenai.org/olmo
• GitHub repository:https://github.com/allenai/olmo
• Hugging Face address:https://huggingface.co/allenai/OLMo-7B
• Research Papers:https://allenai.org/olmo/olmo-paper.pdf

Key features of OLMo

• Large-scale pre-training dataBased on the AI2 Dolma dataset, a large-scale open corpus containing 3 trillion tags, it provides rich language learning materials for the model.
• Diverse model variantsThe OLMo framework includes four different model variants, each trained with at least 2 trillion tokens, providing researchers with a variety of options to suit different research needs.
• Detailed training and assessment resourcesIn addition to model weights, OLMo provides complete training logs, training metrics, and over 500 checkpoints, which can help researchers better understand the model's training process and performance.
• Openness and transparencyAll of OLMo's code, weights, and intermediate checkpoints are released under the Apache 2.0 license, meaning that researchers are free to use, modify, and distribute these resources to promote knowledge sharing and innovation.

OLMo's model performance

According to the OLMo paper, the OLMo-7B model's performance in zero-shot evaluation was compared with several other models, including Falcon-7B, LLaMA-7B, MPT-7B, Pythia-6.9B, RPJ-INCITE-7B, and LLaMA-7B.

The following are the comparison results of OLMo-7B on some core tasks:

1. Downstream task assessmentIn zero-sample evaluations across nine core tasks, OLMo-7B performed best on two tasks (scientific questions and causal reasoning) and remained in the top three on eight tasks. This demonstrates OLMo-7B's strong competitiveness in these tasks.
2. Perplexity-based assessmentWithin the Paloma evaluation framework, OLMo-7B also demonstrated competitive performance in perplexity (bits per byte) across multiple data sources. Particularly on code-related data sources (such as Dolma 100 Programming Languages), OLMo-7B significantly outperformed other models.
3. Additional task assessmentOn the additional 6 tasks (headqa en, logiqa, mrpcw, qnli, wic, wnli), OLMo-7B also outperformed or nearly matched other models in zero-shot evaluation.