OpenELM - Apple's open-source, high-efficiency language model series
OpenELM is Apple's latest series of high-performance open-source language models, including versions with different parameter scales such as OpenELM-270M, OpenELM-450M, OpenELM-1_1B, and OpenELM-3B. This large model utilizes layers...
What is OpenELM?
OpenELM is Apple's latest series of high-performance, open-source language models, including versions with different parameter scales: OpenELM-270M, OpenELM-450M, OpenELM-1_1B, and OpenELM-3B (both pre-trained and instruction-tuned versions). This large model utilizes an inter-layer scaling strategy to non-uniformly distribute parameters across each layer of the Transformer model, thereby improving accuracy and efficiency. The model is pre-trained on public datasets and demonstrates excellent performance on multiple natural language processing tasks. OpenELM's code, pre-trained model weights, and training and evaluation processes are all open-source, aiming to promote open research and further development within the community.
Basic information about OpenELM
- Parameter sizeOpenELM has a total of eight models, four of which are pre-trained and four are fine-tuned by instructions, covering different parameter scales (270M, 450M, 1.1B and 3B) between 270 million and 3 billion parameters.
- Technical ArchitectureOpenELM employs a Transformer-based architecture and uses a layer-wise scaling strategy to achieve non-uniform parameter distribution by adjusting the number of attention heads and the multiplier of the feedforward network (FFN). This model uses Grouped Query Attention (GQA) instead of Multi-Head Attention (MHA), the SwiGLU activation function instead of the traditional ReLU, and RMSnorm as the normalization layer.
- pre-training dataOpenELM uses multiple public datasets for pre-training, including RefinedWeb, duplicated PILE, subsets of RedPajama, and subsets of Dolma v1.6, totaling approximately 1.8 trillion tokens.
- Open source licenseThe OpenELM code, pre-trained model weights, and training guidelines are all released under an open-source license. In addition, Apple has released code to convert the model into an MLX library for inference and fine-tuning on Apple devices.
OpenELM official website entrance
- arXiv research paper:https://arxiv.org/abs/2404.14619
- GitHub model weights and training configuration:https://github.com/apple/corenet
- Address of the finely tuned Hugging Face model:https://huggingface.co/collections/apple/openelm-instruct-models-6619ad295d7ae9f868b759ca
- The pre-trained Hugging Face model can be found here:https://huggingface.co/collections/apple/openelm-pretrained-models-6619ac6ca12a10bd0d0df89e
OpenELM technical architecture
- Transformer architectureOpenELM employs a decoder-only Transformer model architecture, a widely used architecture in natural language processing, particularly suitable for processing sequential data.
- Layer-wise scalingOpenELM effectively distributes parameters across each layer of the model through inter-layer scaling. This means that earlier layers (closer to the input) use smaller attention and feedforward network dimensions, while layers closer to the output gradually increase these dimensions.
- Grouped Query Attention (GQA)OpenELM uses GQA instead of traditional multi-head attention (MHA). GQA is a variant of the attention mechanism designed to improve the model's ability to handle long-range dependencies.
- RMSNorm normalizationOpenELM uses RMSNorm as its normalization layer, a technique that helps stabilize the training process.
- SwiGLU activation functionIn the Feed Forward Network (FFN), OpenELM uses the SwiGLU activation function, a gated activation function that helps the model capture complex patterns.
- RoPE Location CodeTo encode positional information, OpenELM uses Rotary Positional Embedding (RoPE), an encoding method that can handle the order of elements in a sequence.
- Flash attentionWhen computing scaled dot-product attention, OpenELM uses Flash attention, a fast and memory-efficient method for calculating attention.
OpenELM performance
Researchers compared OpenELM with models such as PyThia, Cerebras-GPT, TinyLlama, OpenLM, MobiLlama, and OLMo. At similar model sizes, OpenELM demonstrated higher accuracy on most tasks across mainstream testing platforms including ARC, BoolQ, HellaSwag, PIQA, SciQ, and WinoGrande. In particular, OpenELM achieved higher accuracy compared to the OLMo model, despite having fewer parameters and less pre-training data.