AB
AB
AiBoss

All Categories

project

Jamba - AI21's first large-scale open-source model based on the Mamba architecture

Jamba, launched by AI21 Labs, is the first production-grade large language model based on the Mamba architecture. Currently, most large language models (such as GPT, Gemini, and Llama) are based on the Transformer architecture. Jamba, however...

What is Jamba?

Jamba, launched by AI21 Labs, is the first production-ready large language model based on the Mamba architecture. Most large models currently available (such as GPT, Gemini, and Llama) are based on the Transformer architecture. Jamba combines the Mamba Structured State Space (SSM) model with the traditional Transformer architecture, offering the advantages of high-quality output, high throughput, and low memory consumption. This hybrid model provides a 256K context window, significantly improving throughput and efficiency when processing long texts.

Jamba is released with open weights, under the Apache 2.0 open-source license, encouraging further research and optimization by the community. Currently, Jamba is released as a research model, with no guarantees for necessary commercial use. AI21 Labs plans to release a finely tuned, more secure version in the coming weeks.

Jamba's official website entrance

Jamba's main features

  • SSM-Transformer Hybrid ArchitectureJamba is the first production-grade model to combine Mamba SSM (Structured State Space Model) with the Transformer architecture. This innovative hybrid architecture aims to improve the performance and efficiency of the model.
  • Massive Context WindowJamba provides a 256K context window, which enables the model to handle longer text sequences and is suitable for more complex natural language processing tasks.
  • High throughputCompared to a Mixtral 8x7B model of the same size, Jamba achieves a 3x throughput improvement when handling long contexts, enabling more efficient processing of large amounts of data.
  • Single GPU large-capacity processingJamba is able to handle up to 140K contexts on a single GPU, significantly improving model accessibility and deployment flexibility.
  • Open weight licenseJamba's weights are released under the Apache 2.0 license, giving researchers and developers the freedom to use, modify, and optimize the model, thus promoting technology sharing and innovation.
  • NVIDIA API IntegrationJamba will be provided as an NVIDIA NIM inference microservice in the NVIDIA API catalog, enabling enterprise developers to easily deploy Jamba models using the NVIDIA AI Enterprise software platform.
  • Optimized MoE layerJamba utilizes the MoE (Hybrid Expert) layer in its hybrid architecture to activate only some parameters during inference, thereby improving the model's efficiency and performance.

Jamba's technical architecture

The Jamba architecture employs a block and layer approach, enabling it to successfully integrate both the Mamba SSM and Transformer architectures. Each Jamba block contains either an attention layer or a Mamba layer, followed by a multilayer perceptron (MLP), resulting in an overall ratio of one Transformer layer out of every eight layers.

Furthermore, Jamba leverages MoEs to increase the total number of model parameters while simplifying the number of active parameters used in inference, thereby achieving higher model capacity without a corresponding increase in computational demands. To maximize model quality and throughput on a single 80GB GPU, AI21 Labs optimized the number of MoE layers and experts used, leaving sufficient available memory for common inference workloads.

Jamba's performance comparison

According to AI21 Labs' report, the Jamba model has demonstrated excellent results in various benchmark tests (such as HellaSwag, ArcChallenge, MLLU, etc.) and is comparable to or even surpasses state-of-the-art models in the same size category (such as Llama2 13B, Llama2 70B, Gemma 7B, Mixtral 8×7B) in a wide range of task tests (such as language understanding, scientific reasoning, common sense reasoning, etc.).

Back