AB
AB
AiBoss

All Categories

project

Arctic - Snowflake Open Source Enterprise-Grade AI Large Model

Arctic is a high-performance, open-source, enterprise-grade large-scale language model developed by the AI research team of cloud computing giant Snowflake. This large model boasts 48 billion (480B) parameters and is constructed by 128 fine-grained experts...

What is Arctic?

Arctic is a high-performance, open-source, enterprise-grade large-scale language model developed by the AI research team at cloud computing giant Snowflake. This large model is a hybrid expert model (MoE) with 48 billion (480B) parameters and 128 fine-grained experts. It offers significant advantages in cost-effectiveness, training efficiency, and inference efficiency, making it particularly suitable for performing enterprise tasks such as SQL generation, programming, and instruction following. Arctic is licensed under the Apache 2.0 license, providing open access to the model weights and code, and also open-sources its dataset and research insights.

Key characteristics of the Arctic

  • Parameter sizeArctic has 48 billion parameters, but only 17 billion parameters are activated during inference to improve efficiency.
  • Technical ArchitectureIt adopts a hybrid architecture, combining a dense transformer and a MoE model with 128 experts, each with 366 million parameters.
  • Context windowThe model uses a 4K attention context window during training and is planned to be expanded to 32K to support the processing of longer sequences.
  • Training costsArctic's training computation budget is approximately less than $2 million (i.e., less than 3,000 GPU weeks) to achieve the capabilities of top-tier large models.
  • Model LicenseArctic is released under the open Apache 2.0 license, allowing free use and modification.
  • Main usesDesigned specifically for enterprises, it excels at performing tasks such as SQL generation, programming, and instruction compliant, making it suitable for creating custom enterprise models.

Arctic's model architecture

  • Architecture CompositionArctic employs a Dense-MoE Hybrid Transformer architecture, which combines the features of a dense transformer and a hybrid expert model (MoE).
  • Dense converter scaleThe model contains a dense transformer model with 10B (10 billion) parameters.
  • MoE architecture scaleIn the MoE part, Arctic designed 128 fine-grained experts, each with 3.66B (3.66 billion, or 3.66 billion) parameters. Therefore, the total number of parameters in the MoE part is 128 × 3.66B, or approximately 470B (470 billion, or 470 billion) parameters.
  • Total number of parametersBy combining the dense transformer and the MoE part, Arctic's total number of parameters reaches 480 billion (480 billion, or 4.8 trillion).
  • Active parameter selectionDuring the inference process, Arctic uses a top-2 gating mechanism to select the two most suitable experts from 128 experts, which means that at any given time, approximately 17B (17 billion) of parameters are active.
  • Communication and computing overlapTo improve training efficiency, Arctic's architecture allows the communication overhead between experts to overlap with computational tasks during training, thus hiding the communication overhead.
  • Reasoning efficiencyIn interactive inference with small batches (e.g., batch size of 1), the Arctic model requires 4 times fewer memory reads than Code-Llama 70B and 2.5 times fewer than Mixtral 8x22B, which helps improve inference performance.
  • System optimizationArctic collaborated with NVIDIA, utilizing the technologies of the TensorRT-LLM and vLLM teams to provide an initial inference implementation for Arctic, and through FP8 quantization technology, enabled Arctic to run on a single GPU node.
  • attention context windowArctic uses a 4K attention context window during training. The team is developing a sliding window implementation based on attention-sinks to support the ability to generate infinite sequences, and plans to expand it to a 32K attention window in the future.

Arctic's performance

Snowflake compared Arctic with models such as DBRX, Llama, and Mixtral on enterprise and academic benchmarks. Arctic performed better on enterprise intelligence metrics, and its performance on general knowledge benchmarks such as MMLU may be slightly lower than some of the latest models, but it still remains competitive on these benchmarks.

Enterprise IndicatorsArctic demonstrates top-tier performance compared to all other open-source models for tasks including:

  • Coding: Code capabilities were tested using HumanEval+ and MBPP+ benchmarks.
  • SQL GenerationUse Spider to benchmark SQL query generation capabilities.
  • Instruction FollowingThe IFEval benchmark tests the ability to follow complex instructions.

How to use Arctic

Snowflake Arctic is now available on platforms such as Hugging Face and Replicate, and will be rolled out to platforms including Snowflake Cortex, Amazon Web Services (AWS), Microsoft Azure, NVIDIA API Catalog, Lamini, Perplexity, and Together in the future.

Back