Instella - AMD's open-source language model with 3 billion parameters

What is Instella?

Instella is a series of open-source language models from AMD with 3 billion parameters. The model was trained entirely from scratch on an AMD Instinct™ MI300X GPU, based on an autoregressive Transformer architecture, containing 36 decoder layers and 32 attention heads, supporting sequences with up to 4096 labeled characters. Instella undergoes multi-stage training, including large-scale pre-training, supervised fine-tuning, and preference optimization, improving natural language understanding, instruction following, and dialogue capabilities. Instella outperforms existing open-source models in multiple benchmarks and is competitive with state-of-the-art open-source weighted models. AMD has fully open-sourced Instella's model weights, training configurations, datasets, and code, fostering collaboration and innovation within the AI community.

Instella's main functions

• Natural Language UnderstandingIt can understand complex natural language text and handle various language tasks, such as question answering, text generation, and semantic analysis.
• Instruction FollowBased on Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO), it accurately understands and executes user instructions to generate answers that conform to human preferences.
• Multi-turn dialogue capabilitySupports multi-turn interaction, enabling coherent dialogue based on context.
• Problem-solving abilityThey excel in tasks such as mathematical problems, logical reasoning, and knowledge-based question answering.
• Multi-domain adaptabilityBased on diverse training data, it adapts to multiple fields, such as academic, programming, mathematics, and everyday conversation.

Instella's technical principles

• Transformer architectureBased on the autoregressive Transformer architecture, it contains 36 decoder layers, each with 32 attention heads, and supports sequence lengths of up to 4096 labels.
• High-efficiency training techniques:FlashAttention-2, Torch Compile and bfloat16 mixed precision training to optimize memory usage and computational efficiency.
• Multi-stage trainingThe system uses 4.065 trillion tags for large-scale pre-training to establish basic language understanding capabilities. Further training is then conducted based on the first stage, using an additional 57.575 billion tags to enhance task-specific capabilities.
• Supervisory fine-tuning (SFT)Fine-tune the data with high-quality command-response pairs to improve command following capabilities.
• Direct Preference Optimization (DPO)The model is optimized based on human preference data to make the output more in line with human values.
• Distributed trainingBased on Fully Sharded Data Parallelism (FSDP) technology, model parameters, gradients, and optimizer states are sharded within nodes and copied between nodes to achieve large-scale cluster training.
• DatasetTraining is based on diverse, high-quality datasets, including academic, programming, mathematical, and conversational data, as well as synthetic datasets, to ensure that the model possesses broad knowledge and capabilities.

Instella's project address

• Project official website:https://rocm.blogs.amd.com/artificial-intelligence/introducing-instella
• GitHub repository:https://github.com/AMD-AIG-AIMA/Instella
• HuggingFace model library:https://huggingface.co/collections/amd/instella

Instella's application scenarios

• Intelligent Customer ServiceAutomatically answer questions, provide personalized services, and enhance customer experience.
• Content creationGenerate copy, stories, etc., to help content creators improve efficiency.
• Educational guidanceAnswering academic questions, providing learning advice, and assisting students in their studies.
• Programming aidsGenerates code snippets, provides programming suggestions, and helps developers solve problems.
• Enterprise knowledge managementIntegrate company knowledge, provide internal consulting, and improve collaboration efficiency.