Llama 4 - Meta, an open-source multimodal AI model series, reclaims the open-source throne.

What is Llama 4?

Llama 4 is a multimodal AI model series open-sourced by Meta. It is the first to adopt a hybrid expert (MoE) architecture, resulting in higher computational efficiency during training and inference. Llama 4 currently has two versions: Scout and Maverick. Scout has 17 billion active parameters, 16 "expert" models, and 109 billion total parameters, supports 10 million contexts, can process over 20 hours of video, and can run on a single H100 GPU, outperforming models like Gemma 3. Maverick has 17 billion active parameters, 128 "expert" models, and 400 billion total parameters, excelling in accurate image understanding and creative writing, making it suitable for general assistants and chat applications, and ranking second on the LMSYS leaderboard for large models. Llama 4 Behemoth is a preview version still under training, boasting 2 trillion parameters and excellent performance in STEM benchmark tests. Llama 4 supports open-source fine-tuning through pre-training on 200 languages with over 30 trillion tokens of training data.

Main features of Llama 4

• Powerful language understanding and generation capabilitiesTrained on a large amount of text data, it boasts accurate language understanding. It can generate coherent and logical text suitable for creative writing, article writing, and interactive dialogue. For example, in story creation, it can generate rich and vivid story content based on given themes and plot clues; in dialogue scenarios, it can understand user intentions and provide appropriate responses.
• Multimodal processing capabilityTrained on image data, it possesses image understanding capabilities, capable of recognizing objects, scenes, colors, and other elements in images, and describing and analyzing image content. The Scout version supports 10 million tokens in the context window and can process up to millions of words of text, making it suitable for tasks such as summarizing long documents and reasoning based on large codebases.
• Highly efficient reasoning and computational capabilitiesThe hybrid expert (MoE) architecture divides the model into multiple task-specific "expert" sub-models, which are more efficient in training and answering user queries, reducing model service costs and latency, and improving inference efficiency.
• Multilingual processing capabilitiesBy pre-training on 200 languages, it can process and generate text in multiple languages, supporting cross-language communication and processing tasks such as language translation, analysis and generation of text in different languages, and helping users overcome language barriers.

Technical principles of Llama 4

• Hybrid Expert (MoE) ArchitectureLlama 4 is the first model in the Llama series to adopt the MoE architecture. In the MoE model, a single token activates only a small subset of all parameters. For example, the Llama 4 Maverick model has 17 billion active parameters out of its 400 billion total parameters. To improve inference efficiency, Meta alternates between dense and MoE layers. The MoE layer uses 128 routing experts and one shared expert. Each token is sent to the shared expert and also to one of the 128 routing experts. This ensures that only a subset of parameters are activated when the model is running, improving inference efficiency and reducing the cost and latency of model services.
• Native Multimodal DesignLlama 4 is a native multimodal model that employs early fusion technology to seamlessly integrate text and visual tokens into a unified model framework. It can be pre-trained using massive amounts of unlabeled text, image, and video data. Meta has upgraded Llama 4's visual encoder, which is based on MetaCLIP and trained separately from a frozen Llama model, allowing for better encoder tuning and better adaptation to Large Language Models (LLMs).
• Model hyperparameter optimizationMeta developed the MetaP training method, which allows for more reliable setting of key model hyperparameters, such as the learning rate and initialization size per layer. These hyperparameters adapt well to different batch sizes, model widths, depths, and training token counts.
• Efficient model trainingEmploying FP8 precision ensures high FLOPs utilization of the model without sacrificing quality. For example, when pre-training the Llama 4 Behemoth model using FP8 precision and 32K GPUs, it achieved a performance of 390 TFLOPs per GPU. The training data contains over 30 trillion tokens, covering text, image, and video datasets. The model is further trained through "mid-training," extending long contexts with specialized datasets to enhance core capabilities, unlocking a leading 10 million input context length for Llama 4 Scout.
• Post-training process optimizationThe training process is as follows: Lightweight Supervised Fine-Tuning (SFT) > Online Reinforcement Learning (RL) > Lightweight Direct Preference Optimization (DPO). To address the issue that SFT and DPO might overly restrict the model, Meta uses an Llama model as the evaluator, removing more than 50% of the data labeled "easy," performing Lightweight SFT on the remaining more difficult data, carefully selecting more difficult prompts during the multimodal online RL phase, and implementing a continuous online RL strategy that alternates between model training and data filtering, retaining medium to high difficulty prompts, and finally performing Lightweight DPO to balance the model's intelligence and conversational capabilities.

Llama 4 project address

• Project official website:https://ai.meta.com/blog/llama-4
• HuggingFace model library:https://huggingface.co/collections/meta-llama/llama-4

Application scenarios of Llama 4

• Dialogue systemLlama 4 can be used to build intelligent chatbots, such as Maverick, which is suitable for general AI assistants and chat applications. It can understand user questions and generate natural and fluent answers, provide information, answer questions, and chat.
• Text generationIt can perform creative writing, such as creating stories, poems, and scripts, and can be used to write news reports, product descriptions, copywriting, etc., generating high-quality text based on given themes and requirements.
• Code generation and assistanceIt can help developers generate code, producing corresponding code snippets or complete programs based on functional descriptions. It can also provide auxiliary functions such as code explanation, comment generation, and code style checking to improve development efficiency.
• Code understanding and analysisScout can perform inference based on large codebases, helping developers understand the logic and functionality of complex codebases, and perform tasks such as code review and vulnerability detection.
• Image understanding and descriptionMaverick excels in accurate image understanding, capable of recognizing objects, scenes, colors, and other elements in images, and describing and analyzing image content. For example, it can describe the content of a photo and determine the location where it was taken.
• Information retrieval and recommendationCombining its language understanding and generation capabilities, it can be used in information retrieval systems to understand user query intent, provide more accurate search results, and make personalized recommendations based on user interests and behaviors.