AB
AB
AiBoss

All Categories

project

Llama 3 - A new generation of large language model released by Meta Open Source

Llama 3 is Meta's latest open-source release of a new generation of large-scale language models (LLMs), including models with 8B and 70B parameters, marking another significant advancement in the field of open-source artificial intelligence. As the third in the Llama series...

What is Llama 3?

Llama 3 is Meta's latest open-source next-generation large language model (LLM), available in 8-byte and 70-byte parameter sizes, marking another significant advancement in the open-source AI field. As the third generation of the Llama series, Llama 3 not only inherits the powerful capabilities of its predecessors but also provides more efficient and reliable AI solutions through a series of innovations and improvements. It aims to support a wide range of applications, including but not limited to programming, problem-solving, translation, and dialogue generation, through advanced natural language processing technology.

Llama 3 series models

Llama 3 is currently available in two models: the 8B (8 billion parameters) and the 70B (70 billion parameters) versions. These two models are designed to meet different levels of application needs, providing users with flexibility and freedom of choice.

  • Llama-3-8BThe 8B parameter model is a relatively small but efficient model with 8 billion parameters. It is designed for applications requiring fast inference and limited computing resources, while maintaining high performance standards.
  • Llama-3-70BThe 70B parameter model is a larger-scale model with 70 billion parameters. It can handle more complex tasks, provides deeper language understanding and generation capabilities, and is suitable for applications with higher performance requirements.

Later, Llama 3 will also release a model with 400B parameters, which is currently under training. Meta also stated that a detailed research paper will be published once the training of Llama 3 is complete.

Llama 3 official website entrance

Improvements in Llama 3

  • Parameter sizeLlama 3 offers models with two parameter scales: 8B and 70B. Compared to Llama 2, the increased number of parameters allows the model to capture and learn more complex language patterns.
  • Training datasetThe training dataset for Llama 3 is 7 times larger than that for Llama 2, containing over 15 trillion tokens, including 4 times the amount of code data, which makes Llama 3 superior in understanding and generating code.
  • Model ArchitectureLlama 3 employs a more efficient tokenizer and Grouped Query Attention (GQA) technology, which improves the model's inference efficiency and ability to handle long texts.
  • Performance improvementThrough improved pre-training and post-training processes, Llama 3 has made progress in reducing false rejection rates, improving response alignment, and increasing model response diversity.
  • SecurityNew trust and security tools such as Llama Guard 2, as well as Code Shield and CyberSec Eval 2, have been introduced to enhance the security and reliability of the model.
  • Multilingual supportLlama 3 incorporates high-quality non-English data in over 30 languages into its pre-training dataset, laying the foundation for future multilingual capabilities.
  • Reasoning and code generationLlama 3 demonstrates significantly improved capabilities in inference, code generation, and instruction following, making it more accurate and efficient in handling complex tasks.

Llama 3 Performance Evaluation

According to Meta's official blog, the finely tuned Llama 3 8B model outperforms models of similar parameter size (Gemma 7B, Mistral 7B) in benchmark tests on datasets such as MMLU, GPQA, HumanEval, GSM-8K, and MATH. The finely tuned Llama 3 70B also outperforms models of similar size, such as Gemini Pro 1.5 and Claude 3 Sonnet, in benchmark tests on MLLU, HumanEval, and GSM-8K.

In addition, Meta developed a new high-quality human evaluation set containing 1800 prompts covering 12 key use cases: seeking advice, brainstorming, classification, closed-ended question answering, coding, creative writing, extraction, character/role creation, open-ended question answering, reasoning, rewriting, and summarizing. Human evaluators ranked Llama 3 based on this evaluation set, comparing it with competing models such as Claude Sonnet, Mistral Medium, and GPT-3.5. The results showed that Llama 3 performed exceptionally well in real-world scenarios, achieving a minimum win rate of 52.9%.

Llama 3 technical architecture

  • Decoder architectureLlama 3 employs a decoder-only architecture, a standard Transformer model architecture primarily used for natural language generation tasks.
  • Word segmenter and vocabularyLlama 3 uses a tokenizer with 128K tokens, which enables the model to encode language more efficiently, thus significantly improving performance.
  • Grouped Query Attention (GQA)To improve inference efficiency, Llama 3 employs GQA (Generative Quality Assurance) in both the 8B and 70B models. This technique reduces computation by grouping queries within the attention mechanism while maintaining model performance.
  • Long sequence processingLlama 3 supports sequences of up to 8,192 tokens and uses masking techniques to ensure that self-attention does not cross document boundaries, which is especially important for processing long texts.
  • pre-trained datasetLlama 3 was pre-trained on over 15TB of tokens. This dataset is not only massive in scale but also of high quality, providing the model with rich linguistic information.
  • Multilingual dataTo support multilingual capabilities, the Llama 3 pre-trained dataset contains more than 5% high-quality non-English data, covering more than 30 languages.
  • Data filtering and quality controlThe Llama 3 development team developed a series of data filtering pipelines, including heuristic filters, NSFW (Not Suitable for the Workplace) filters, semantic deduplication methods, and text classifiers, to ensure high-quality training data.
  • Scalability and ParallelizationLlama 3 employs data parallelism, model parallelism, and pipeline parallelism during training, which enables the model to be trained efficiently on a large number of GPUs.
  • Instruction Fine-TuningLlama 3 further improves the model's performance on specific tasks, such as dialogue and programming, by fine-tuning the pre-trained model through instructions.

How to use Llama 3

Developers

Meta has open-sourced its Llama 3 model on GitHub, Hugging Face, and Replicate. Developers can use tools like Torchtune to customize and fine-tune Llama 3 to suit specific use cases and requirements. Interested developers can check out the official documentation.Getting Started GuideThen proceed to download and deploy.

Regular users

For non-technical users who want to experience Llama 3, the following methods can be used:

  • Experience Meta's newly launched Meta AI chat assistant (Note: Meta.AI is region-locked and only available in certain countries).
  • Visit Replicate's Chat with Llama to experience it.https://llama3.replicate.dev/
  • Use Hugging Chat (https://huggingface.co/chat/You can manually switch the model to Llama 3.
Back