Phi-3 - Microsoft's latest generation of small model kits

What is Phi-3?

Phi-3 is a new generation of advanced small language models from Microsoft Research, including three different sizes: phi-3-mini, phi-3-small, and phi-3-medium. These models, while maintaining a small parameter size, achieve language understanding and reasoning capabilities comparable to large models through carefully designed training datasets and optimized algorithms. In particular, the phi-3-mini model, with only 380 million parameters, outperforms models with larger parameter sets in multiple benchmark tests, and due to its small size, it can even run on smartphones. The development of the Phi-3 series of models demonstrates the latest advancements in data engineering and model optimization, providing new possibilities for the development of future miniaturized and high-performance AI models.

Technical Report:https://arxiv.org/abs/2404.14219

Hugging Face Phi-3 model address:https://huggingface.co/collections/microsoft/phi-3-6626e15e9585a200d2d761e3

Ollama Phi-3 model address:https://ollama.com/library/phi3

Phi-3 model series

• phi-3-miniIt is the smallest language model with 380 million parameters. Despite its small size, it performs comparably to larger models such as Mixtral 8x7B and GPT-3.5 on benchmark tests for multiple language understanding tasks. Its design allows for deployment on mobile devices and it can achieve a processing speed of 12 tokens per second on the A16 chip used in the iPhone 14 Pro and iPhone 15.
• phi-3-smallThe model has 700 million parameters. It uses the TikTok tokenizer to support multiple languages and adds an additional 10% of multilingual data. phi-3-small scored 75.3% in the MMLU test, performing exceptionally well and surpassing Meta's recently released Llama 3 8B Instruct model.
• phi-3-mediumThis medium-sized model, with 1.4 billion parameters, was trained on more data and outperformed GPT-3.5 and Mixtral 8x7b MoE in most tests. Its 78.2% score on the MMLU test demonstrates its powerful language processing capabilities.

Phi-3 training methods

• High-quality datasetsThe Phi-3 model was trained using a massive dataset of 3.3 trillion to 4.8 trillion tokens, which underwent rigorous screening and filtering to ensure the educational value and quality of the data.
• Synthetic data generation: Utilize large language models (LLMs) to generate synthetic data, which is used to teach model-based logical reasoning and various professional skills.
• Phased trainingThe training process is divided into two stages. The first stage mainly uses network data to enable the model to master general knowledge and language understanding. The second stage further filters the network data and combines it with synthetic data for training.
• Data optimizationThe training data is calibrated to be close to "data optimality," which means that web data that can improve the model's inference ability will be given priority in data selection.
• Post-training optimizationAfter pre-training, the model underwent supervised instruction fine-tuning and preference adjustment (DPO), as well as red team testing and automated testing to improve the model's safety, robustness, and adaptability to dialogue formats.
• Security and AlignmentThe development of Phi-3-mini follows Microsoft’s Responsible AI Principles, using safety alignment in the post-training phase, training with helplessness and harmlessness preference datasets, and further identifying areas for improvement through iterative review by an independent red team.
• QuantificationTo enable the model to run on mobile phones, Phi-3-Mini can perform 4-bit quantization, which significantly reduces the model's memory footprint.
• Multilingual supportAlthough Phi-3-Mini is primarily designed for English, Microsoft is also exploring the multilingual capabilities of small language models. The Phi-3-Small model was trained with more multilingual data.

Phi-3 performance

• Benchmark score:phi-3-mini It scored 69% on the MMLU (Massive Multitask Language Understanding) benchmark and 8.38 on MT-bench.phi-3-small and phi-3-medium Scores of 75.3% and 78.2% were achieved on MMLU, respectively, demonstrating the performance improvement brought about by the increase in model size.
  
• Performance comparison with other models:The Phi-3 series boasts performance comparable to models with larger parameter sets, such as the Mixtral 8x7B and GPT-3.5. In particular...phi-3-miniDespite having only 380 million parameters, it outperformed the 8-B Llama 3 in multiple benchmark tests.In comparison with Meta's Llama-3-8B-Instruct model,phi-3-small(7B parameters) scored higher on MMLU (75.3% vs 66%), demonstrating the performance advantage of the Phi-3 series at smaller model sizes.
  
  
• Running on mobile devices:phi-3-mini Specifically designed to run on mobile devices, after 4-bit quantization, it can achieve a speed of 12 tokens per second on the A16 chip used in the iPhone 14 Pro and iPhone 15, which was difficult to achieve in previous large models.
  
• Multilingual ability:phi-3-small By using the TikTok tokenizer and adding 10% more multilingual data, the model's multilingual processing capabilities were enhanced, which is a significant advantage in comparison.

Limitations and shortcomings of Phi-3

• Knowledge storage limitationsDue to limitations in model size, particularly the phi-3-mini, its ability to store large amounts of factual knowledge is limited, which is particularly evident in tests that require extensive background knowledge, such as its low scores in the TriviaQA test.
• Dependence on search functionalityTo alleviate the limitations of knowledge storage, Microsoft proposes enhancing model performance through integration with search engines. This suggests that without external search support, the model may not be able to independently provide the best answer or information.
• Limitations of multilingual abilityAlthough phi-3-small has been optimized by adding multilingual data, the main language capabilities of the Phi-3 series models are still primarily focused on English. This may be a limitation for applications requiring broad multilingual support.
• Performance of specific tasksOn certain tasks or datasets, the Phi-3 model may not yet have reached its optimal performance, which may require further training data optimization and model tuning.
• Security challengesDespite optimizations in terms of security, like most large language models, the Phi-3 series of models may still face challenges such as generating illusions, inappropriate content, bias amplification, or security issues.
• Hardware deployment limitationsWhile the Phi-3-Mini can run on smartphones, even the miniaturized model may face hardware performance limitations for some tasks that require extremely high computing resources.