Phi-4 - Microsoft's open-source 14-parameter small language model, adept at complex reasoning in fields such as mathematics.

What is Phi-4?

Phi-4 is a small 14B language model from Microsoft, excelling in complex reasoning in fields such as mathematics and traditional language processing. Phi-4 prioritizes data quality in its training, incorporating a large amount of synthetic data to improve its performance in STEM question answering and math competition problems. Phi-4 introduces a new training paradigm, midtraining, enhancing its ability to handle long texts, with a window size of up to 16K. Phi-4 performs exceptionally well on programming tasks, achieving an 82.6% accuracy on the HumanEval benchmark, outperforming other open-source models (such as the 70B Llama 3.3 and the 72B Qwen 2.5). Phi-4 is available for download and open-source use on Hugging Face.

Main functions of Phi-4

• Question answering abilityThey can understand and answer a wide range of questions, and excel in STEM (science, technology, engineering, and mathematics) fields.
• Mathematical Problem SolvingThe model scored over 90 in the American Mathematics Competition (AMC) 10/12, demonstrating strong mathematical reasoning ability.
• Programming taskPhi-4 excels in programming tasks, capable of understanding and generating code to solve programming problems.
• Long text processingBased on the midtraining stage, Phi-4 can handle up to 16K contexts while maintaining a high recall rate.
• Complex ReasoningThe model demonstrates its ability to handle complex inference tasks, such as MMLU and GPQA, across multiple benchmark tests.
• Safe InteractionPhi-4 performs safety alignment during post-training to ensure that interactions with users comply with responsible AI principles.

Phi-4 Technical Principles

• Synthetic data trainingPhi-4 utilizes a large amount of synthetic data during training, generated using techniques such as multi-agent prompting, self-correction, and instruction inversion, to improve the model's reasoning and problem-solving capabilities.
• midtraining stageThe midtraining stage, added between pre-training and post-training, enhances the model's ability to process long texts.
• Long context data selection: Filter samples longer than 8K tokens from high-quality non-synthetic text, weight samples longer than 16K tokens, and match the target length.
• Comparative learningThe Pivot Token Search (PTS) method is used to identify the key tokens that have the greatest impact on the model output, generating high signal-to-noise ratio contrastive learning data.
• Human feedbackBy combining human feedback contrastive learning (Human Feedback DPO), high-quality positive and negative sample pairs are constructed, making the model output more in line with human preferences.

Phi-4 project address

• Project official website:introducing Phi-4
• HuggingFace model library:https://hf-mirror.com/microsoft/phi-4
• arXiv technical paper:https://arxiv.org/pdf/2412.08905

Application scenarios of Phi-4

• Educational SupportAs an educational aid, it helps students solve complex problems in STEM fields and provides tutoring for math and programming assignments.
• Technology ResearchIn scientific research, it helps to understand and generate concepts and data in research papers, assisting researchers in conducting literature reviews and data analysis.
• Software development: Assisting software development, including code generation, debugging, and function implementation.
• Smart AssistantAs an intelligent assistant, it understands and responds to user queries, providing information retrieval, schedule management, and other personal assistant services.
• Enterprise Decision SupportIt analyzes large amounts of data to provide enterprises with market trend analysis, risk assessment, and decision support.