DeepSeek R1-Zero - An open-source inference model from DeepSeek, trained using pure reinforcement learning.

What is DeepSeek R1-Zero?

DeepSeek R1-Zero is an inference model developed by the DeepSeek team that is trained entirely on pure reinforcement learning (RL) without using any supervised fine-tuning (SFT) data. It performs exceptionally well on inference tasks, improving its pass@1 score in the AIME 2024 math competition from 15.6% to 71.0%, approaching the level of OpenAI-o1-0912. The model demonstrates self-evolutionary capabilities during training, such as reflecting on and re-evaluating problem-solving methods.

Main functions of DeepSeek R1-Zero

• Strong reasoning abilityThrough large-scale reinforcement learning, DeepSeek R1-Zero has demonstrated outstanding performance in tasks such as mathematics, coding, and natural language reasoning. In the AIME 2024 mathematics competition, its Pass@1 score improved from an initial 15.6% to 71.0%, approaching the level of OpenAI-o1-0912.
• Pure reinforcement learning drivenThe model is the first inference model trained entirely through reinforcement learning, demonstrating that efficient inference capabilities can be achieved even without supervised fine-tuning of data.
• Self-evolution abilityDuring training, the model exhibits complex behaviors such as reflection and re-evaluation of reasoning steps, which emerge naturally through reinforcement learning.
• Open source and community supportThe model weights of DeepSeek R1-Zero are open source and follow the MIT License, allowing users to train other models using distillation techniques.
• Distillation technologySeveral small models distilled from DeepSeek R1-Zero (such as 7B, 32B, and 70B) have performed well in inference tasks, with performance approaching or even exceeding that of some closed-source models.
• Multilingual support and optimizationAlthough the model performs well in multilingual tasks, it suffers from language mixing issues. Introducing a language consistency reward can effectively improve this problem.
• Efficient Training and ApplicationThe training method of DeepSeek R1-Zero provides new ideas for the development of future inference models, and its open-source strategy also provides strong support for the research community.

The technical principles of DeepSeek R1-Zero

• Pure reinforcement learning trainingDeepSeek R1-Zero starts from a base model (such as DeepSeek-V3-Base) and directly improves inference capabilities through large-scale reinforcement learning, skipping the traditional supervised fine-tuning steps. It allows models to learn complex inference strategies through trial and error even without labeled data.
• GRPO algorithmThe model employs the GRPO (Group Relative Policy Optimization) algorithm, which optimizes the reward signal by normalizing it within the group. GRPO samples a set of outputs (e.g., 16), calculates the mean and standard deviation of the rewards within the group, and generates an advantage function, avoiding the high cost of additional value model training required in traditional PPO (Proximal Policy Optimization).
• Reward Mechanism DesignThe model drives exploration through sparse rewards (e.g., rewarding only when the answer is correct) while supporting long contexts (e.g., 32k tokens), enabling the model to perform multi-step reasoning and verification. This allows the model to autonomously explore effective reasoning paths.
• Self-evolution and emergent behaviorDuring training, the model exhibits the ability to self-evolve, such as complex behaviors like reflection and re-evaluation of reasoning steps. These behaviors are not pre-set but emerge naturally through reinforcement learning.
• Long context supportDeepSeek R1-Zero supports generating extremely long thought chains (CoTs), with an average length of up to 1200 words. This provides ample space for complex reasoning, enabling the model to perform multi-step backtracking and verification.
• Multi-task generalizationThe model also demonstrated strong generalization capabilities in multiple tasks such as code generation and knowledge question answering. This generalization ability is attributed to the flexibility of reinforcement learning and the model's autonomous learning mechanism.

DeepSeek R1-Zero project address

• HuggingFace model library:https://huggingface.co/deepseek-ai/DeepSeek-R1-Zero

Application scenarios of DeepSeek R1-Zero

• EducationThe DeepSeek R1-Zero can be used to create personalized learning plans and as an intelligent tutoring system. It provides targeted exercises and feedback based on students' learning progress and interests, helping them better master knowledge.
• HealthcareIn the medical field, DeepSeek R1-Zero can be used to assist in diagnosis and drug development. It can analyze large amounts of medical data, identify lesion characteristics, and support early screening for diseases such as cancer.
• autonomous drivingThe DeepSeek R1-Zero has potential in the field of autonomous driving, enabling it to make rapid decisions based on traffic conditions and emergencies, optimize driving routes, and improve driving safety.
• Code generation and optimizationIn the programming field, DeepSeek R1-Zero can be used for code generation and optimization. It performs exceptionally well in programming competitions such as Codeforces, generating high-quality code solutions.
• Natural Language ProcessingDeepSeek R1-Zero performs exceptionally well in natural language reasoning tasks, handling complex language logic problems and is suitable for scenarios such as question answering systems and text analysis.