In late 2024 and early 2025, a new kind of language model arrived that stops and thinks before it answers. It writes pages of private reasoning, checks itself, and only then replies. Many people assumed this skill had to be taught with huge numbers of step-by-step examples written by people. DeepSeek's team asked a bolder question: what if nobody teaches it at all?
Their answer is the paper behind DeepSeek-R1. It was first released as a preprint in January 2025, and a longer version was published in Nature in September 2025. The recipe is surprisingly small: give a model hard questions, reward it only when the final answer is right, and let it practice.
DeepSeek-R1-Zero learned to reason through reinforcement learning alone, with no human-written reasoning examples. Its score on hard competition math rose from 15.6 percent to 77.9 percent. Along the way it learned to think for longer, check its own work and change its mind. A multi-stage recipe then fixed its messy writing and produced DeepSeek-R1.
The old way: teach reasoning by example
Chain-of-thought prompting showed that a model does better when it writes out its steps. The natural next move was to train on many examples of good step-by-step solutions, written or checked by people. That works, but the paper names two limits.
- It is slow and expensive. Every example needs a person.
- It caps the model at human habits. A model that copies human reasoning cannot easily find a way of thinking that people never used.
The authors wanted to see what happens if the model is free to explore. Their bet was that if the only signal is whether the final answer is correct, useful ways of thinking will be discovered along the way.
The new way: reward only the answer
The starting point is a large pretrained model, called DeepSeek-V3-Base. The team skipped the usual step of fine-tuning on human examples. Instead they ran reinforcement learning straight away. For each question the model writes a long answer. A short rule then decides the reward.
- Accuracy reward. Is the final answer correct? For math, the answer is checked against the known result. For code, the program is run against test cases.
- Format reward. Did the model put its thinking inside
<think>tags and its answer inside<answer>tags?
The two rewards are added with the same weight. The authors deliberately did not use a neural reward model to score the thinking. They report that such models can be gamed during large-scale training, and that retraining them costs a lot.
Think of a student with an answer key but no teacher. They never see how to solve a problem. They only learn whether the number at the end was right. With enough practice they invent their own methods, including double-checking, because careful students get more right.
GRPO: grading on a curve
The training algorithm is called Group Relative Policy Optimization, or GRPO. For every question the model writes a group of answers. In this paper the group has 16. Each answer gets a reward from the rule above.
GRPO then asks, for each answer, how it compares with its own group. The answer's advantage is its reward minus the group average, divided by the group's spread. A better-than-average answer gets a positive advantage and is made more likely. A worse-than-average answer gets a negative one and is made less likely.
Try it: score a group of answers
Eight answers to one question are below. Tick which ones are correct and which used the tags. Watch the advantages move. Then press the buttons to see what happens with a very hard question, and with one that is too easy.
Notice what a group buys. There is no separate model judging how good an answer is. The group itself is the yardstick, and the only thing the training needs is a rule that can say right or wrong.
What the model learned on its own
After many thousands of training steps, DeepSeek-R1-Zero improved enormously. The paper reports its average score on AIME 2024, a set of hard competition math problems, climbing from 15.6 percent at the start to 77.9 percent. Majority voting over many answers lifted it to 86.7 percent.
The more surprising result is how it got there. Nobody told the model to think for longer, but its answers grew steadily longer as training went on, reaching thousands of tokens. Reflective words such as wait, verify, check and retry became five to seven times more common than at the start.
The paper highlights one moment. In an intermediate version, the model stopped in the middle of a solution and wrote: "Wait, wait. Wait. That's an aha moment I can flag here." It then restarted its work step by step. The authors also report that the word "wait" was almost absent early in training and spiked after about step 8,000.
The reward only works where a rule can check the answer. Math and code qualify. Writing a good essay or advising a patient does not have such a rule. The authors note that R1-Zero was strong at reasoning problems but weaker in broader areas such as open-ended writing.
From R1-Zero to R1, and to small models
R1-Zero had flaws. Its thinking was hard to read, and it sometimes mixed English and Chinese in the same chain of thought. To fix this the team built DeepSeek-R1 with several stages: a small set of cold-start examples, reinforcement learning, a round of collecting good answers to fine-tune on, and a final round that also aligned the model with human preferences.
The paper reports that R1 scored 79.8 percent on AIME 2024, against 79.2 percent for OpenAI's o1-1217 in the same table. The team also showed that a big model's reasoning can be passed down. They fine-tuned smaller open models on about 800,000 samples written by the larger one. The small models reasoned much better than their original versions.
Key takeaways: Reasoning can emerge from a plain reward for correct final answers, with no human-written steps. GRPO scores each answer against its own group, so no separate judge model is needed. The model learned to think longer and to check itself, including the famous "wait" moment. The method needs answers that can be checked, which is why it shines on math and code.
Go deeper
- Read the original: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning (DeepSeek-AI)
- PPO, RLHF and reasoning models: how reasoning models are trained, step by step
- DPO and test-time compute: why thinking longer at answer time helps
- Reinforcement learning from reward: the basic idea behind learning from a score
