# Post-Training and RLHF

> How a base model becomes a helper that does what you ask. Example answers, people's picks, a model that scores, and answers a program can check.

**Course:** [Large Language Models (LLMs)](https://learn.benibix.com/learn/ai-llms)  
**Time:** 14 min

**After this lesson you can:**

- Tell a base model from an instruct model
- Explain how RLHF uses people's choices
- Say where each method gets its score

## How a base model becomes an assistant

Post-training is what turns a base model into a helper. A base model only carries on text. After post-training it answers questions, does what you ask and says no to some things. In 2022, OpenAI showed that people liked replies from a small model trained this way more than from GPT-3. The small one had 1.3 billion parameters. GPT-3 had 175 billion.

The two kinds have names. A base model is the raw result of pretraining. An instruct model is a base model after post-training, built to follow instructions. Most chat apps use instruct models. Base models are still released, because they are a good starting point for other work.

The first step is instruction tuning. The model is shown thousands of example requests, each with a good answer written by a person. It learns the shape of a helpful reply: answer the question asked, in the format asked, then stop. This step is short and cheap next to pretraining.

Examples only go so far. People are better at judging answers than at writing perfect ones. RLHF, short for reinforcement learning from human feedback, makes use of that. The model writes two answers to one request, and a person picks the better one. Many thousands of these picks are collected.

Those picks train a reward model: a second model that learns to score any answer the way the people would. Then the main model practises against it. It writes answers, gets scores, and shifts towards whatever scores high. No person is needed at this step, which is what makes it cheap enough to run at scale.

DPO came in 2023 as a simpler route. Its full name is direct preference optimization. It skips the reward model and learns straight from the pairs of better and worse answers. It is easier to run and does about as well on many tasks, so many open models use it.

Some tasks have answers a program can check. A sum is right or wrong, and code passes its tests or fails. RL with verifiable rewards uses that check as the score, with no person and no reward model. It is how many reasoning models learned maths and coding in 2025.

A reward model can be fooled. If it likes long, sure-sounding answers, the main model learns to write long, sure-sounding answers, right or not. Much of the polite, padded style of chat models comes from what scored well at this stage, not from what is true.

## Where each method gets its score

| Method | Where the score comes from | Good for |
| --- | --- | --- |
| RLHF | People pick, and a reward model learns their taste | Tone, helpfulness, safety |
| DPO | Pairs of better and worse answers, used directly | The same jobs, more simply |
| RLVR | A program that checks the answer | Maths and code |

![Three stages in a row. Pretraining on trillions of tokens over weeks gives a base model. Instruction tuning on thousands of examples gives a model that follows requests. Human feedback, where people pick the better answer, gives a helpful chat model.](https://learn.benibix.com/diagrams/llm-training-stages.svg)

### Example: A support team trains its tone

Jack's team starts from a base model that rambles. They write 2,000 sample questions with short, ideal answers, and run instruction tuning.

Then staff pick the better of two replies 10,000 times. DPO learns from those pairs, and the replies get shorter and friendlier.

**Words in this lesson:** [Post-Training](https://learn.benibix.com/term/ai/post-training), [Base Model](https://learn.benibix.com/term/ai/base-model), [Instruct Model](https://learn.benibix.com/term/ai/instruct-model), [Instruction Tuning](https://learn.benibix.com/term/ai/instruction-tuning), [Reinforcement Learning from Human Feedback (RLHF)](https://learn.benibix.com/term/ai/reinforcement-learning-from-human-feedback), [Reward Model](https://learn.benibix.com/term/ai/reward-model), [Direct Preference Optimization (DPO)](https://learn.benibix.com/term/ai/direct-preference-optimization), [RL with Verifiable Rewards (RLVR)](https://learn.benibix.com/term/ai/reinforcement-learning-with-verifiable-rewards)

> **The one thing to remember.** Pretraining teaches what text looks like. Post-training teaches which answers people want, and that is not always the true one.

**Question:** What does a reward model do in RLHF?

**Answer:** Scores answers the way people would. People's picks train the reward model to score answers as they would. The main model then practises against those scores, with no person needed.

---

Source: https://learn.benibix.com/learn/ai-llms/post-training-and-rlhf
Author: Behnia Zarei (a developer in Vienna, Austria, who builds and teaches data, AI and business intelligence at BENIBIX)
Publisher: BENIBIX Learn (https://learn.benibix.com/)
Last updated: 2026-11-18
Licence: CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). Quote freely with attribution.
