RLHF vs RLVR
People think RLVR is just a new name for RLHF. It is not. In RLHF, people rank answers and a reward model learns their taste, so it can reward any kind of reply. In RLVR, a program checks the answer against a key or runs tests, so it only works where a right answer exists. RLHF shapes tone and help. RLVR trains math and code.
Reinforcement Learning from Human Feedback (RLHF)
A model learns which answers people like best. Then it gives more of those.
Reinforcement learning from human feedback trains a model to give the kind of answers people prefer. First, people compare pairs of answers and pick the better one. A reward model learns from those picks to score any answer. Then the language model is trained to write answers that score high. It makes replies more helpful and polite, and less likely to cause harm.
RL with Verifiable Rewards (RLVR)
Points for a model each time a check shows it got the right answer.
Reinforcement learning with verifiable rewards, or RLVR, trains a model on tasks a program can check. The model tries a problem, such as a sum or a coding task. A checker compares the final answer with the key, or runs the code's tests. It gives a reward only for a pass, so no person or reward model has to judge. The name comes from a late 2024 paper. The method is used to train reasoning models on math and code.
In one line
- Reinforcement Learning from Human Feedback (RLHF)
- A model learns which answers people like best. Then it gives more of those.
- RL with Verifiable Rewards (RLVR)
- Points for a model each time a check shows it got the right answer.