Skip to content
BENIBIX Learn

Reward Model

AI💬 LLMsLanguage Model Training

Also known as Preference Model, RM

A model that scores answers the way people would. It guides training.

Think of it like

Think of it like a judge at a cake show. They give each cake a score.

Example

Emma's team trains a reward model on 80,000 picks. It now gives a clear, kind reply 0.9 and a rude one 0.2.

Why it matters

It lets a few thousand human choices guide millions of training steps.

Definition

A reward model is a model that reads an answer and gives it a score for how good people would find it. It is trained on many choices where a person picked the better of two answers. Once trained, it can score far more answers than people ever could. The language model is then trained to write answers it scores high. If the reward model has a blind spot, the language model may learn to exploit it.
The reward model sits in the middle. It turns many human picks into a score for any answer.

Common mistake

Treating its score as truth. It is a guess at what people like, and it can be fooled.

Key takeaways

  1. 1
    Scores answers. Like a human judge.
  2. 2
    Learns from picks. Better of two, many times.
  3. 3
    Can be gamed. Blind spots get exploited.

Learn it properly