Reinforcement Learning from Human Feedback

Fri Jul 10 2026

The Core Intuition: Why RLHF?

Imagine you are teaching a language model how to write poetry.

During Supervised Fine-Tuning (SFT), you give the model 1,000 perfect poems. The model learns to mimic the formatting and structure. But what happens when you ask it to write a poem about something it hasn’t seen? It will guess. Sometimes it guesses beautifully, and sometimes it fails terribly. SFT only teaches the model how to answer, but it doesn’t teach it the difference between a masterpiece and garbage.

RLHF (Reinforcement Learning from Human Feedback) fixes this. Instead of giving the model the exact correct answer every time, we let the model guess, and then we grade its guess.

To do this, we need three distinct “characters” in our architecture.

The Cast: The Three Models of RLHF

To perform RLHF, we actually load three different models into our GPU memory at the exact same time. If you understand these three characters, you understand RLHF.

1. The Student (The Policy Model)

This is the model we are actually trying to train. It takes a prompt and generates a response.

  • Status: Training (Weights are updating).

2. The Judge (The Reward Model)

We cannot ask humans to manually grade millions of generated responses. Instead, we train a separate model—the Judge—to score the responses for us. It reads the Student’s response and outputs a single number (e.g., 8.5/10).

  • Status: Frozen (We don’t train the judge during RLHF, it already knows how to grade).

3. The Anchor (The Reference Model)

This is the most misunderstood part of RLHF. The Reference Model is simply an exact, frozen copy of the Student model from before the RLHF training started.

Why do we need a frozen copy? Imagine the Judge gives a massive reward (10/10) every time a response includes the word “Absolutely”. Without an anchor, the Student would immediately realize this trick and change all of its weights to just spam “Absolutely Absolutely Absolutely” to get a high score. It would completely forget how to speak English!

The Anchor (Reference Model) prevents this. During training, we constantly compare the Student’s responses to what the Anchor would have said. If the Student drifts too far away from the Anchor’s behavior, we apply a massive penalty (called the KL Penalty). This forces the Student to improve its answers without forgetting how to act like a normal language model.

  • Status: Frozen (If the Anchor moved, the Student would have nothing stable to hold onto).

The Complete Flow: Step-by-Step

Now that we know the three characters, here is the exact, step-by-step loop of RLHF. We will explain the intuition, then show the code.

Step 1: The Rollout (The Student Takes the Test)

We give the Student (Policy Model) a prompt and tell it to generate a response. In RLHF, generating a response is called a Rollout.

import torch

# Give the student a prompt
prompts = ["Explain quantum computing in one sentence."]

# 1. The Student generates a response
policy_model.eval()
rollouts = []

for prompt in prompts:
    inputs = tokenizer(f"### Instruction:\n{prompt}\n\n### Response:\n", return_tensors="pt").to(device)
    
    with torch.no_grad():
        outputs = policy_model.generate(**inputs, max_new_tokens=100)
    
    response = tokenizer.decode(outputs[0], skip_special_tokens=True).split("### Response:")[-1].strip()
    rollouts.append({"prompt": prompt, "response": response})

Step 2: The Scoring (The Judge Grades It)

Now that the Student has generated an answer, we hand it over to the Judge (Reward Model) to get a score.

rewarded_rollouts = []

for item in rollouts:
    # Combine the prompt and the student's answer
    text = f"### Instruction:\n{item['prompt']}\n\n### Response:\n{item['response']}\n"
    inputs = tokenizer(text, return_tensors="pt", truncation=True, max_length=512).to(reward_device)
    
    # 2. The Judge grades the text
    with torch.no_grad():
        reward = reward_model(**inputs)
    
    score = reward.logits.squeeze().item()
    rewarded_rollouts.append({
        "prompt": item["prompt"],
        "response": item["response"],
        "reward": score
    })

# Output: [{'prompt': '...', 'response': '...', 'reward': 7.4}]

Step 3: Expectation vs Reality (The Value Head)

Before the Student is updated, it must understand why it got the score it got.

In RLHF, the Student model actually has two “heads” attached to its brain:

  1. The Language Head: Predicts the next word (what we just did in Step 1).
  2. The Value Head: Predicts its own score.

Before the Judge even grades the paper, the Value Head makes a guess: “I think this response will get a 5.0.”
If the Judge actually gives it an 8.0, the difference is +3.0. This difference (Actual Score - Expected Score) is called the Advantage.

If the Advantage is positive, the model is pleasantly surprised and learns to increase the probability of that response. If the Advantage is negative, the model is disappointed and learns to avoid that response.

Step 4: The PPO Update (Learning Without Breaking)

Finally, we use an algorithm called PPO (Proximal Policy Optimization) to update the Student’s weights.
PPO takes the Advantage, applies the KL Penalty (checking against the frozen Anchor model to ensure the Student isn’t cheating), and takes a tiny, safe step to update the Student.

from trl import PPOConfig, PPOTrainer

# Configure PPO
ppo_config = PPOConfig(learning_rate=1e-5, batch_size=60, mini_batch_size=2, ppo_epochs=4)

# Load the Trainer with all 3 Characters
ppo_trainer = PPOTrainer(
    config=ppo_config,
    model=policy_model,      # The Student
    ref_model=ref_model,     # The Frozen Anchor
    tokenizer=tokenizer
)

# Format the data for the trainer
queries, responses, scores = [], [], []
for item in rewarded_rollouts:
    q_ids = tokenizer.encode(f"### Instruction:\n{item['prompt']}\n\n### Response:\n", return_tensors="pt")[0]
    r_ids = tokenizer.encode(item["response"], return_tensors="pt", add_special_tokens=False)[0]
    
    queries.append(q_ids)
    responses.append(r_ids)
    scores.append(torch.tensor(item["reward"]))

# 4. The Teacher (PPO) safely updates the Student!
stats = ppo_trainer.step(queries, responses, scores)

How to Read the Metrics

When you run that PPO step repeatedly, you will see a dashboard of metrics. If your intuition is correct, these metrics will make perfect sense:

  • Mean Reward: The average score from the Judge. We want this to increase.
  • Value Loss: How bad the Student’s Value Head was at predicting its own score. As the Student learns what the Judge likes, its guesses get better, so this should decrease.
  • KL Divergence: How far the Student has drifted from the frozen Anchor model. If this hits zero, the Student isn’t learning anything new. If it skyrockets, the Student is cheating and destroying its own language capabilities. We want this to increase slightly, but stay stable.
================================================================================
PPO Update 1/30
================================================================================
Mean Reward : 5.4716
Policy Loss : -0.0706
Value Loss  : 16.4773
KL Divergence : 0.0000

...

================================================================================
PPO Update 30/30
================================================================================
Mean Reward : 8.1201     <-- Reward went up!
Policy Loss : -0.0284
Value Loss  : 0.8746     <-- Value Loss went down!
KL Divergence : 0.0240   <-- Drifted safely!

PPO Training Finished.

Summary

And that is the entire flow of RLHF!
The Student generates an answer, the Judge scores it, the Student compares the score to what it expected, and PPO safely updates the Student while the Anchor prevents it from going crazy.

← Back to writing ← Back to home
machine view