Reinforcement Learning from Human Feedback (RLHF) fine-tunes a model to produce outputs people prefer, using human judgments as the reward signal. Instead of writing a reward function for "helpful, honest, harmless," you collect comparisons ("response A is better than B"), train a reward model to predict them, and optimize the language model against that reward with PPO or a related method. It is the step that turns a pretrained next-token predictor into an instruction-following assistant.
This page covers the classic three-stage pipeline, the math behind reward modeling and the KL-regularized objective, Direct Preference Optimization (DPO) as the popular RL-free alternative, and the failure modes practitioners watch for.
Stage 1 — Supervised fine-tuning (SFT). Starting from a pretrained decoder-only model, fine-tune on curated (prompt, ideal response) demonstrations. This teaches format and basic instruction following and produces the starting policy.
Stage 2 — Reward model (RM). Sample several responses per prompt from the SFT model, have annotators rank or pick between them, and train a model to output a scalar score consistent with those preferences.
Stage 3 — RL optimization. Treat generation as an episode: the prompt is the state, each token is an action, and the RM score arrives at the end. Optimize the policy with PPO while penalizing divergence from the SFT model.
Comparisons are used instead of absolute scores because people are far more consistent at saying which of two answers is better than at assigning a 1–10 rating.
The reward model is typically the SFT model (or a smaller sibling) with the LM head replaced by a scalar head. Under the Bradley-Terry preference model, the probability a human prefers yw over yl is σ(r(x,yw) − r(x,yl)), so the training loss is
LRM = −E[ log σ( rφ(x,yw) − rφ(x,yl) ) ]
Only differences are learned, so the RM's absolute scale is arbitrary — normalize rewards before RL. RM accuracy on held-out comparisons is the main health metric; human inter-annotator agreement sets a practical ceiling on it. Rewards can also come from AI feedback (RLAIF, e.g. constitution-guided model judgments) or from verifiable checks such as unit tests and math answers.
The policy maximizes
maxθ Ex, y~πθ[ rφ(x,y) ] − β · KL( πθ(·|x) ‖ πref(·|x) )
In practice the KL term is applied per token as −β log(πθ(yt|…)/πref(yt|…)) and added to the reward. It does two jobs: it keeps the policy in the region where the reward model was trained (and so is trustworthy), and it preserves the fluency and knowledge of the pretrained model. β is the central knob: too small and the policy exploits the RM; too large and nothing changes. Some implementations adapt β to hit a target KL.
PPO-based RLHF is expensive and finicky: it keeps a policy, a reference model, a reward model and a value model in memory, and it needs online generation every step. Direct Preference Optimization (Rafailov et al., 2023) shows that the optimal policy for the KL-regularized objective has a closed form, which lets the reward be rewritten in terms of the policy itself. The result is a simple classification loss on preference pairs:
LDPO = −log σ( β[ log πθ(yw|x)/πref(yw|x) − log πθ(yl|x)/πref(yl|x) ] )
DPO needs no reward model and no sampling during training. Trade-offs: it is offline, so it cannot improve beyond the coverage of its preference data, and it can over-optimize on pairs where both responses are poor. Other variants include IPO, KTO (works with unpaired thumbs-up/down data), ORPO (folds preference into SFT), and GRPO, a PPO variant that drops the value network and uses the mean reward of a group of samples for the same prompt as the baseline — popular for reasoning models trained on verifiable rewards.
TRL (Transformer Reinforcement Learning) implements SFT, reward modeling, PPO, DPO and GRPO trainers on top of Hugging Face Transformers. A minimal DPO run on a small model (TRL's API evolves; check argument names against your installed version):
# pip install trl transformers datasets accelerate
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import DPOConfig, DPOTrainer
model_id = "Qwen/Qwen2.5-0.5B-Instruct" # any small instruct (SFT) model
model = AutoModelForCausalLM.from_pretrained(model_id)
tok = AutoTokenizer.from_pretrained(model_id)
# Each row has "prompt", "chosen", "rejected"
train = load_dataset("trl-lib/ultrafeedback_binarized", split="train[:2000]")
args = DPOConfig(output_dir="dpo-out", beta=0.1, learning_rate=5e-7,
per_device_train_batch_size=2, gradient_accumulation_steps=8,
num_train_epochs=1, logging_steps=10)
trainer = DPOTrainer(model=model, args=args, train_dataset=train, processing_class=tok)
trainer.train() # ref model defaults to a frozen copy of `model`
Watch rewards/margins and rewards/accuracies in the logs (implicit reward of chosen minus rejected), and evaluate generations on held-out prompts — preference accuracy rising does not by itself mean the model got better.