RLHF (Reinforcement Learning from Human Feedback)
A post-training method in which people compare model outputs, a reward model learns their preferences, and reinforcement learning tunes the model toward the preferred answers. Labs now pair it with AI feedback and reinforcement learning on automatically checkable tasks.
A pre-trained language model predicts likely text. It does not follow instructions or decline harmful requests on its own. RLHF was the technique that turned such models into usable assistants: OpenAI's InstructGPT work in 2022 used it, and it shaped the first versions of ChatGPT and Claude. Today it is one stage in a larger post-training stack rather than the whole recipe.
The classic RLHF pipeline
- Supervised fine-tuning (SFT): Fine-tune the pre-trained model on curated demonstrations so it learns basic instruction following.
- Reward model training: Human raters compare pairs of model outputs and pick the better one. A separate model (the reward model) learns to predict those preferences.
- RL optimization: The language model generates responses and is updated with a reinforcement learning algorithm such as PPO to raise its reward score, with a KL-divergence penalty that keeps it close to the SFT model.
How labs post-train today
Published system cards and technical reports describe a mix of methods rather than RLHF alone:
- Reinforcement learning with verifiable rewards (RLVR): The reward comes from a check a program can run, such as unit tests passing or a math answer matching the reference, instead of a learned preference model. Ai2 coined the term with its Tülu 3 recipe in 2024, and DeepSeek-R1 (January 2025) made large-scale RL with its GRPO algorithm the center of reasoning training. OpenAI's GPT-6 Astra system card says "OpenAI reasoning models are trained to reason through reinforcement learning" and describes alignment improvements that extend to "grading during reinforcement learning." Google's Gemini 4 Argon model card lists instruction-tuning data, reinforcement learning data and human-preference data, with RL techniques that can use multi-step reasoning, problem-solving and theorem-proving data.
- Constitutional methods and AI feedback (RLAIF): A model judges outputs against written principles, which replaces part of the human labeling. Anthropic says it has trained Claude with Constitutional AI since 2023, and it uses the constitution it published in January 2026 at several stages of training: Claude generates synthetic data from it, including responses in line with its values and rankings of possible responses. The Claude Opus 5.5 system card describes post-training aimed at behavior that aligns with that constitution, and Anthropic's system cards also describe crowd workers who contribute through preference selection. OpenAI's Astra card says it teaches the model its Model Spec "by working through a large and varied set of examples."
- DPO and other offline preference methods: Direct Preference Optimization (2023) learns from preference pairs directly, with no separate reward model or RL loop, which makes it cheaper and more stable to run. Meta's Llama 3 used rejection sampling, SFT and DPO, and Ai2's OLMo 3 runs SFT, then DPO, then RLVR. A June 2026 review of frontier post-training recipes on Nathan Lambert's Interconnects found that DPO has largely dropped out of leading recipes.
- Specialists merged by on-policy distillation: The same review names multi-teacher on-policy distillation as the pattern of 2026: train separate domain experts with RL, then distill them into one model. It points to DeepSeek V4 (April 2026), which merged more than ten domain experts this way. Moonshot AI says Kimi K3.1 Pro got substantially more post-training compute than Kimi K3, including joint RL across hundreds of domains.
Failure modes
A model optimized against a reward signal learns to exploit that signal's blind spots. Human-preference rewards can push toward flattery: OpenAI rolled back an April 2025 GPT-4o update in ChatGPT after it turned sycophantic, and its post-mortem said a new reward signal based on thumbs-up and thumbs-down feedback was part of the cause. Verifiable rewards invite reward hacking, such as special-casing tests so they pass without solving the task. Anthropic's system cards report reward-hacking evaluations, and OpenAI's Astra card says its training "does penalize cheating and unauthorized actions."
Related terms
Models relevant to RLHF (Reinforcement Learning from Human Feedback)
Claude Opus 5.5
Anthropic's current Opus and its recommended starting model, priced at $4/$20 per 1M tokens
View model →GPT-6 Astra
OpenAI's most capable model for the most demanding work
View model →Gemini 4 Argon
Google's frontier model for real-world coding, enterprise knowledge work, and cybersecurity defense
View model →DeepSeek V4
Open-weight 1.6T-param MoE frontier model with a 1M-token context built for agents.
View model →Kimi K3.1 Pro
Moonshot AI's open-weight 1M-context flagship for long-horizon coding
View model →