Deep Reinforcement Learning from Human Preferences

The title matches the catalog. Credits are Paul F Christiano, Jan Leike, Tom B Brown, Miljan Martic, Shane Legg and Dario Amodei. Institutions are OpenAI, DeepMind and Google Brain; Brown’s work was done at OpenAI. The footer identifies NIPS 2017. No numbered revision is printed.

The paper addresses goals hard to encode as rewards or demonstrate. It learns rewards from comparisons to economize on human feedback. The abstract claims feedback on less than 1% of interactions; the conclusion describes roughly three orders of magnitude reduction in interaction complexity.

The authors build on earlier preference-based RL, emphasizing scaling to deep RL and complex behaviors. They distinguish short clips from whole trajectories and comparisons from TAMER’s ratings.

Observations and actions form trajectory segments. Quantitative evaluation uses a hidden true reward; novel goals are evaluated qualitatively. Segments generally start in different states. Atari reward estimation uses the preceding four observations; whole-history recurrent reward modeling is discussed as a possibility, not an experiment.

Separate deep networks represent policy and reward estimate. Policy rollouts generate clips, humans compare them, and supervised reward fitting supplies updated rewards to RL. The three processes run asynchronously.

The method specifies A2C for Atari and TRPO for simulated robotics. It adjusts the TRPO entropy bonus for changing rewards and normalizes predicted rewards to zero mean and constant standard deviation.

Clips last 1–2 seconds. A preferred clip receives all label probability, equal preference receives a uniform label, and incomparable pairs are excluded from the comparison database.

A Bradley–Terry preference model exponentiates each clip’s summed predicted reward and normalizes across the two clips. Cross-entropy fits the label distribution. Clip reward sums are undiscounted.

Predictors use bootstrap samples of the comparison database; independently normalized predictions are averaged. A fraction 1/e is held out for validation, with adaptive L2 regularization targeting validation loss 1.1–1.5 times training loss, and dropout in some domains. The likelihood includes a 10% uniform-random-response component.

Recent trajectory pairs are ranked by variance in ensemble preference predictions. The authors call this a crude uncertainty approximation, acknowledge that it can impair performance, and leave expected-value-of-information selection to future work.

Experiments use TensorFlow, MuJoCo and ALE through OpenAI Gym. Contractors receive 1–2 sentence task descriptions and average 3–5 seconds per comparison; these real-human experiments require 30 minutes to 5 hours. Synthetic-oracle labels derive from true rewards; direct true-reward RL is a separate baseline.

Architectures, environment modifications and RL configuration are referred to Appendix A; contractor instructions to Appendix B; Atari exceptions to A.2. None occurs in the supplied nine pages. Page 9 ends mid-acknowledgment; no bibliography or appendix follows in this artifact.

Eight tasks compare true reward, 350/700/1400 synthetic queries and human feedback. Prose reports near-RL performance with 700 labels, whereas the legend says 750 human queries. Nonhuman curves average five runs; human feedback is one run, with five-batch smoothing. Reacher and Cheetah use author feedback; other tasks use contractors. Ant human feedback exceeds synthetic feedback; the authors suggest upright-posture reward shaping.

Seven games compare true-reward RL, synthetic labels and 5500 human labels. Figure 2’s synthetic budgets are 3.3k, 5.6k and 10k, conflicting with 350/700/1400 in p. 6 prose. Nonhuman curves average three runs; human feedback is one run; points average about 150000 frames. Pong curves approach reward +20 by 5×10^7 plotted timesteps. Human Qbert fails to beat level one; human Enduro exceeds the plotted baselines. Enduro prose says A3C although Section 2.2.1 specifies A2C.

Authors report Hopper backflips with 900 queries in under an hour, one-legged Half-Cheetah locomotion with 800 queries in under an hour, and staying alongside Enduro cars with roughly 1300 queries and four million frames. Enduro behavior is confused by background changes. These tasks use author feedback and linked videos. The backflip item cites Figure 4, but the supplied Figure 4 contains Atari ablations.

Variants replace disagreement selection with random queries, remove the ensemble while also using random queries, restrict queries to the beginning, remove L2 while retaining dropout, use length-one segments in robotics, or fit oracle segment reward totals using squared error.

MuJoCo ablations average five runs with 700 synthetic labels each. Single-step segments and absolute-score regression substantially underperform the original method on several tasks. Ant is a counterexample to uniform superiority of online/disagreement variants. The authors associate regression difficulty with varying reward scale.

Six Atari ablation panels average three runs with 5500 synthetic labels, subject to unavailable Appendix A.2 exceptions. Absolute targets help Breakout but perform poorly on Pong; neither targets nor comparisons consistently dominates across games. Atari targets use clipped rewards. No single-frame Atari ablation is run because its reward model uses consecutive frames.

The authors report that offline reward training can produce Pong policies that avoid losing without scoring, leading to very long volleys. They argue that feedback generally needs to remain intertwined with RL.

Longer clips provide more useful supervision per comparison but less per frame. Early experiments found that clips shorter than 1–2 seconds did not significantly reduce human labeling time. Single-frame comparisons were tested only in continuous control.

The method appears as prose, numbered processes and equations. The supplied artifact’s four figures are MuJoCo and Atari results and ablations; it contains no architecture diagram, quantitative table or backflip illustration.