Learning to summarize with human feedback
1. Paper overview
In one sentence: Learning a reward from human comparisons improves summary generation, but optimizing that reward too aggressively can reverse the improvement. e-probleme-pipelinee-ppoe-tldre-transfere-overoptimizatione-visual-scope
| At a glance | What to know |
|---|---|
| Research problem | Source description Likelihood training rewards imitation of every reference, including poor demonstrations, while ROUGE measures overlap rather than the full quality judgment. The authors target short English summaries that faithfully convey a Reddit post to someone who cannot read it. Human comparisons supply a task-specific objective, with researcher judgments defining the intended quality standard. e-probleme-data |
| Core mechanism | Source description The work combines batch preference collection, reward learning and policy optimization, with sustained labeler training and quality monitoring. It reports 64,832 TL;DR summary comparisons, alongside evaluation data, as a public release. e-pipelinee-feedback |
| A key reported result | TL;DR human preference against reference summaries: 1.3B human-feedback policy: 61% Fraction of summaries preferred to the dataset reference. Filtered TL;DR evaluation; final decoding T = 0. The main text describes a validation holdout but leaves detailed evaluation sampling to appendices. Approximately ten-times-larger supervised policy: 43%; these are separate comparisons against references, not a direct 61% head-to-head win rate. Feedback outperforms a much larger supervised model. Length contributes: the main text separately reports about 65% for the 6.7B feedback policy after length control; the Appendix F procedure is unavailable. e-tldre-datae-models |
| Reading caution | Reader analysis Longer summaries can earn higher ratings by increasing coverage. The reward model also prefers human improvements that shorten summaries only 62.6% of the time, versus 76.4% for humans. Length control and targeted edits therefore matter alongside aggregate quality scores. e-tldre-probes |
Core contributions
- Source description
The work combines batch preference collection, reward learning and policy optimization, with sustained labeler training and quality monitoring. It reports 64,832 TL;DR summary comparisons, alongside evaluation data, as a public release. e-pipelinee-feedback
- Author claim
Human feedback improves TL;DR preferences and transfers to CNN/DM without news-specific fine-tuning. Separate diagnostics expose reward overoptimization and shortcomings of ROUGE as an optimization target. e-tldre-transfere-overoptimizatione-metrics
Figure 2. Human comparisons train the evaluator; its scores then train the generator. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each column, then move right. First, several policies produce candidate summaries of the same Reddit post; a human selects the better pair member. The middle column feeds each post–summary combination to the reward model and compares scores r_j and r_k using the preference label. Its green dotted arrows return to the reward model. The right column instead sends a newly generated summary through the scorer, with a purple dotted arrow updating the policy through PPO. Section 3.4 supplies two components omitted from this overview: a KL penalty against the supervised policy and a separately parameterized value function. Humans label batches rather than each PPO token. e-pipelinee-feedbacke-rewarde-ppo
What it supports. The procedure separates generating text from judging completed text. A human label supervises a relative comparison, while the learned scalar becomes the policy's optimization signal. The reward model can therefore reuse accumulated preferences across many policy samples, subject to the limits of its learned judgments.
Where the evidence stops. The figure and p. 6 both print positive log-sigmoid as 'loss'. Mathematically it should be maximized, or negated for minimization. The crop preserves that sign; the supplied source does not resolve implementation handling. The diagram also simplifies an evolving batch schedule.
2. Motivation
2.1 The problem and the proposed response
Likelihood training rewards imitation of every reference, including poor demonstrations, while ROUGE measures overlap rather than the full quality judgment. The authors target short English summaries that faithfully convey a Reddit post to someone who cannot read it. Human comparisons supply a task-specific objective, with researcher judgments defining the intended quality standard. e-probleme-data
2.2 What this reading follows
A useful summary must preserve what matters, yet neither matching a reference word for word nor maximizing its likelihood fully captures that goal. This paper turns pairwise human judgments into a learned scoring function, then uses PPO to improve a pretrained summarizer under a constraint toward its supervised starting point. The visual sequence follows that training loop, the resulting preference gains, transfer from Reddit posts to news, and the point where reward optimization becomes counterproductive. The central distinction is between predicting which summaries people like and remaining a reliable objective as a policy changes. This edition covers the supplied NeurIPS 2020 main-paper PDF; its cited appendices are unavailable. e-probleme-pipelinee-ppoe-tldre-transfere-overoptimizatione-visual-scope
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational training-optimization category is supported by the preference-learning and PPO procedure. Architecture, prediction paradigm and quadrant are appropriately not applicable to the world/action-model taxonomy: multiple text models score and generate summaries, without joint future-world/action prediction or inverse dynamics. Merely using RL does not make this a world-action model. e-pipelinee-modelse-ppo
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a relative judgment into a reusable score
The supervised summarizer supplies a starting distribution of plausible answers. Human comparisons then ask a narrower question than writing a new ideal summary: which of two candidates better conveys the same post? A scalar reward model receives the post together with each candidate, and the preferred-minus-rejected score difference is the quantity used in its training expression. This is why the two reward-model branches in Figure 2 meet before the preference label affects training. The paper normalizes reference rewards to mean zero after training, providing a reference for the scalar outputs. Reader analysis: a useful ranking score can be learned without treating every original reference as optimal. However, the printed positive log-sigmoid must be interpreted with its optimization direction made explicit; calling it a loss does not by itself specify a correct minimization objective. e-modelse-pipelinee-rewarde-problem
5.2 Use the score to change generation, with a reference constraint
During PPO training, the policy emits BPE tokens and the reward model evaluates the completed summary. The full reward subtracts a β-weighted log ratio between the learned policy and the original supervised policy. This connects the training procedure to the failure mode in Figure 5: moving farther from the supervised distribution can reveal errors in a reward model trained on a narrower collection of summaries. The PPO value function is a third Transformer, initialized from the reward model but parameterized separately from the policy. Its training role should not be confused with that of the fixed reward evaluator during a policy-optimization stage. At evaluation, the resulting generator produces summaries with temperature zero. Candidate reranking in Figure 7 is a separate analysis of scoring objectives, rather than an additional mandatory stage in that main generation path. e-ppoe-overoptimizatione-modelse-metricse-pipeline
5.3 Keep three kinds of success separate
Figure 1 asks whether a generated summary beats its reference in a pairwise judgment. Figure 4 instead measures average quality on a seven-point scale when a Reddit-trained policy summarizes news. Figure 6 evaluates whether a reward model predicts a held-out human comparison correctly. These quantities answer different questions even when all improve with model size. The main text adds another distinction: the 45% perfect-rating result is the fraction of summaries receiving an overall 7/7, not a preference rate. Reader analysis: a convincing reproduction should retain all three measurement levels and control summary length, because a model can improve coverage by writing more while still missing important semantic details. The shortening-edit diagnostic and reward-overoptimization curve show why an aggregate evaluator score cannot replace independent judgments of the outputs it helps select or train. e-tldre-qualitye-transfere-scalinge-probese-overoptimization
5.4 Training and inference
During training
Filtering retains 123,169 Reddit posts, with approximately 5% held out for validation. References contain 24–48 tokens; the stated generation task is fewer than 48 tokens. Filtering also uses a subreddit whitelist. The detailed filter recipe is deferred to absent Appendix A. e-data
Feedback experiments use 1.3B and 6.7B models. The reward model, policy and PPO value function have equal model size but separate roles; the value function has completely separate parameters from the policy and is initialized from the reward model. The original supervised model provides the reference policy for the KL penalty. e-modelse-ppo
Labelers receive detailed instructions, answers to questions and ongoing feedback. On a comparison subset, labeler–researcher agreement is 77% ± 2%, while researcher–researcher agreement is 73% ± 4%. This checks agreement with this research team's standard, not universal preferences. e-feedback
During inference
The trained policy generates summary tokens from a supplied post. Final human evaluations use temperature T = 0. The main policy evaluation does not require a fresh human comparison or reward-model ranking for each generated summary. e-modelse-pipelinee-ppo
CNN/DM transfer evaluates the Reddit-trained policy without further news training. Best-of-N rejection sampling is a separate metric-optimization experiment; it should not be conflated with PPO training or with the main decoding protocol. e-transfere-metrics
5.5 Implementation flow
- Initialize a task-capable policy
Start from GPT-3-style Transformer decoders pretrained for next-token prediction, then fine-tune them on the filtered TL;DR reference summaries. These supervised models initialize the policy and reward model and provide evaluation baselines. e-models
- Collect comparisons in batches
For each post, draw summaries from sources including current and initial policies, references and other baselines. Humans choose the better of two summaries. Retraining uses accumulated labels; the project changed procedures over time rather than following an invariant three-step cycle. e-pipelinee-feedback
- Learn a scalar preference score
Add a randomly initialized linear scalar head to the supervised baseline. Train on the preferred versus rejected summary's score difference, then normalize outputs so dataset reference summaries have mean score zero. The scalar evaluates a complete summary conditioned on its post. e-reward
- Optimize the generator
PPO maximizes learned reward with a penalty relative to the original supervised policy. Each action is a BPE token; the reward model scores only the complete summary, the episode ends at EOS, and the discount factor is one. No environment-state predictor or physical action executor is introduced. e-ppo
6. Experiments & results
A pretrained summarizer first learns from reference summaries, then a separate reward model learns which candidate summaries humans prefer. PPO uses that learned score, constrained toward the supervised policy, to improve generation. A 1.3B feedback policy obtains 61% preference against reference summaries, versus 43% for a roughly ten-times-larger supervised policy. News transfer is promising, but excessive reward optimization eventually reduces human preference. The contribution concerns learning a text-generation objective, rather than modeling physical dynamics or executing robot actions (e-pipeline, e-ppo, e-tldr, e-transfer, e-overoptimization).
The supplied proceedings PDF contains seven figures and one table. Its only table, Table 1, is a selected post and example summaries, so an original numerical results table cannot be supplied. Quantitative evidence is presented here through the original plots. The main PDF includes a model/data-size ablation and optimization diagnostics, but the separate-value-network ablation and detailed numerical tables referenced in the body are in appendices absent from this artifact. Their contents are not reconstructed or treated as visually reviewed. e-visual-scopee-scalinge-overoptimizatione-metrics
6.1 Read the original evidence
Figure 1. The optimization objective changes preferences more than simply enlarging the supervised baseline. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the dashed horizontal line at 0.5: each model is being compared against the dataset's human-written reference, so values above this line mean its summaries are preferred more often. The orange curve is human-feedback training, green is supervised learning, and blue is pretraining only. Compare the orange point at 1.3B with the green point at 12.9B, not just points at equal size. Section 4.1 supplies the corresponding raw scores, 61% and 43%. Error bars represent one standard error. These are generated-summary evaluations with final temperature zero, whereas reference-length filtering and labeler training help define the task being measured. e-tldre-datae-feedbacke-models
What it supports. A 1.3B feedback policy surpasses a supervised model about ten times its size on preference against references. Both feedback policies exceed the reference line. This supports the usefulness of the learned objective under this evaluation, rather than establishing that additional model capacity is unnecessary.
Where the evidence stops. These are separate comparisons against references, not direct model-versus-model win rates. Longer summaries explain part of the improvement. The body reports about 65% for the 6.7B policy after length control, but the detailed control in Appendix F is absent.
Figure 4(a–b). Reddit preference training transfers to news, with summary length affecting the comparison. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) uses average overall quality on a seven-point scale, so its vertical axis differs from the preference fractions in Figure 1. Follow the orange feedback-transfer line and compare it with the green supervised-transfer line. The purple and pink points are models trained specifically on CNN/DM; horizontal lines mark references and lead-3. Panel (b) changes the horizontal axis to summary log-length. Its orange and pink trends become close where their length ranges overlap. The paper presents this as evidence that the Reddit-trained feedback model approaches T5 at similar lengths, despite having no news-specific fine-tuning. e-transfere-datae-tldr
What it supports. The transferred feedback model retains an advantage over supervised transfer when the documents change from Reddit posts to news articles. The right panel helps explain why an unconditioned average can obscure that advantage: the transferred models tend to produce much shorter summaries than models trained on CNN/DM.
Where the evidence stops. The length-conditioned trends are observational comparisons, not an intervention proving that forcing longer output improves quality. Summary-length distributions overlap little, and their exact evaluation instructions are deferred to an absent appendix. Lead-3's strong performance also makes CNN/DM different from TL;DR.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| TL;DR human preference against reference summaries Filtered TL;DR evaluation; final decoding T = 0. The main text describes a validation holdout but leaves detailed evaluation sampling to appendices. | 1.3B human-feedback policy: 61% Fraction of summaries preferred to the dataset reference | Approximately ten-times-larger supervised policy: 43%; these are separate comparisons against references, not a direct 61% head-to-head win rate. Feedback outperforms a much larger supervised model. Length contributes: the main text separately reports about 65% for the 6.7B feedback policy after length control; the Appendix F procedure is unavailable. e-tldre-datae-models |
| TL;DR perfect overall-quality ratings Additional human evaluation across coverage, accuracy, coherence and overall quality | 6.7B PPO policy: 45% Fraction receiving overall score 7/7 | 6.7B supervised baseline: 20%; reference summaries: 23% This measures the top category on a Likert scale, not pairwise preference or factual-error-free rate. e-quality |
| CNN/DM summary generation transfer Reddit-trained policies; no news-specific fine-tuning; four 7-point quality scales | Figure 4 shows feedback transfer above supervised transfer; exact means are plotted rather than tabulated. Average overall human quality rating | The 6.7B transferred policy approaches the CNN/DM-trained 6.7B model; length-conditioned comparisons approach T5. Transferred summaries average about half as many tokens as news-trained outputs, limiting direct comparisons. These are news-summary ratings, not TL;DR win rates. e-transfer |
| Reward-model agreement on CNN/DM Reward models trained on TL;DR evaluated on CNN/DM summary comparisons | 1.3B: 62.4%; 6.7B: 66.5% Agreement with labeler preferences | Inter-labeler agreement: 66.9% The larger reward model nearly matches this agreement baseline. These numbers assess the evaluator, not the summary policy; sample counts and uncertainty are absent from the supplied main-text account. e-probes |
| Reward overoptimization and scaling diagnostics Earlier reward model with varying PPO KL penalties; separate model/data-size ablation | More PPO optimization eventually reduces human preference while predicted preference rises. Both model size and comparison count improve validation accuracy in Figure 6. Human preference versus predicted preference; held-out comparison accuracy | The scaling study spans seven model sizes, 160M–13B parameters, and 8k–64k comparisons. Better held-out prediction does not establish unlimited reliability under optimization. Figure 5's earlier reward model must remain distinct from the final policies. e-overoptimizatione-scaling |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. A reward model can become less trustworthy as the policy learns to optimize it. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis is the resulting KL divergence from the supervised baseline, not the penalty coefficient β and not elapsed training time. The authors obtain differently optimized policies by varying the KL penalty. Follow the dashed line for the reward model's prediction and the solid line for actual human preference against reference summaries. Under light optimization, both move upward. Farther right, the predicted quality continues rising while observed preference falls. Read this diagnostic together with the reward equation on p. 6: the KL term limits policy drift, but the plotted divergence shows that the learned score can become misleading once optimization pushes beyond its reliable region. e-overoptimizatione-ppo
What it supports. The reward is useful over part of the explored range and harmful as an unchecked target at stronger optimization. The divergence between the two curves provides direct evidence that maximizing a learned proxy is a different test from predicting preferences on ordinary held-out examples.
Where the evidence stops. This experiment uses an earlier reward model identified as rm3 in the caption, not the final headline policy's reward model. It demonstrates a failure mode, not a universal KL threshold. Detailed training history and extreme-output samples are in unavailable appendices.
Figure 6. Both reward-model capacity and comparison data improve preference prediction. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally to compare reward-model sizes while holding the number of comparisons fixed. Read vertically at a fixed size to compare the four dataset sizes, whose labels appear at the right ends of the curves. The y-axis is validation accuracy for predicting human preferences; it does not measure the quality of a generated summary. The study spans seven model sizes from 160M to 13B parameters and training subsets from 8k to 64k comparisons. The two black horizontal lines mark a human baseline and a human ensemble. Keep those reference lines separate from the CNN/DM inter-labeler agreement number discussed elsewhere on the same page. e-scalinge-probese-overoptimization
What it supports. Larger models and more comparisons both improve the learned evaluator on this validation distribution. The comparison across dataset sizes makes the role of supervision visible, while movement along each curve shows that data quantity alone does not explain all of the gain from scale.
Where the evidence stops. Validation accuracy is not a guarantee against the exploitation shown in Figure 5. This experiment does not hold training compute or annotation cost constant across all configurations. Exact tabulated scores and full training settings are not provided in the supplied artifact.
Figure 7. Selecting harder for ROUGE improves overlap more reliably than it improves human judgment. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Here optimization means best-of-N rejection sampling: generate candidates, score them and select the best. The horizontal coordinate is log2(n), so moving one unit to the right doubles the candidate count. This changes selection pressure rather than performing an additional PPO update. The vertical axis returns to preference against reference summaries. Preserve the legend's exact identifiers—RM4_6B, RM3, RM4 and ROUGE—because the source distinguishes reward-model iterations. The red ROUGE curve peaks early, while reward-based selectors reach higher preferences. Error bars use the paper's one-standard-error convention. The caption identifies an earlier 1.3B reward model trained with less comparison data. e-metricse-modelse-tldre-visual-scope
What it supports. The choice of scoring function matters when selecting among generated summaries. Increasing the candidate pool under ROUGE does not consistently improve what humans prefer, whereas the learned rewards support stronger selection in the plotted range. This complements the PPO result through a different optimization mechanism.
Where the evidence stops. This is a ranking experiment, not the main temperature-zero policy evaluation. The source defers its detailed sampling protocol and reward-model history to Appendix G.3 and Table 11, which are absent. The plot therefore does not supply a complete executable recipe.
7. Analysis & limitations
7.1 What the evidence leaves open
Longer summaries can earn higher ratings by increasing coverage. The reward model also prefers human improvements that shorten summaries only 62.6% of the time, versus 76.4% for humans. Length control and targeted edits therefore matter alongside aggregate quality scores. e-tldre-probes
The authors report approximately 320 GPU-days for 6.7B RL fine-tuning and thousands of labeling hours. They did not collect an equally costly set of high-quality supervised demonstrations, leaving that alternative explanation for the benefit untested. e-cost
Researcher-aligned labeling is a chosen standard. The authors caution that other tasks require input from affected groups and that Reddit material can yield biased or offensive summaries. This study establishes neither universal preference alignment nor safety on harder-to-evaluate tasks. e-impact
7.2 Questions for discussion
- How much of the gain would remain against supervised learning given an equal budget for high-quality demonstrations?
- Can a reward model improve on held-out comparisons yet become easier to exploit by the policy?
- Which length and factuality controls are necessary before interpreting transferred summary ratings?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful reproduction needs the filtered split, supervised initializations, accumulated comparison batches, reward normalization, separate value network, KL schedule and labeling/evaluation instructions. The supplied body specifies their roles but delegates detailed recipes to absent appendices. GPU models/counts, software versions and a complete hyperparameter configuration cannot be recovered from this artifact. e-datae-feedbacke-modelse-rewarde-ppoe-cost
Proposed checks: sweep PPO optimization strength against independent, length-stratified human judgments; then compare best-of-N reward ranking with ROUGE using identical candidate pools. Track preference, length and factual edits separately. The printed reward-loss sign must be checked before implementation. These are proposed experiments, not completed replications. e-rewarde-overoptimizatione-probese-metrics
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Locate reward exploitation while controlling summary length
Reader-proposed experiment: hold the supervised initialization, reward checkpoint, training posts and PPO update budget fixed, then vary the KL penalty. Evaluate multiple seeds on the same held-out posts using blinded comparisons against references, keeping evaluators independent of reward training. Plot reward predictions and human preference against measured policy KL; also report results within matched length bins and score factual accuracy separately. If predicted reward continues rising while human preference falls even after length matching, the overoptimization mechanism survives that confound. If the divergence disappears, length or evaluation composition may explain the observed pattern. The absent hyperparameter and labeling appendices must be recovered through a separately authorized process before claiming an exact replication. e-ppoe-overoptimizatione-tldre-qualitye-probes
Check 2: Compare reward and ROUGE on identical candidate pools
Reader-proposed experiment: freeze one supervised candidate generator and construct nested candidate pools for each held-out post. Apply ROUGE and each available reward checkpoint to exactly the same candidates at each pool size. Record blinded human preference, output length and accuracy for the chosen summaries, with random selection as a control. Separately rank minimal human edits, splitting length-preserving, shortening and lengthening improvements. The paper's interpretation predicts a higher preference ceiling for learned-reward selection than for ROUGE, while allowing reward failures under stronger selection. A gap that disappears on shared pools or length-controlled edits would weaken attribution to the learned quality objective. Fix and document sampling settings; the supplied Figure 7 does not specify the full recipe. e-metricse-probese-qualitye-tldr
8.3 Reading coverage
Visual audit: Visually inspected the title, nine-author block and OpenAI affiliation on p. 1; Figure 1 and uncertainty convention on p. 2; comparison pipeline and release statement on p. 3; Figure 2 and dataset/task details on p. 4; qualitative Table 1, labeling, model sizes and decoding on p. 5; both equations, separate value function, token/EOS details and TL;DR numbers on p. 6; Figures 3–4 and quality/transfer protocols on p. 7; Figures 5–6, reward validation and edit diagnostics on p. 8; Figure 7 and computational/annotation limitations on p. 9; and broader impacts on p. 10. All six final PNG crops were opened and checked for intact labels, legends and axes. Figure 2's arrows were cross-checked against Section 3.1 and the p. 6 equations; its positive log-sigmoid matches the printed equation, whose sign convention remains unresolved. All five text chunks include the complete main paper and references; references on pp. 11–14 were read as text. No referenced appendix, separate supplement, linked code or experiment execution is included in this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related work
- 3 Method and experiment details
- 3.1 High-level methodology
- 3.2 Datasets and task
- 3.3 Collecting human feedback
- 3.4 Models
- 4 Results
- 4.1 Summarizing Reddit posts from human feedback
- 4.2 Transfer to summarizing news articles
- 4.3 Understanding the reward model
- 4.4 Analyzing automatic metrics for summarization
- 5 Discussion, including limitations, future directions and broader impacts
- Acknowledgements
- References [1]–[73]
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The preceding acquisition limitation was addressed by visually inspecting PDF pages 1–10, all seven figures, Table 1 and the equations; six final original crops were also inspected.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks, covering the complete 14-page proceedings PDF, were read. Appendices A–H referenced by the body are absent from this artifact and were not supplied or read. Detailed filtering, hyperparameters, labeling instructions, training history, length controls, additional ablations and samples therefore remain outside this reading.
- Identity notes: the supplied catalog says 'Learning to summarize with human feedback'; the inspected title page says 'Learning to summarize from human feedback'. The exact catalog proceedings PDF identifier and the substantially matching nine-author list resolve the intended work. Catalog names Jeffrey Wu, Daniel Ziegler and Paul F. Christiano appear on the title page as Jeff Wu, Daniel M. Ziegler and Paul Christiano. Metadata follows the observed title page. No primary revision chain or revision date is supplied; this wording discrepancy is not evidence of a documented title change or an earlier edition. The reviewed artifact identifies itself as NeurIPS 2020.
- Linked code, models, datasets, websites and other papers were not inspected; no experiments were run. References on pages 11–14 were read as text, not visually inspected.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block, affiliation and conference footer
Observed title is Learning to summarize from human feedback; nine authors are Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M. Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei and Paul Christiano; affiliation is OpenAI; footer identifies NeurIPS 2020.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section 1
The paper contrasts reference likelihood and ROUGE with human-assessed quality, motivating preference-based English summarization.
Go to primary source ↓e-pipelinePDF p. 3, Section 3.1 and contribution (4); p. 4, Figure 2 and following paragraph
Comparison collection, reward fitting and PPO form the three-stage procedure; 64,832 comparisons are reported. Data collection evolved rather than following a fixed iteration schedule. Figure 2 prints log-sigmoid of the preferred-minus-rejected score as loss, with no minus sign.
Go to primary source ↓e-dataPDF p. 4, Section 3.2, Datasets and Task
Filtered TL;DR has 123,169 posts with about 5% validation, a subreddit whitelist and 24–48-token references; the generation task specifies fewer than 48 tokens. Appendix A is cited for detailed filters.
Go to primary source ↓e-feedbackPDF p. 5, Section 3.3
The study alternates batch comparisons with cumulative retraining and uses labeler instructions, interaction and monitoring. Agreement is 77% ± 2% for labelers versus researchers and 73% ± 4% among researchers on the reported subset.
Go to primary source ↓e-modelsPDF p. 5, Section 3.4, introductory paragraph, Pretrained models and Supervised baselines
Models are GPT-3-style Transformer decoders; feedback experiments use 1.3B and 6.7B parameters. Supervised TL;DR training initializes policy/reward models. Final human evaluations use T = 0; detailed training and sampling procedures are deferred to Appendix B.
Go to primary source ↓e-rewardPDF p. 6, Section 3.4, Reward models and first displayed equation
A random linear scalar head is added to a supervised baseline; preference learning compares candidate scores. The printed loss is a positive expected log-sigmoid, without a leading minus sign. Reference-summary rewards are normalized to mean zero.
Go to primary source ↓e-ppoPDF p. 6, Section 3.4, Human feedback policies, second displayed equation and footnote 8
PPO uses full-summary reward minus β times the learned/reference policy log ratio. Value-network parameters are separate from the policy and initialized from the reward model; all three models have the same size. Tokens are BPE steps, EOS ends the episode, and γ = 1.
Go to primary source ↓e-tldrPDF p. 2, Figure 1 and footnote 4; p. 6, Section 4.1, first two paragraphs
The 1.3B feedback policy has 61% preference against references versus 43% for the approximately ten-times-larger supervised model. Length accounts for some gains; the body reports about 65% preference for the 6.7B policy after length control. Figure error bars denote one standard error.
Go to primary source ↓e-qualityPDF p. 6, Section 4.1, final paragraph; p. 7, Figure 3 and adjacent results paragraph
Additional 7-point ratings assess coverage, accuracy, coherence and overall quality. Overall 7/7 occurs for 45% of 6.7B PPO summaries, 20% of supervised summaries and 23% of references.
Go to primary source ↓e-transferPDF p. 7, Figure 4(a–b), caption and Section 4.2
TL;DR feedback policies transfer to CNN/DM without further training and outperform supervised transfer. The 6.7B policy approaches a CNN/DM-trained model; at similar lengths it approaches T5. Transfer outputs average about half the tokens of news-trained summaries; four Likert axes are used.
Go to primary source ↓e-overoptimizationPDF p. 7, Section 4.3, first two paragraphs; p. 8, Figure 5, caption and first body paragraph
PPO policies trained with different KL penalty coefficients against an earlier reward model first improve then decline in human preference, while model-predicted preference keeps rising. The horizontal axis is KL from the supervised baseline, not β or training steps.
Go to primary source ↓e-scalingPDF p. 8, Figure 6 and Section 4.3, How does reward modeling scale with increasing model and data size?
Seven reward-model sizes spanning 160M–13B parameters are trained with 8k–64k comparisons. Validation accuracy improves with both model and dataset size. Separate single-human and human-ensemble reference lines appear in the plot.
Go to primary source ↓e-probesPDF p. 8, Section 4.3, What has the reward model learned?
CNN/DM agreement is 62.4%/66.5% for 1.3B/6.7B versus 66.9% inter-labeler agreement. Human-edited and role-reversed summaries are probed. The 6.7B model prefers improving shortening edits 62.6% of the time versus 76.4% for humans; detailed validation sets are deferred to appendices.
Go to primary source ↓e-metricsPDF p. 8, Section 4.4 and footnote 9; p. 9, Figure 7, caption and Evaluation/Optimization text
Automatic-metric analysis includes ROUGE, length, copying and supervised log probability. Best-of-N ranking by ROUGE peaks earlier and at lower human preference than learned reward ranking. Legend labels are RM4_6B, RM3, RM4 and ROUGE; the caption distinguishes an earlier 1.3B model with less data and defers details to Appendix G.3 and Table 11.
Go to primary source ↓e-costPDF p. 9, Section 5, Limitations
6.7B RL fine-tuning required approximately 320 GPU-days. Training labels took thousands of hours and researcher effort; an equivalent-budget high-quality demonstration baseline was not collected.
Go to primary source ↓e-impactPDF p. 10, Section 5, Broader impacts
The authors discuss harmful uses, the choice of whose judgments define good behavior, representation of affected groups, and biased/offensive output risks from Reddit material.
Go to primary source ↓e-visual-scopePDF p. 5, Table 1 and caption; p. 6, value-function paragraph; p. 9, Figure 7 caption
Table 1 contains a selected post and example summaries, not numerical results. The separate-value-network ablation is referred to Appendix G.1, and further metric details to Appendix G.3 and Table 11, which are outside the supplied main-paper artifact.
Go to primary source ↓8.5 Primary sources
Learning to summarize from human feedback ↗
PDF · 8,322 extracted words
Source fingerprint
684358e00bc065438b9e54f6af3dcfa6badbc13aeb50ab727ab5cc4b9850fb84