Deep Reinforcement Learning at the Edge of the Statistical Precipice
1. Paper overview
In one sentence: Reliable few-run RL comparisons require matched evaluation protocols, uncertainty intervals and complementary summaries of the run-score distribution. e03e04e06e08e12e14e10e15e17
| At a glance | What to know |
|---|---|
| Research problem | Source description A benchmark ranking based on a few training runs can reflect random conditions more than a reproducible improvement. The authors ask how to report useful comparisons when additional runs are expensive. They distinguish uncertainty about an aggregate score from variation across tasks, and both from changes in the evaluation protocol. e02e03e06 |
| Core mechanism | |
| A key reported result | Procgen probability of improvement: UCB-DrAC versus PLR: approximately 40–50%, the range stated by the authors and consistent with the plotted interval. Average P(X > Y), with 95% bootstrap intervals. Easy mode; 16 tasks and 10 runs per task for all compared methods. Prior work claimed UCB-DrAC improved over PLR. This measures the chance of a higher run score on a randomly selected task, not effect magnitude or the probability that a scientific claim is true. e14e11 |
| Reading caution |
Core contributions
- Author claim
Presents rliable and reports releasing the individual runs needed for subsequent statistical analyses; this is an author-reported release, not verified software availability. e16
Table 1. Three reporting questions require three complementary tools. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from the desideratum on the left to the recommendation on the right. The first row addresses uncertainty in a benchmark aggregate: replace a bare point estimate with an interval. The second asks how performance varies across tasks and runs: preserve that variation in a score-distribution profile. The last distinguishes summary statistics. A mean can be driven by extreme task scores, while a median can disregard changes away from its middle positions. IQM retains the central half of pooled runs. Probability of improvement and optimality gap answer additional questions, so they appear as complements rather than interchangeable replacements. e03e07e11
What it supports. The table organizes the paper's contribution as an evaluation workflow. Intervals quantify estimation uncertainty, profiles reveal distributional tradeoffs, and robust summaries make concise comparisons possible. Each addresses information that the other two can hide; reporting a single improved statistic does not automatically provide a complete evaluation.
Where the evidence stops. This is a recommendations table, not numerical evidence that the tools always work. In particular, Table 1's few-run framing must be read alongside Figure 6's under-coverage at three runs.
2. Motivation
2.1 The problem and the proposed response
A benchmark ranking based on a few training runs can reflect random conditions more than a reproducible improvement. The authors ask how to report useful comparisons when additional runs are expensive. They distinguish uncertainty about an aggregate score from variation across tasks, and both from changes in the evaluation protocol. e02e03e06
2.2 What this reading follows
An RL leaderboard compresses many random training outcomes into a few numbers. This paper asks how much of an apparent improvement survives a different set of runs, and whether the methods were even evaluated in the same way. Its answer is a reporting workflow: retain individual scores, resample within tasks, inspect their distribution and choose robust summaries. The Atari case study makes the problem concrete; subsequent benchmark comparisons show why uncertainty changes the interpretation of progress. Read these visuals as evidence about evaluation reliability, while keeping the paper's absent appendices and several visible editorial inconsistencies in view. e03e04e06e08e12e14e10e15e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Evaluation metrics |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Evaluation metrics category is supported: the contribution transforms observed scores into uncertainty estimates and comparative summaries. Architecture, prediction paradigm and WAM quadrant are not applicable. Evaluating agents such as Dreamer does not make this paper a world/action predictor. e03e08e11e12
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Build the score matrix before summarizing it
Begin with the paper's formal object: normalized scores for M tasks and N independent runs per task. Normalization makes task scores comparable, but it does not remove the randomness of training or evaluation. In Atari 100k, the authors collect 100 runs per algorithm on 26 games, with each score averaging 100 post-training episodes. They then repeatedly take smaller samples from this pool to ask what another few-run study might have reported. The median of per-task averages changes because those averages move and their ordering can change. Figure 3 shows that its expected value also depends on the run count. Before interpreting a stronger median as algorithmic progress, therefore, separate the trained algorithm, the finite-run estimator and the rule used to choose its evaluation scores. e02e04e05e06
Figure 7. Keep individual runs visible before reducing a benchmark to one number. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a human-normalized threshold on the horizontal axis, then read the fraction above it vertically. The left panel counts individual runs pooled across tasks; the right counts tasks after averaging each task's runs. Both use a strict greater-than threshold. Equation (1), the axes and the caption establish this orientation, despite Section 4.2's mistaken reference to the recommended panel as 'right.' The horizontal halfway line locates each distribution's median, which has a different meaning in the two panels. The comparison uses ten runs per method except SimPLe's five, and shaded regions are pointwise 95% percentile stratified-bootstrap bands. e02e09e10
What it supports. The recommended run-score profiles are less coarse and show narrower bands here. They retain run variation that averaging by task obscures. Equation (1) also bounds the effect of one extreme run on the profile at any threshold by 1/(MN), explaining why a huge score need not dominate this display.
Where the evidence stops. Bands are pointwise, not simultaneous guarantees over every threshold. Below Equation (1), the printed population mixture uses 1/N rather than the equation's task weight 1/M; this edition follows the equation and discloses the mismatch.
5.2 Make uncertainty an empirical object too
The stratified bootstrap holds the set of tasks fixed and resamples runs independently inside each task. Each reconstructed score matrix has the same task structure and run count, so recomputing IQM or median yields a distribution of plausible estimates under this resampling approximation. Percentile intervals summarize that distribution, but an interval labeled 95% is not automatically calibrated. Figure 6 checks coverage against a larger reference pool and shows why width alone is insufficient: three-run intervals are too optimistic. IQM helps because it averages the central half of pooled scores, balancing the mean's sensitivity to extremes against the median's instability. The paper recommends reporting effect sizes compatible with the observations, and Figure 2 explicitly directs readers toward intervals for differences when individual method intervals overlap. e08e07e11e05
5.3 Choose what an improvement is supposed to mean
After uncertainty comes the choice of performance question. A profile asks how frequently a normalized threshold is exceeded. IQM emphasizes typical performance across the central half of runs. Optimality gap asks how much shortfall remains below a desirable target, while probability of improvement asks how often one method beats another without rewarding a larger winning margin. Figure 9 shows that these choices can reorder Atari methods; Figure 12 shows that a published Procgen improvement can correspond to only a roughly even chance of winning. Reader interpretation: the metric should be chosen for the intended use before examining which algorithm it favors. Keeping the profile alongside that metric makes crossing distributions and remaining weaknesses visible, while matched evaluation protocols prevent a reporting change from masquerading as a learning improvement. e09e11e12e14e06
5.4 Training and inference
During training
The statistical method learns no parameters. Its Atari 100k case study evaluates DER, OTR, DrQ, CURL and SPR using 100 independent runs per algorithm across 26 games at 100k steps. Each run score averages 100 post-training evaluation episodes. DrQ(ε) uses standard ALE ε-greedy evaluation parameters; their values are deferred beyond the supplied main text. e04
During inference
Application is post-training statistical analysis, with no predicted action, action extraction or feedback controller. Compare intervals and profiles under matched protocols. Overlapping individual intervals alone do not establish equal performance; the paper recommends intervals for score differences and effect-size interpretation. e08e09e05
5.5 Implementation flow
- Preserve the unit of observation
Represent each normalized run score as x_{m,n}. Atari normalization assigns random performance 0 and average-human performance 1. Task averages and their across-task median are random estimates; finite-run medians need not equal the median of true task means. e02e05
- Resample within each task
Independently sample N runs with replacement for each of the M tasks, recompute the aggregate statistic and repeat. This preserves the task strata while approximating the statistic's sampling distribution. The empirical calibration favors percentile intervals among the variants examined. e08e07
- Read a distribution before a ranking
For each threshold, count the fraction of all runs exceeding it. A higher curve means more runs exceed that threshold; crossing curves expose tradeoffs. Bands are pointwise. The profile median mixes tasks and runs, unlike the median of per-task means. e09
- Choose a summary for the question
IQM averages the middle 50% of pooled run scores after trimming 25% from each tail. Optimality gap measures shortfall from a chosen desirable score, set to 1 here. Average probability of improvement measures how often one algorithm beats another on a randomly selected task, without measuring the gain's size. e11
6. Experiments & results
This paper turns multi-task RL evaluation into an uncertainty-aware statistical workflow: keep individual runs, bootstrap within tasks, inspect score distributions and summarize them with robust metrics. Its experiments show how sampling noise and evaluation protocols can change apparent progress. The contribution is evaluation methodology, with no new agent architecture or controller.
The supplied source is a statistical evaluation-methodology paper. It proposes no agent architecture, so Table 1 and the performance-profile figure explain its method. Its sole table is a qualitative recommendations table; quantitative results appear in plots, so original result plots substitute for a numerical results table. Bootstrap calibration and protocol sensitivity provide diagnostics rather than neural-component ablations. Referenced appendix diagnostics and detailed configurations are absent from this PDF and have not been substituted from another edition. e03e06e07e09e12e14e17
6.1 Read the original evidence
Figure 9. A leaderboard changes when the question encoded by its metric changes. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow one algorithm's row across all four panels, reading each horizontal scale independently. Higher is better for median, IQM and mean, while lower is better for optimality gap. Colored bars represent 95% percentile stratified-bootstrap intervals on 55 sticky-action Atari games. Most methods have five runs per game; M-IQN has three and DreamerV2 eleven, so the information supporting their intervals differs. The much larger mean scale is substantive: the caption identifies normalized scores above 50 on James Bond for DreamerV2 and M-IQN. Such extremes affect the mean differently from trimming scores or measuring shortfall from the human target. e12e11e05
What it supports. The metric panels expose both uncertainty and changes in ordering. M-IQN's and Rainbow's median intervals overlap, and DQN (Adam) and C51 are difficult to separate from their intervals. The figure supports qualified performance comparisons that account for which aspect of the score distribution the chosen metric rewards.
Where the evidence stops. Overlapping individual intervals do not prove equivalence; assess differences directly. The Rainbow here is Dopamine Rainbow, whose components and evaluation setting differ from the original Rainbow result identified in the paper's footnote.
Figure 12. An improvement claim can be weak when tested across random tasks and runs. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left panel shows the fraction of scores above a threshold, with each method identified in the legend. Read its normalization from the actual x-axis: Min-Max Normalized Score. Section 5's prose instead invokes this figure while discussing PPO normalization, an inconsistency that should not be silently carried into interpretation. On the right, select a row by matching Algorithm X on the left to Algorithm Y on the right. The horizontal interval concerns the average chance that X scores above Y on a task. The comparison uses Procgen easy mode, sixteen tasks and ten runs per task, with 95% bootstrap intervals. e14e15e11
What it supports. For UCB-DrAC against PLR, the authors report only about a 40–50% chance of improvement, consistent with the displayed interval near or below one half. This directly qualifies the earlier improvement claim. Other rows also reveal how a binary statement about progress can hide substantial variation in comparative outcomes.
Where the evidence stops. Probability of improvement ignores the size of wins and losses. It is not the probability that X has a larger benchmark mean or that a scientific hypothesis is true; those are different inferential questions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Atari 100k median-estimator bias 26 games; repeated subsampling with replacement from 100 runs; Figure 3 uses 100,000 resamples. | SPR's difference between 5-run and 100-run sample medians is about +0.03 normalized-score points. Expected median human-normalized score | The paper compares this with approximately +0.08 over DrQ(ε), calling it about 36% of that improvement. These rounded author-reported figures demonstrate an estimator effect, not improved training from running more seeds. e04e05 |
| Atari 100k evaluation-protocol sensitivity DER evaluated using final scores and the alternative protocols associated with CURL and SUNRISE. | DER under CURL's protocol exceeds CURL's reported score; protocol changes explain much of the apparent gains. Mean and median normalized scores with 95% intervals | Figure 5 separates DER's changed-protocol distributions from published point estimates. This supports a protocol confound; it does not establish a universal ordering under all implementations. e06 |
| Stratified bootstrap calibration on DER 10,000 sets sampled without replacement from 200 runs; the 200-run statistic approximates the reference value. | Percentile intervals give useful coverage around 10 runs; 3-run intervals under-cover. IQM intervals are narrower than median intervals. Coverage and width of nominal 95% intervals | Figure 6 compares basic, bc, bca and percentile methods. Empirical calibration on this benchmark is not a universal small-sample coverage guarantee. e07e08 |
| Atari 200M aggregate comparison 55 games with sticky actions; 5 runs per game except M-IQN with 3 and DreamerV2 with 11. | M-IQN and Rainbow median intervals overlap; DQN (Adam) and C51 also have overlapping intervals. Ordering changes with the metric. Median, IQM, mean and optimality gap with 95% percentile stratified-bootstrap intervals | DreamerV2 and M-IQN exceed 50 human-normalized points on James Bond, illustrating extreme scores affecting means. Uncertainty and metric choice qualify simple leaderboard claims; overlapping intervals do not prove equivalence. e12e05 |
| DeepMind Control Suite ranking stability 6 tasks at 100k and 500k steps; scores divided by 1000. | Many mean intervals overlap and methods do not consistently rank above the baselines they claimed to improve. Mean-score intervals and bootstrap rank distributions | CURL, CURL-D2RL and SUNRISE use studentized intervals; methods with individual runs use stratified bootstrap. The comparison uses mixed interval constructions and concerns simulated control. e13 |
| Procgen probability of improvement Easy mode; 16 tasks and 10 runs per task for all compared methods. | UCB-DrAC versus PLR: approximately 40–50%, the range stated by the authors and consistent with the plotted interval. Average P(X > Y), with 95% bootstrap intervals | Prior work claimed UCB-DrAC improved over PLR. This measures the chance of a higher run score on a randomly selected task, not effect magnitude or the probability that a scientific claim is true. e14e11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. Changing the scoring protocol moves the apparent performance of the same algorithm. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: colored distributions and horizontal intervals describe DER under three evaluation procedures, while dashed vertical lines mark published scores for DER, CURL and SUNRISE. Compare colors within one panel before moving between the median and mean panels, which use different horizontal scales. The blue distribution represents final-performance evaluation. Pink and green represent DER evaluated with the procedures associated with the other papers. Their rightward shifts show how the reporting procedure itself can raise a benchmark summary. Section 3 links these differences to protocols involving maxima and warns that such results are not directly comparable with final performance. e06
What it supports. DER evaluated with CURL's procedure scores above CURL's published estimate in this comparison. The experiment therefore supplies a concrete alternative explanation for apparent algorithmic improvement: changing evaluation can produce a large gain even while retaining DER. The source also attributes much of SUNRISE's apparent gain to the protocol change.
Where the evidence stops. The exact checkpoint and maximum-selection mechanics are referred to absent Appendix A.4. The plot supports protocol sensitivity, but cannot alone establish that every implementation or every claimed improvement has the same confound.
Figure 6. Check interval coverage as well as interval width. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The columns compare median and IQM; the rows compare empirical coverage and average interval width. In the top row, the dashed horizontal line marks the nominal 95% target. A narrow interval is useful only if it also covers the reference often enough. Follow each method's color as the number of runs increases, then inspect the corresponding widths below. The caption describes 10,000 sampled sets drawn without replacement from a 200-run DER pool, with the statistic from that pool serving as an approximation to the true value. This diagnostic tests the uncertainty procedure itself rather than comparing new RL architectures. e07e08
What it supports. IQM achieves smaller intervals than the median in this experiment. Percentile intervals provide favorable coverage among the variants examined, while three-run intervals fall below the nominal target. Section 4.1 reports useful percentile estimates around ten runs, giving an empirical basis for the recommendation and a visible boundary to its smallest-sample use.
Where the evidence stops. The 200-run reference remains an estimate, and coverage is measured on DER's Atari data. These curves do not guarantee 95% coverage on a new benchmark, distribution or extremely small run budget.
7. Analysis & limitations
7.1 What the evidence leaves open
Bootstrap calibration deteriorates at three runs. More resampling cannot supply variability absent from the observed runs; this is a reader interpretation of the bootstrap assumption and calibration experiment. e07e08
No single aggregate captures every performance question. IQM trades some outlier robustness for efficiency; optimality gap needs a meaningful target, and probability of improvement ignores effect size. e11
Editorial inconsistencies remain: Section 4.2 calls Figure 7's run-score panel 'right' although axes and caption place it left; text below Equation (1) prints 1/N instead of its task-average 1/M. Section 5 describes Figure 12 as PPO-normalized although the displayed axis is Min-Max Normalized Score. e10e15
The authors distinguish results reproducibility from methods reproducibility and acknowledge that statistical tools can themselves be abused. Fixed seeds do not answer whether conclusions survive new random conditions. e16e18
7.2 Questions for discussion
- When performance profiles cross, which target or effect size best matches the intended use?
- How would conclusions change if uncertainty over task selection were included alongside run uncertainty?
8. Reproducibility audit
8.1 Requirements and known gaps
A statistical replication needs per-task, per-run scores, normalization references and identical checkpoint/evaluation rules. Reconstructing only published means loses the run distribution needed by these analyses. e02e06e08e09
A training replication additionally needs the implementation, hyperparameters and compute details referred to Appendix A.2. The supplied checklist asserts these were supplied elsewhere; it does not provide their values. Exact CURL/SUNRISE protocol mechanics and per-task probability estimation also require the absent appendix. e04e06e11e17
Proposed checks: recalibrate median versus IQM intervals against a larger independent reference pool, and compare final-checkpoint versus maximum-selected scores while holding trained agents fixed. These test sampling reliability and protocol confounding separately. e05e06e07
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Recheck calibration with an independent reference
Reader-proposed check, not an executed experiment: obtain two independent DER run pools on the same 26 Atari 100k games, using identical normalization and post-training evaluation. Use a larger held-out pool to estimate reference median and IQM; repeatedly draw 3, 5, 10 and 20 runs per game from the other pool and form 95% percentile stratified-bootstrap intervals. Report empirical coverage, interval width and uncertainty in the reference itself, alongside basic-bootstrap controls. The falsifiable expectation motivated by Figure 6 is that IQM retains narrower intervals while three-run coverage falls short. Repeat with another evaluated algorithm to test whether DER-specific calibration generalizes. e04e07e08e11
Check 2: Separate learning gains from score-selection gains
Reader-proposed check, not an executed experiment: hold trained DER and CURL runs, game set and normalization fixed, and record evaluation returns at a prespecified common set of checkpoints with equal episode budgets. Score those same runs once using the final checkpoint and once using the maximum across checkpoints. This is a deliberately specified control, not a reconstruction of the absent Appendix A.4 protocol. Bootstrap within tasks to compare each algorithm's protocol-induced score change and the DER–CURL difference under each matched rule. A large apparent advantage that shrinks or reverses when both methods share a rule supports the confound; a persistent matched-protocol gap indicates remaining algorithmic differences. e04e06e08e17
8.3 Reading coverage
Visual audit: Visually inspected the title/author block and venue on page 1, Figure 1 on page 2, Table 1 and formalism on page 3, Figures 2–3 and experimental setup on pages 4–5, Figures 4–6 and bootstrap method on page 6, Figures 7–8 and Equation (1) on page 7, Figures 9–12 and associated benchmark protocols on pages 8–10, Societal Impacts on page 11, and the appendix/compute references in the page 17 checklist. All six final crops were opened and inspected; margins were corrected where labels were too close to crop boundaries. Reference-only pages 12–16 were read as text. No appendix or separate supplement is present in the supplied PDF. The Figure 7 orientation/normalization mismatches and Figure 12 normalization mismatch are disclosed in the relevant explanations. No plotted point estimates or interval endpoints were digitized.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 17. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Formalism and statistical interpretation (pp. 2–4)
- 3 Case Study: The Atari 100k benchmark (pp. 4–5)
- 4 Recommendations and Tools for Reliable Evaluation, including 4.1–4.3 (pp. 5–8)
- 5 Re-evaluating Evaluation on Deep RL Benchmarks (pp. 8–10)
- 6 Discussion (p. 10)
- Societal Impacts and Acknowledgments (p. 11)
- References (pp. 11–17)
- Checklist (p. 17)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the observed title matches exactly. All five authors agree; the title page credits Marc G. Bellemare, where the catalog author string abbreviates this to Marc Bellemare. The artifact identifies NeurIPS 2021; no numbered revision or revision date is established, and no other edition was compared.
- Every supplied text chunk was read. The complete main paper, references and checklist are present, but referenced Appendix A sections and Figures A.13–A.32 are absent from this 17-page artifact. Appendix status below describes this supplied PDF, not the existence of separate supplements.
- Separate supplemental material availability has not been fully verified. Consequently the cited appendix details on implementation, training configuration, hardware/compute, bootstrap variants and additional diagnostics remain outside this reading.
- The extraction does not reconstruct figure images; this limitation was addressed by visually inspecting the retained PDF on pages 1–11 and 17 and all six final crops. Reference-only pages 12–16 were read as text.
- Linked code, notebooks and individual-run data were not opened or inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block and venue footer
Exact title and five authors are printed, including Marc G. Bellemare; affiliations are Google Research, Brain Team and MILA, Université de Montréal. Footer identifies NeurIPS 2021.
Go to primary source ↓e02PDF pp. 2–3, Section 2 Formalism
Defines M tasks, N independent runs, normalized scores x_{m,n}, random variables, per-task means and across-task mean/median; Atari normalization maps random to 0 and average human to 1.
Go to primary source ↓e03PDF p. 3, Table 1, all rows
Recommends stratified bootstrap intervals, performance profiles, IQM across combined runs, probability of improvement and optimality gap.
Go to primary source ↓e04PDF p. 4, Section 3, experimental setup and footnote 5
Atari 100k uses 26 games and 100k steps; five named algorithms are evaluated with 100 runs and 100 post-training evaluation episodes per score. DrQ(ε) standardizes evaluation parameters; further setup is referred to Appendix A.2.
Go to primary source ↓e05PDF p. 4, Figure 2 and Remark; p. 5, Figure 3 and 'Substantial bias in sample medians'
Figures expose sampling variability and bias. Figure 3 uses 100,000 resamples; SPR's +0.03 change is compared with +0.08 improvement and described as about 36%. Figure 2 advises intervals for differences when individual intervals overlap.
Go to primary source ↓e06PDF p. 5, 'Changes in evaluation protocols invalidates comparisons to prior work'; p. 6, Figure 5 and caption
Maximum-based protocols differ from final performance. DER scored under CURL's protocol exceeds CURL's published score; protocol changes explain much of CURL/SUNRISE's apparent gains. Further mechanics are referred to Appendix A.4.
Go to primary source ↓e07PDF p. 6, Figure 6, all panels and caption
DER interval calibration uses 10,000 sets sampled without replacement from 200 runs and treats the 200-run statistic as a reference approximation. Percentile intervals have favorable coverage; IQM is narrower; three-run intervals under-cover.
Go to primary source ↓e08PDF pp. 5–6, Section 4.1 Stratified Bootstrap Confidence Intervals
Resamples runs independently with replacement within each task, retains N runs for M tasks and recomputes statistics repeatedly. Notes the empirical-distribution assumption and useful percentile estimates around N=10.
Go to primary source ↓e09PDF pp. 6–7, Section 4.2, Figure 7 axes/caption and Equation (1)
Run-score tails count pooled run exceedances with 1/M task weighting and 1/N within-task weighting. Figure 7 uses ten runs, except five for SimPLe; bands are pointwise 95%. Run-score profiles are left and task-average profiles right.
Go to primary source ↓e10PDF p. 6, Section 4.2 final paragraph; p. 7, Figure 7 and text immediately below Equation (1)
Prose references the run-score distribution as Figure 7 right, conflicting with its caption and axes. The population-mixture expression below Equation (1) prints 1/N multiplying a sum over M tasks, whereas Equation (1) uses 1/M.
Go to primary source ↓e11PDF pp. 7–8, Section 4.3 Robust and Efficient Aggregate Metrics
Defines IQM as the mean after trimming the bottom/top quarters of combined runs. Describes optimality gap with target γ=1, and average per-task P(X_m>Y_m), which ignores magnitude and refers to Equation A.2 for per-task estimation.
Go to primary source ↓e12PDF p. 8, Figure 9 and caption; Section 5 ALE discussion; footnote 9
Atari 200M comparison uses 55 sticky-action games with stated 5/3/11 run counts. Metric rankings differ; M-IQN/Rainbow and C51/DQN-Adam intervals overlap. DreamerV2 and M-IQN score above 50 on James Bond. Dopamine Rainbow is distinguished from the original implementation.
Go to primary source ↓e13PDF p. 9, Figure 11 and caption; Section 5 DeepMind Control Suite
Six-task comparisons at 100k/500k steps divide scores by 1000. Includes percentile stratified-bootstrap and studentized intervals, rank distributions estimated with 200,000 resamples, and substantial ranking uncertainty.
Go to primary source ↓e14PDF pp. 9–10, Section 5 Procgen; p. 10, Figure 12, right panel/caption and following paragraph
Easy-mode Procgen comparison has 16 tasks, 10 runs per task and 95% bootstrap intervals. The authors state a 40–50% chance for UCB-DrAC to outperform PLR, while explaining that probability of improvement ignores magnitude.
Go to primary source ↓e15PDF p. 9, Section 5 Procgen paragraph; p. 10, Figure 12 left-panel x-axis
The prose invokes Figure 12 to discuss PPO-normalized scores; the actual displayed x-axis reads Min-Max Normalized Score. The figure is interpreted using its visible label.
Go to primary source ↓e16PDF p. 10, Section 6 Discussion
Distinguishes results from methods reproducibility, rejects fixed seeds as a complete solution, and reports release of rliable, a notebook and individual-run scores.
Go to primary source ↓e17PDF p. 17, Checklist, items 3(a), 3(b) and 3(d)
The checklist points to Appendix A.1 for code/data/instructions and Appendix A.2 for training and compute details; these details are not printed on the checklist page.
Go to primary source ↓e18PDF p. 11, Societal Impacts
Acknowledges that increased statistical sophistication can enable statistical abuses and describes reliable evaluation as an ongoing process.
Go to primary source ↓8.5 Primary sources
Deep Reinforcement Learning at the Edge of the Statistical Precipice ↗
PDF · 10,671 extracted words
Source fingerprint
7553e280d4a1560860bc5f6be2d70539fc5b510e0e3bb2d556824e51e9a80252