Leveraging Procedural Generation to Benchmark Reinforcement Learning
1. Paper overview
In one sentence: Procedurally varied game levels expose memorization that training scores can hide, while wider visual policies improve generalization at a higher, unquantified compute cost. e02e03e07e08e09e10e11e12
| At a glance | What to know |
|---|---|
| Research problem | Source description Repeated exposure to near-identical game states can reward trajectory memorization. Procgen makes generalization measurable within each game by generating diverse training and test levels, while retaining heterogeneous visual-control tasks across games. e02 |
| Core mechanism | |
| A key reported result | Zero-shot generalization from 500 levels: Approximately 0.28 test versus 0.52 train at 200M timesteps, read from Figure 4. Mean normalized return across the benchmark. Hard mode; PPO with default IMPALA CNN; 200M timesteps per environment; 500 training levels and unseen test levels; three seeds. The same trained baseline evaluated on familiar versus unseen levels. A substantial aggregate generalization gap remains. These are visual estimates, not tabulated values; shading is standard deviation across seeds. e04e06e09 |
| Reading caution |
Core contributions
- Source description
Separate protocols measure sample efficiency over the full level distribution and zero-shot generalization after training on a finite level set. e04
Figure 1. A common observation/action interface spans visibly different game environments. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the montage as sixteen different environments, not successive frames of a single rollout. Platforms, corridors, moving entities and strongly different backgrounds make the visual-control demands concrete. The surrounding method text identifies a second source of diversity that this montage alone cannot show: within each game, procedural generation varies layouts, assets, entity positions and spawn times. All environments expose 64 × 64 × 3 RGB observations and the shared discrete action interface. This lets the authors apply common training machinery while changing the visual task distribution. The figure has no environment labels or architecture blocks, so its individual screenshots should not be treated as a complete task specification. e02e03e12e13
What it supports. Procgen combines diversity across games with variation within each game. The montage supports the first point visually; the procedural-generation description supports the second. Together they explain why repeatedly mastering a few familiar scenes is an incomplete measure of the intended capability.
Where the evidence stops. Screenshots do not establish solvability, dynamics fidelity or agent success. The source only states a belief that more than 99% of levels are solvable, and the detailed environment descriptions are deferred to absent Appendix A.
2. Motivation
2.1 The problem and the proposed response
Repeated exposure to near-identical game states can reward trajectory memorization. Procgen makes generalization measurable within each game by generating diverse training and test levels, while retaining heterogeneous visual-control tasks across games. e02
2.2 What this reading follows
Procgen turns the level distribution into an experimental variable. An agent may become excellent at a familiar game sequence yet perform poorly when the next episode presents a new layout. The paper makes this distinction measurable across sixteen games with a shared visual-control interface. Read the figures in three stages: first inspect the environment diversity, then separate familiar-level learning from unseen-level performance, and finally compare model capacity and algorithms under the appropriate protocol. Wider IMPALA networks improve held-out return, but that finding does not erase overfitting or establish better wall-clock efficiency. The supplied proceedings PDF also leaves important implementation details in appendices that are absent from this artifact. e02e03e07e08e09e10e11e12
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The benchmark/simulator category and reinforcement-learning/generalization subcategories are supported. Architecture, prediction paradigm and quadrant are appropriately not applicable: this contribution defines environments and evaluation protocols, rather than a One Model or joint future/action-prediction system. The recorded Physics simulation subcategory is not established as a distinct contribution; game dynamics alone do not demonstrate physical fidelity. e02e03e06e11
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate a changing level from a stochastic transition
Procgen randomizes the content an agent encounters while retaining deterministic state transitions within a specified level. That distinction matters: repeated observations need not change their consequences for the benchmark to require generalization. A new layout or entity placement can already make memorized trajectories unreliable. The common RGB interface and discrete actions support a shared experimental pipeline, while each environment retains its own mechanics and reward scale. Training optimizes a visual policy with PPO by default; evaluation lets that policy act on levels sampled under the chosen protocol. There is no proposed learned world-model rollout to interpret here. The paper's central experimental object is the distribution of game instances, and its main question is whether behavior learned on some instances transfers to others. e02e03e04e06
5.2 Read generalization as two scores and an exposure history
The 500-level protocol makes familiar-level return and unseen-level return separately visible. A policy that improves only on the former is learning something useful for its training distribution, but that does not establish broad transfer. Figure 2 adds a second dimension: increasing the level pool can improve test scores and, after a threshold, sometimes training scores as well. The authors interpret this as an implicit curriculum, because experience on varied levels can help solve other training levels. Reader analysis: the fixed-sequence experiment adds a caution about exposure. Later levels are encountered only after earlier successes, so access to an arbitrarily long sequence does not mean diverse experience in practice. A convincing mechanism test should therefore measure which levels the agent actually visits, alongside both returns. e07e08e09
5.3 Distinguish capacity, optimization and evaluation budget
The model-size experiment is strongest when read as a comparison of training recipes under a fixed interaction budget. Multiplying IMPALA channels by k increases parameter count approximately by k², and the authors simultaneously scale the learning rate by 1/√k. Figure 5 shows improved test return for wider models under that recipe. Reader analysis: this supports an operational choice if environment interactions are scarce, but it does not separately identify capacity as the sole cause or establish the cheapest model in compute terms. The Rainbow experiment reinforces the optimization issue: the authors needed substantial implementation adjustments before obtaining useful behavior, and some failures remained unexplained. Architecture and algorithm rankings therefore depend on both the level-distribution protocol and the training configuration, not just a model name. e04e10e11
5.4 Training and inference
During training
Default hard-mode PPO training lasts 200M timesteps per environment; the authors estimate about 24 GPU-hours and 60 CPU-hours. Easy-mode recommendations are 25M timesteps, roughly 3 GPU-hours, and 200 training levels for generalization. These are implementation-dependent estimates without hardware models. e04
Scaling experiments multiply IMPALA convolutional channels by k = 1, 2 or 4, increasing parameter count approximately by k². Learning rate is multiplied by 1/√k; Nature-CNN uses the smallest IMPALA model's learning rate. Thus capacity and learning rate change together. e10
Rainbow uses IMPALA convolutions and experience from 64 parallel environment copies. The authors introduce eight workers with shared parameters and averaged gradients, then increase batch size 16-fold, decrease update frequency 16-fold and increase learning rate fourfold. Full settings are deferred to Appendix D. e11
During inference
The trained policy acts in the game from visual observations; generalization evaluation measures zero-shot return on unseen levels, without described test-time adaptation. Ordinary evaluation episodes contain one level. In the sequence ablation, success advances to the next level and failure ends a potentially multi-level episode. e03e07e08e09
These experiments concern executed actions in game environments. They supply no evidence of physical robot deployment, learned world simulation or future-video prediction. e02e03e06e11
5.5 Implementation flow
- Generate a task instance
Procedural logic selects layouts, visual assets, entity locations and spawn times. State transitions are deterministic once the level is specified; Chaser's enemy decisions are pseudorandom conditional on the level seed. Diversity comes from the level distribution. e02e03
- Choose the evaluation question
Sample-efficiency runs train and test on the full distribution. Generalization runs constrain training to a finite set, normally 500 hard-mode levels, and evaluate zero-shot performance on unseen levels. The paper describes testing on the full distribution and also calls these levels held out; explicit seed-exclusion mechanics are absent. e04e09
- Learn visual control
The default agent uses PPO and the IMPALA convolutional architecture, without frame stacking. The benchmark supplies observations and accepts game actions; the evaluated baselines do not introduce a learned future-state predictor or an action-planning rollout. e03e06e11
- Aggregate without mixing reward scales
Report raw episodic returns within games and average normalized returns across games. Approximate bounding constants align disparate reward scales. They are referenced to absent Appendix C, so exact aggregate-score reconstruction is unavailable from this artifact alone. e05e12
6. Experiments & results
Procgen benchmarks reinforcement learning with 16 procedurally generated games. Its central intervention is to vary the levels encountered by an agent and evaluate unseen levels separately. PPO baselines expose substantial overfitting; wider convolutional models improve held-out returns, while fixed level sequences can produce impressive training scores with little transfer (e02, e07–e10).
The supplied nine-page proceedings PDF contains a screenshot montage and five quantitative plots, but no architecture diagram or numerical table. Figure 1 therefore illustrates the benchmark environment mechanism rather than a neural architecture, and Figures 2–6 supply the original quantitative evidence. No table has been reconstructed. The PDF references Appendices A–I but ends with references on page 9; appendix architecture/configuration detail, normalization constants, additional diagnostics and easy-mode results cannot be illustrated from this artifact. e03e06e12
6.1 Read the original evidence
Figure 2. Increasing level diversity can improve held-out performance and sometimes training performance. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axis is the number of distinct training levels, on a logarithmic scale; it is not elapsed training time. Each training set receives 200M PPO timesteps. Blue curves report performance on training levels, orange curves on test levels, and each episode contains one level. The panels use different raw-return scales, so compare the two curves within a game before comparing patterns across games. Four seeds contribute to the displayed means and standard deviations. Heist and Maze illustrate rising held-out performance as the level set grows; other games, such as CoinRun and Plunder, show that the dependence need not be smooth or monotonic. e03e04e07
What it supports. The authors report needing as many as 10,000 levels to close the generalization gap. Several blue curves eventually rise alongside their orange counterparts, motivating the hypothesis that a broader level distribution helps the agent learn reusable behavior even within its training set.
Where the evidence stops. Ten thousand is not a universal sufficiency threshold. Equal timestep budgets give different exposure per level as the set grows, and the proposed implicit curriculum is an interpretation rather than an independently controlled mechanism.
Figure 4. The standard 500-level protocol leaves a substantial aggregate generalization gap. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the large right-hand panel: blue is training performance and orange is zero-shot performance on unseen levels. At 200M timesteps, the mean normalized scores are approximately 0.52 and 0.28. Then inspect the left panels to see which games contribute to that separation. Their vertical axes show per-game raw scores, whereas the right panel averages normalized returns using environment-specific constants. Three seeds determine the means and standard deviations. The 500-level training set is chosen near a region where generalization begins to emerge, making this a practical diagnostic protocol rather than a claim that 500 levels suffice for mastery. e04e05e06e09e12
What it supports. The default PPO baseline continues to improve on familiar levels while remaining substantially worse on unseen ones. The per-game panels show why the aggregate should be accompanied by individual curves: similar aggregate performance can conceal very different learning and transfer behavior across environments.
Where the evidence stops. The endpoint values are approximate readings from the plot. Normalized return is not a success rate, and the constants are absent with Appendix C. The authors warn that a small gap can also arise when both splits perform poorly.
Figure 6. Adapted Rainbow wins some games, while PPO performs more consistently across the suite. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Both algorithms train and test on the full distribution, so this figure addresses sample efficiency rather than the finite-level generalization gap. Blue denotes PPO and orange denotes Rainbow; each panel has its own raw-score scale. Compare curve height and learning speed within a panel. Rainbow is stronger in StarPilot and BigFish, whereas PPO is stronger in CoinRun, Maze and Heist. These are the authors' implementations using IMPALA convolutions. Section 5 explains that Rainbow required gradient averaging across workers and changes to batch size, update frequency and learning rate. The displayed means and standard deviations use three seeds. e03e04e11e12
What it supports. There is no uniform winner across environments. The authors characterize PPO as more consistent while acknowledging substantial Rainbow improvements in some games. This heterogeneity is useful benchmark evidence: an aggregate alone would obscure algorithm-specific strengths and persistent failures.
Where the evidence stops. The comparison concerns a specifically adapted Rainbow implementation. The source does not isolate the cause of its remaining instability, and the complete hyperparameter list is deferred to absent Appendix D. It does not establish equal wall-clock cost.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Zero-shot generalization from 500 levels Hard mode; PPO with default IMPALA CNN; 200M timesteps per environment; 500 training levels and unseen test levels; three seeds. | Approximately 0.28 test versus 0.52 train at 200M timesteps, read from Figure 4. Mean normalized return across the benchmark | The same trained baseline evaluated on familiar versus unseen levels. A substantial aggregate generalization gap remains. These are visual estimates, not tabulated values; shading is standard deviation across seeds. e04e06e09 |
| Training-level requirements PPO; 200M timesteps per training set; 100–100,000 levels; default hard mode; four seeds. | The authors report needing as many as 10,000 levels to close the gap; requirements vary by game. Train and test mean raw episodic return; one level per episode | Small versus large training sets within each environment. Some training scores also improve with more levels. The proposed implicit-curriculum explanation is an author interpretation, not an isolated mechanism test. e03e07 |
| Fixed-sequence transfer ablation Easy mode; fixed starting level and progression sequence during training; randomized level sequences at test; curves through 200M timesteps; four seeds. | Training returns rise markedly in most games while test returns remain low; no exact scalar summary is provided. Raw episodic return over potentially multiple sequential levels | Fixed versus randomized evaluation sequences for the trained agents. Progress through familiar levels provides weak evidence of transfer. Multi-level returns and easy difficulty prevent direct comparison with ordinary hard-mode, single-level scores. e08 |
| Model width and held-out performance Hard-mode PPO; 500 training levels; 200M timesteps; three seeds; learning rate scaled with model width. | Approximately 0.50 for IMPALA ×4, 0.42 for ×2 and 0.28 for ×1; Nature-CNN is near zero, from Figure 5. Mean normalized test return | Models under the same timestep and level-count protocol. Wider models improve absolute held-out performance but retain train–test gaps. This is not a matched-compute or fixed-learning-rate comparison. e04e10 |
| Model width and sample efficiency Hard-mode PPO; training and testing on the full level distribution; three seeds. | Approximately 0.85 for IMPALA ×4 versus 0.75 for ×1 and 0.13 for Nature-CNN, from Figure 5. Mean normalized return at 200M timesteps | Different architectures with the source's learning-rate choices. Larger models learn more per environment interaction; the plot does not establish superior wall-clock efficiency. e04e10 |
| PPO versus adapted Rainbow Hard mode; full-distribution training and testing; IMPALA convolutional architecture; curves through 200M timesteps; three seeds. | Rainbow leads in games including StarPilot and BigFish; PPO leads in games including CoinRun, Maze and Heist. Per-game raw score learning curves | The authors' PPO and modified Rainbow implementations. Performance is game-dependent. The authors report greater PPO consistency but leave Rainbow's remaining instability undiagnosed; no universal algorithm ranking follows. e03e11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Fixed progression can produce large familiar-sequence scores with little transfer. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Here the horizontal axis returns to training timesteps, in millions. Each training episode starts at the same first level; success unlocks the next level in a fixed sequence, while failure terminates the episode. Test episodes use randomized sequences. Read the blue and orange curves together: large blue gains frequently coexist with very low orange returns. The shaded bands are standard deviations over four seeds, and some are wide. Unlike Figure 2, an episode can include multiple completed levels, so a large score can accumulate across a familiar progression. The experiment uses easy difficulty to make this sequential setup more tractable. e07e08
What it supports. Competence on the first several familiar levels gives a misleading impression of broadly learned game skill. The persistent separation between training and randomized-test returns is the key diagnostic; it supports the need to evaluate variation explicitly rather than infer transfer from training progress alone.
Where the evidence stops. The intervention changes level ordering, exposure and episode progression. Its easy-mode, multi-level scores are not directly comparable with hard-mode, single-level results. It diagnoses this setup without proving that every fixed-level benchmark behaves identically.
Figure 5. Wider visual models improve both full-distribution learning and held-out return. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the two evaluation protocols separate. The left panel trains and tests over the full level distribution, measuring sample efficiency. The right panel restricts training to 500 levels: dotted curves are training scores and solid curves are held-out scores. Colors identify IMPALA channel multipliers one, two and four, with Nature-CNN in red. The multiplier scales channels, not parameter count directly; parameters grow approximately quadratically. Section 4 also reduces the learning rate by 1/√k for width multiplier k. At the right edge, compare solid curves of different colors to assess test performance, then compare dotted and solid curves of the same color to assess overfitting. e04e05e09e10e12
What it supports. At 200M timesteps, approximate test scores rise from 0.28 for IMPALA ×1 to 0.42 for ×2 and 0.50 for ×4. Larger capacity improves unseen-level performance under the source's training recipe, but the dotted curves remain higher. Better generalization does not mean the train–test gap disappears.
Where the evidence stops. Width and learning rate change together, and a common interaction budget does not imply equal compute. The plot shows means and standard deviations across three seeds; it provides neither exact endpoint tables nor a wall-clock comparison.
7. Analysis & limitations
7.1 What the evidence leaves open
The default tasks emphasize visual recognition and low-level control, deliberately minimizing memory requirements and avoiding mandatory custom exploration rewards. Transfer to more complex settings is an expectation in the conclusion, not a demonstrated result. e13e15
Level solvability is not guaranteed: the authors believe more than 99% are solvable. This is a design belief rather than a measured success statistic. e13
A small train–test gap can reflect poor learning on both splits. The sequence ablation also changes reset/progression structure and uses easy mode, so it does not isolate every cause of poor generalization. e08e09
Reported uncertainty is standard deviation over three or four seeds, depending on the figure. Exact endpoint tables, seed identities and confidence intervals are not supplied in the main PDF. e07e08e09e10e11e12
7.2 Questions for discussion
- How much of the large-level-set benefit comes from broader coverage versus the authors' proposed implicit curriculum (e07)?
- Would width still improve held-out return under matched compute and learning-rate controls (e10)?
- Which additional evaluation would distinguish poor optimization from overfitting when both train and test scores are low (e09, e11)?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproducing aggregate scores requires Appendix C's normalization constants and Appendix D's hyperparameters, plus precise environment/seed configuration. The main PDF gives no release identifier, hardware model or full software configuration; the linked open-source project alone cannot establish a reproducible installation. e04e05e11e12e14
Proposed check: compare uniform resets within a fixed level pool against fixed sequential progression, with matched difficulty, model and interaction budget. Log level visitation and evaluate both policies on the same unseen single-level set to separate exposure imbalance from memorization. e07e08
Proposed check: cross IMPALA width with fixed versus source-scaled learning rates, keeping the 500-level split constant. Compare held-out return at matched timesteps and measured compute, testing whether width gains survive both controls. e04e10
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Measure whether fixed progression fails through restricted exposure
Reader-proposed check, not performed: use the same easy-mode level pool, PPO model and 200M interaction budget for two training conditions—fixed sequential progression and uniform single-level resets. Log visits and successful completions per level. Evaluate both policies on the identical unseen single-level set; also evaluate the sequential agent under the paper's randomized-sequence protocol. If the uniform-reset agent sees broader content and transfers better, exposure restriction is a plausible contributor. If transfer remains equally poor despite broad visitation, exposure alone is insufficient. This comparison requires the missing implementation settings before it can be called a reproduction. e04e06e08e12
Check 2: Test whether wider IMPALA remains better after optimization controls
Reader-proposed check, not performed: hold the hard-mode 500-level split fixed and cross channel multipliers 1, 2 and 4 with a common learning rate versus the source's 1/√k scaling. Keep other PPO settings fixed and evaluate on the same unseen levels across multiple seeds. Report per-game returns and mean normalized return both at 200M timesteps and at matched measured compute. A width advantage that survives learning-rate controls and compute matching would support a stronger capacity-efficiency claim; an advantage that disappears would narrow the original result to its stated recipe and interaction budget. e04e05e06e10e12
8.3 Reading coverage
Visual audit: All nine supplied PDF pages were rendered and visually inspected: title/authors/affiliation and benchmark motivation on page 1; environment montage, interface and protocol on page 2; level-count results, normalization and baseline configuration on page 3; fixed-sequence ablation on page 4; 500-level results and scaling setup on page 5; model-size results and Rainbow configuration on page 6; algorithm results and conclusion on page 7; references and artifact ending on pages 8–9. All six final original crops were inspected at their saved dimensions. Figure legends, line styles, axes and captions were cross-checked against the method text; no claim-relevant discrepancy was found. Referenced appendices and separate supplements were not supplied or inspected. The quantitative values labeled approximate are visual readings, not recovered raw data.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Procgen Benchmark
- 2.1. Environment Desiderata
- 2.2. Experimental Protocols
- 2.3. Hyperparameter Selection
- 3. Generalization Experiments
- 3.1. Level Requirements
- 3.2. An Ablation with Deterministic Levels
- 3.3. 500 Level Generalization
- 4. Scaling Model Size
- 5. Comparing Algorithms
- 6. Related Work
- 7. Conclusion
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity: the observed title and all four authors match the catalog. The inspected title page identifies the ICML 2020 proceedings edition, PMLR 119. No finer revision identifier or revision history is supplied; equivalence to another edition was not established.
- The complete supplied nine-page PDF body and references were read. Appendices A–I are referenced but absent from this artifact: environment descriptions, exploration/memory variants, normalization constants, hyperparameters, additional learning curves, ALE comparisons, frame-stacking discussion and easy-mode results remain unreviewed.
- Separate supplemental material availability has not been fully verified.
- The supplied extraction does not reconstruct figure images. All nine PDF pages and Figures 1–6 were therefore inspected visually; the PDF contains no numbered tables.
- Code and external links were not inspected, and experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author line and affiliation/proceedings footnote
The title matches the catalog. Authors are Karl Cobbe, Christopher Hesse, Jacob Hilton and John Schulman, all affiliated with OpenAI, San Francisco, CA, USA. The footer identifies ICML 2020, PMLR 119; no finer revision label appears.
Go to primary source ↓e02PDF p. 1, Abstract; Sections 1–2
Procgen contains 16 procedurally generated game-like environments for RL sample efficiency and generalization. Generation governs layout, assets, entity locations and spawn times; the motivation is to distinguish robust skills from memorized trajectories.
Go to primary source ↓e03PDF p. 2, Figure 1; Section 2 and Section 2.1, Shared Action and Observation Space, Tunable Difficulty; footnote 1
Figure 1 presents screenshots of all games. The interface uses RGB observations of shape 64 × 64 × 3 and a shared discrete action space described as 15 dimensional, including no-op choices in some games. Transitions are deterministic; Chaser uses seed-conditioned pseudorandom enemy decisions. Hard mode is the reporting default.
Go to primary source ↓e04PDF pp. 2–3, Section 2.2; p. 5, Section 3.3
Default PPO uses 200M timesteps, estimated at 24 GPU-hours and 60 CPU-hours per environment. Easy-mode recommendations are 25M timesteps, approximately 3 GPU-hours and 200 training levels. Generalization normally uses 500 levels; sample efficiency uses the full distribution. Section 3.3 specifies zero-shot unseen-level testing and notes training duration is not formally restricted.
Go to primary source ↓e05PDF p. 3, Section 2.2, normalized-return formula and following paragraph
Normalized return is (R − R_min)/(R_max − R_min), with R raw expected return and constants approximately bounding it. Benchmark aggregation takes the mean. Constants are deferred to Appendix C; normalization does not guarantee bounds or define clipping.
Go to primary source ↓e06PDF p. 3, Section 2.3, last two paragraphs
Procgen experiments omit frame stacking and use the IMPALA convolutional architecture by default. The paper says smaller architectures struggle under high diversity. Frame-stacking details and ALE comparisons are deferred to Appendices H and F.
Go to primary source ↓e07PDF p. 3, Figure 2 and caption; p. 4, Section 3.1
Training sets range from 100 to 100,000 levels; each gets 200M PPO timesteps. Figure 2 reports raw episodic returns for one-level episodes, mean and standard deviation across four seeds. The authors report up to 10,000 levels to close gaps and attribute some increasing training returns to an implicit curriculum.
Go to primary source ↓e08PDF p. 4, Figure 3 and caption; Section 3.2
The easy-mode ablation starts each episode on the first level of a fixed sequence, advances after success and terminates on failure. Testing randomizes sequences. Agents rarely reach beyond the twentieth level in practice. Curves through 200M timesteps show large train–test gaps; returns can include multiple levels, with mean and standard deviation over four seeds.
Go to primary source ↓e09PDF p. 5, Figure 4, right aggregate panel and caption; Section 3.3
The 500-level generalization protocol evaluates zero-shot return on unseen levels. At 200M timesteps the plotted aggregate endpoints are approximately 0.52 train and 0.28 test, not exact tabulated values. Means and standard deviations use three seeds. The text warns that small gaps can reflect low returns on both splits.
Go to primary source ↓e10PDF p. 5, Section 4; p. 6, Figure 5, both panels, legends and caption
IMPALA channels scale by k = 1, 2, 4, parameters approximately by k² and learning rate by 1/√k. Nature-CNN uses the smallest IMPALA learning rate. At 200M timesteps, visually estimated generalization test scores are about 0.28, 0.42 and 0.50 for widths 1, 2 and 4; Nature-CNN is near zero. Full-distribution scores are about 0.75, 0.85 and 0.13 for IMPALA ×1, ×4 and Nature-CNN respectively. Three-seed mean and standard deviation are shown; dotted lines denote train and solid lines test in the right panel.
Go to primary source ↓e11PDF p. 6, Section 5; p. 7, Figure 6 and caption, especially StarPilot, BigFish, CoinRun, Maze and Heist panels
Rainbow uses IMPALA convolutions, 64 environment copies and a replay buffer. Eight gradient-averaging workers improve performance; batch size and update frequency change by factors of 16, and learning rate by four. Figure 6 compares full-distribution training/testing over three seeds: Rainbow leads in StarPilot and BigFish, PPO in CoinRun, Maze and Heist. The authors cannot diagnose remaining Rainbow instability.
Go to primary source ↓e12PDF pp. 2–6, references to Appendices A–I; pp. 2–7, Figures 1–6; pp. 7–9, References and end of supplied PDF
The artifact contains a screenshot montage and five quantitative figures, but no numbered table or architecture diagram. It ends with references on page 9. Referenced appendices covering environment descriptions, variants, constants, hyperparameters and additional results are not included in this PDF.
Go to primary source ↓e13PDF p. 2, Section 2.1, Level Solvability, Emphasis on Visual Recognition and Motor Control, Tunable Dependence on Exploration and Memory
The authors believe greater than 99% of levels are solvable but do not guarantee this. Default tasks emphasize visual recognition and motor control, minimal memory and tractability without custom exploratory rewards; variants are deferred to Appendix B.
Go to primary source ↓e14PDF p. 1, Section 1, final sentence
The paper describes the environments as open-source and links the OpenAI Procgen repository. This establishes an author-provided code pointer, not inspected code, a verified license or a reproduced implementation.
Go to primary source ↓e15PDF p. 7, Section 7, Conclusion
The authors argue that level diversity supports generalization and sample-efficiency evaluation. Application of insights to more complex settings is expressed as an expectation, not tested deployment evidence.
Go to primary source ↓8.5 Primary sources
Leveraging Procedural Generation to Benchmark Reinforcement Learning ↗
PDF · 4,477 extracted words
Source fingerprint
1a3292f3dbf2187a2254cceb8af970c1022154ce90b9ae1ebb7de5a4d51c712a