Model-Based Reinforcement Learning for Atari
1. Paper overview
In one sentence: SimPLe turns limited Atari experience into a learned simulator for policy training, exchanging real interactions for computation while controlling prediction drift through stochastic latents and short rollouts. e01e02e06e07e10e17
| At a glance | What to know |
|---|---|
| Research problem | Source description Can a policy learn useful Atari behavior from roughly two hours of interaction without emulator RAM? Four visual frames incompletely reveal game state, so predicting one deterministic future can miss hidden events. SimPLe must learn both consequential visual changes and rewards, then transfer behavior trained in an imperfect simulator back to the original game. The target is interaction efficiency, not minimum computation. e01e03e08e14e18 |
| Core mechanism | |
| A key reported result | Atari Pong at the nominal 100K interaction budget: SimPLe: 12.8 (17.2) Game score, mean (standard deviation). Original Atari environment; SimPLe uses 102,400 agent steps with action repeat 4. Table 3 reports means with standard deviations; SimPLe has five training runs and Appendix D describes temperature-based evaluation. PPO_100k: −20.5 (0.6); Rainbow_100k: −19.5 (0.2). Rainbow_500k: 19.9 (0.4). A large early-learning gain over these specific 100K baselines coexists with high variability and reversal against Rainbow with more experience. This is executed Atari performance. e08e10e15 |
| Reading caution | Source description The updated introduction says each of two later tuned model-free methods wins thirteen of twenty-six games, with SimPLe winning the other thirteen. The original nearly-all-games claim must therefore remain tied to the paper's own Rainbow/PPO implementations. e17 |
Core contributions
Figure 1. The simulator and the policy improve through alternating updates and new real experience. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three panels on the right from top to bottom, then return to the top. A policy first generates observations by interacting with Atari. Those observations train the world model, which supplies a simulated environment for reinforcement learning. The circular arrows summarize the same order: observations, world model, policy. The asterisk on Self-Supervised matters: the caption calls observation prediction self-supervised, while rewards provide supervised targets. Algorithm 1 makes the data retention explicit by adding new trajectories to D before each model update. Thus the loop keeps collecting evidence about the states reached by the improving policy. e02e19
What it supports. The design makes model learning and data collection depend on each other. An initial random dataset need not cover every useful game state: later policies can expose additional situations, which enter the next simulator update. This is the paper's operational bridge from video prediction to executed Atari behavior.
Where the evidence stops. The diagram is an information-flow overview. It does not establish online action-sequence planning or a single network that jointly generates observations and actions; Section 7 leaves planning with the simulator to future work.
2. Motivation
2.1 The problem and the proposed response
Can a policy learn useful Atari behavior from roughly two hours of interaction without emulator RAM? Four visual frames incompletely reveal game state, so predicting one deterministic future can miss hidden events. SimPLe must learn both consequential visual changes and rewards, then transfer behavior trained in an imperfect simulator back to the original game. The target is interaction efficiency, not minimum computation. e01e03e08e14e18
2.2 What this reading follows
Learning to predict a game is useful only if the resulting predictions help an agent act in the original environment. SimPLe makes that connection through a repeating cycle: gather real frames and rewards, learn a video model, train a policy inside it, and collect again. The crucial constraints are hidden state and accumulating model error. Discrete latent bits represent alternative futures, while resets to real observations keep simulated training near familiar states. The figures below explain those choices alongside actual game scores and their variability. This reading covers the supplied April 2024 arXiv v5, whose introduction acknowledges competitive later model-free baselines. e01e02e06e07e10e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational/classical model-based RL category fits Algorithm 1: an action-conditioned dynamics model supplies experience to a separate policy learner. Frame-and-reward prediction is not joint future-and-action prediction or inverse dynamics. The paper does not establish a unified world-action architecture; retaining Not applicable for the catalog's architecture, prediction paradigm and quadrant is appropriate. e02e03e19
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make prediction useful through a separate learning loop
Begin with the distinction between the real environment and its learned substitute. The real environment supplies observations and rewards; the substitute learns to generate observations and rewards in the same interface. Algorithm 1 first expands D with real trajectories, then fits the substitute, then runs reinforcement learning inside it. PPO can therefore improve a policy using many simulated transitions while spending relatively few new Atari interactions. The improved policy is then executed in Atari to collect the next dataset increment. This feedback is essential: random initial experience does not necessarily contain the events a competent policy will reach. The world model remains a transition-and-reward predictor, while the policy remains the component selecting actions. Section 7 explicitly treats planning and using model gradients as further possibilities. e02e03e07e19
Figure 2. Discrete latent bits switch from target-informed training to autoregressive simulation. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the four input frames in the lower left and follow the encoder toward the narrow middle, then the decoder toward the predicted frame. Dashed black arrows carry skip connections; green action paths multiply decoder activations. The upper branch additionally receives the actual next frame, but only while training the predictor. Its orange training arrow supplies discretized bits. During simulation, the blue inference arrow supplies bits from the recurrent bit predictor instead. The caption states that backpropagation bypasses discretization, although no separate bypass-gradient arrow is drawn. These two sources of latent bits distinguish supervised model fitting from generating an unknown future. e03e06e14e18
What it supports. Stochasticity enters through a learned discrete code rather than requiring a single deterministic continuation. This gives the model a way to represent events hidden by the short observation history, such as opponent spawns. The auxiliary predictor supplies that code without access to the true next frame during simulation.
Where the evidence stops. Read this as topology, not an exact implementation specification: the schematic's channel widths differ from Appendix C's layer table. Section 4 also describes architectural variants. Neither the graphic nor its latent branch is evidence of action prediction.
5.2 Separate hidden futures from avoidable pixel error
Four frames do not reveal every variable needed to determine the next Atari image. A deterministic predictor can therefore face several legitimate continuations. During training, SimPLe's latent inference network sees the realized next frame and encodes additional information into bits; the frame predictor learns to use that information. During simulation, the auxiliary recurrent network supplies bits without the target frame. This is why the orange and blue arrows in Figure 2 matter. The model also addresses a different problem: easy background pixels can consume optimization effort while small consequential objects remain wrong. Its floored pixel loss removes gradients from sufficiently accurate pixels. Stochastic latent modeling addresses ambiguous futures, whereas loss clipping changes which prediction errors keep influencing training; neither alone guarantees useful control. e03e04e06e14
Figure 11. A plausible continuation can differ from the recorded opponent configuration. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the nearby environment and model examples using the caption's left/right convention. The corridor remains recognizable, while opponent arrangements differ. Section 6.3 explains why this matters: after the current opponents are removed, visually similar recent histories can precede different new opponents, because the four-frame observation does not reveal the emulator's full internal state. Appendix B treats differences in spawned opponents as potentially benign model deviations. The visual therefore helps distinguish a plausible sampled future from a pixelwise replay of the particular future that occurred. It does not supply a calibrated distribution over all possible opponent configurations. e13e14
What it supports. The model can render a coherent game scene while disagreeing with the observed realization. Together with the source's qualitative discussion, this illustrates why exact frame matching is not the sole criterion for usefulness in policy training. Actual control performance must still be assessed in Atari.
Where the evidence stops. This is a qualitative, limited-resolution source example. It cannot establish transition probabilities, long-horizon accuracy or policy success by itself. The paper separately documents failures involving small objects and large scene changes.
5.3 Read short rollouts as an error-management choice
Reader analysis: restarting simulations from D limits how far PPO travels through predictions unsupported by real observations. Short rollouts introduce their own cost, because consequences can extend beyond the rollout boundary; the paper compensates by adding a value estimate at the end. The Seaquest diagnostic supports this overall recipe, but its alternative also changes the horizon to 1,000 steps. It therefore cannot establish which reset ingredient causes the improvement. The benchmark table adds the practical boundary: a large Pong gain over 100K model-free baselines coexists with a large standard deviation, and Rainbow overtakes it with more real data. Useful prediction, stable policy learning and superior performance at every budget are separate conclusions. The evidence supports the first under particular training choices, without establishing the other two generally. e07e10e21
5.4 Training and inference
During training
Section 5 describes fifteen iterations: 45K model-training steps initially and 15K in later iterations, retaining learned parameters. Appendix A says the best results use five times longer world-model training, while other ablations use the shorter budget. These settings must remain distinct. e07e12
Scheduled sampling replaces input frames with the model's previous predictions, ramping replacement probability to 100% around the middle of the first training iteration. Pixel-loss clipping stops already accurate background pixels from continuing to dominate optimization; the proposed explanation about small objects is the authors' conjecture. e04e05
Sixteen parallel PPO agents generate simulated experience; the paper reports 15.2M simulated interactions. Real experience is also used directly for PPO, though the authors regard its effect as negligible relative to simulated data. The claimed interaction economy therefore depends on extensive additional model computation. e07e08
During inference
When generating simulated rollouts, the next-frame inference network is unavailable: the LSTM supplies discrete latents autoregressively. Appendix C specifies 128 bits produced in eight-bit chunks. This is simulator inference during policy training, distinct from executing the policy in Atari. e06e18
For real-game evaluation, use the learned policy on observed frames. Appendix D evaluates softmax(logits(π_i)/T), where π_i is the final policy from run i and T is temperature; T = 0.5 worked best in most cases. Online planning and differentiating through the simulator are future directions, not the reported execution mechanism. e15e19
5.5 Implementation flow
- Aggregate experience
Initialize the policy and collect random experience. Add later trajectories from the current policy to buffer D. Refit the model to the accumulated data, then improve the policy in that model; the new policy changes which states are observed next. e02
- Predict an action-conditioned transition
A convolutional encoder and decoder use skip connections. Learned action embeddings multiply decoder activations. The stochastic architecture predicts categorical pixel values and reward; it does not output the agent's action. e03
- Represent hidden alternatives
During model training, an inference network sees the actual next frame and produces discretized latent bits. Gradients bypass discretization. Uniform noise before discretization and dropout afterward improve robustness; an auxiliary LSTM learns the bit distribution. e06
- Learn with bounded imagined experience
PPO interacts with the latest simulator. Every N steps, usually 50, reset to an observation state sampled uniformly from D. Bootstrap the final rollout reward with a value estimate to retain information beyond the short horizon. e07
6. Experiments & results
SimPLe learns an action-conditioned video-and-reward simulator from limited Atari experience, trains a separate PPO policy inside it, and repeatedly collects new real-game data. Discrete stochastic latents and short simulated rollouts make imperfect prediction useful for control. Its contribution is sample-efficient policy learning, with substantial computation and instability; the supplied revision also acknowledges competitive later model-free baselines.
6.1 Read the original evidence
Table 3. At equal nominal budgets, early gains coexist with large variation and stronger later baselines. Original paper, p. 23 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a game row before comparing columns: reward scales differ across games. SimPLe uses the nominal 100K setting, actually 102,400 agent interactions; each agent step repeats its action for four frames. Compare it first with PPO_100k and Rainbow_100k, then inspect the 500K and one-million columns separately. Parentheses accompany the reported means as standard deviations, and Appendix D establishes five SimPLe training runs. The SimPLe column matches Table 2's SD long values. Random and human columns are reference scores, not additional learned policies under the same budget. Dashes remain missing entries; they should not be read as zero. e08e10e15e16e17
What it supports. Pong reaches 12.8 (17.2), compared with −20.5 (0.6) for PPO_100k and −19.5 (0.2) for Rainbow_100k. However, Rainbow_500k reaches 19.9 (0.4). Freeway likewise improves over the 100K baselines while retaining substantial variability: SimPLe's 16.7 mean has a 15.7 standard deviation.
Where the evidence stops. These comparisons concern the paper's specific baselines. The revised introduction reports a thirteen-versus-thirteen game split against each later tuned model-free method. The appendix also includes additional games beyond the main selected twenty-six-game suite.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Atari Pong at the nominal 100K interaction budget Original Atari environment; SimPLe uses 102,400 agent steps with action repeat 4. Table 3 reports means with standard deviations; SimPLe has five training runs and Appendix D describes temperature-based evaluation. | SimPLe: 12.8 (17.2) Game score, mean (standard deviation) | PPO_100k: −20.5 (0.6); Rainbow_100k: −19.5 (0.2). Rainbow_500k: 19.9 (0.4). A large early-learning gain over these specific 100K baselines coexists with high variability and reversal against Rainbow with more experience. This is executed Atari performance. e08e10e15 |
| Atari Freeway at the nominal 100K interaction budget Same main comparison and real-interaction budget; Table 3's SimPLe values match its SD long configuration in Table 2. | SimPLe: 16.7 (15.7) Game score, mean (standard deviation) | PPO_100k: 8.0 (9.8); Rainbow_100k: 0.1 (0.1); random: 0.0. The mean improves on these baselines, but the large spread prevents treating a successful run as typical. Appendix B explicitly describes exploration failures. e08e10e13e16 |
| World-model architecture ablation Table 1 ranks configurations using each game's mean over five experiments; Appendix A places these ablations under short model training. | SD: 8 / 16; row labeled default: 10 / 21 Number of games with best / at-least-median configuration score | Deterministic: 0 / 7; deterministic recurrent: 3 / 13. The stochastic configurations rank well, but these are counts of within-game rankings, not success percentages or estimates of effect size. The source's default discount labeling is inconsistent. e11e12e20 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 1. Stochastic configurations win more within-game comparisons, but ranking counts hide effect sizes. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row is a model or training configuration. The paper first averages five experiments for each game and configuration, then ranks those configuration means within the game. Best counts first-place games; at least median counts games in the upper half of that comparison. SD denotes the stochastic discrete model. The gamma-marked row changes discounting, and the 25-step and 100-step rows change simulated rollout length. Preserve the row labeled default literally: its relation to the other SD labels is not fully clarified by the paper's inconsistent default-discount descriptions. The caption's averaging rule is essential even though the crop contains only the table. e11e12e20
What it supports. The deterministic row has zero best scores and seven at-least-median scores; SD has eight and sixteen, respectively. The row labeled default has ten and twenty-one. These counts support the usefulness of the stochastic design across the evaluated games, without showing how large each game's improvement was.
Where the evidence stops. The best main results use longer model training, whereas Appendix A says the other ablations use short training. Default discount labels also differ across sections. These counts are not success rates, confidence intervals or a fully isolated comparison of every mechanism.
Figure 9, right panel. Resetting the learned simulator accompanies better Seaquest learning in the reported comparison. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue random starts curve and orange no random starts curve across the horizontal step axis. Despite that short label, the caption defines the horizontal coordinate as iterations of Algorithm 1, not individual emulator actions. The vertical quantity is mean reward in a comparison that clips game rewards to −1, 0 and 1. Keep this scale separate from Table 3's ordinary game scores. The blue curve stays above the orange curve at the displayed iterations. Appendix A supplies a crucial condition absent from the legend: the no-random-start experiment also uses rollouts of length 1,000. e07e21
What it supports. The figure is consistent with the motivation for limiting model drift: repeatedly returning to real observation states accompanies stronger learning than the reported long-rollout alternative. It provides evidence for the practical combined design and motivates testing which part of that combination carries the benefit.
Where the evidence stops. Reset strategy and rollout horizon change together, so the plot does not isolate randomization alone. The caption does not identify the error-bar statistic; the visible bars should not be assigned a confidence level.
7. Analysis & limitations
7.1 What the evidence leaves open
The updated introduction says each of two later tuned model-free methods wins thirteen of twenty-six games, with SimPLe winning the other thirteen. The original nearly-all-games claim must therefore remain tied to the paper's own Rainbow/PPO implementations. e17
The main twenty-six-game suite was selected for non-random performance by SimPLe or Rainbow at 100K; appendix tables include additional games. This limits interpretation as an unselected test of all Atari tasks. The real budget is 102,400 interactions, not exactly 100,000. e08e10
The authors report lower asymptotic scores and unstable runs. Small important objects can disappear, and global scene transitions can fail. Plausible video alone cannot establish correct dynamics; robotics and autonomous driving remain proposed applications. e13e14e19
The Seaquest no-reset comparison also uses 1,000-step rollouts. Its poorer score supports the combined short-rollout/reset design but does not isolate the causal effect of randomized initial states. e21
7.2 Questions for discussion
- Would randomized restart states still help if rollout horizon and total simulated transitions were held fixed?
- Does stochastic prediction improve real-game reward because it preserves hidden alternatives, or because it changes model capacity and optimization?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction needs ALE preprocessing, an aggregated transition buffer, video/reward supervision and PPO, with training budget, rollout horizon and evaluation temperature fixed. Appendix C reports about 74M parameters and NVIDIA Tesla P100 timings: approximately 0.5 s inference at batch 16 and 0.7 s backpropagation at batch 2. These are component timings, not total training time. e07e08e15e18
Resolve configuration inconsistencies first: Section 5 uses γ = 0.95, whereas Section 6.4 and Appendix A call 0.99 the default. Figure 2's channel widths and Section 4's example architecture/dropout differ from Appendix C's detailed model. No single exact configuration should be inferred by silently combining them. e03e18e20
Appendix E tunes Rainbow on Pong with five agents per setting through one million interactions; its chosen target-update period, 8,000, is absent from the listed search set. PPO refers to standard Baselines hyperparameters, and the Tensor2Tensor link is a source availability claim, not a verified runnable release. Software versions, immutable code revision and evaluation-episode count are not established by the supplied implementation description. e09e15e22
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate reset distribution from rollout horizon
Reader-proposed check, not performed: on Seaquest, run a factorial comparison of rollout lengths 50 and 1,000 with randomized starts from D versus a fixed real starting observation. Hold real-data budget, simulator checkpoint, total simulated transitions, discount, PPO updates and evaluation temperature constant; repeat across at least five seeds. Measure original-game returns and prediction/reward error as a function of rollout position. If randomization helps at matched horizon, it supports a reset-distribution effect. If the benefit disappears once horizon is matched, the original diagnostic mainly implicates rollout length. Report both ordinary game scores and any clipped diagnostic rewards explicitly. e07e08e15e21
Check 2: Test whether hidden-event modeling improves control
Reader-proposed check, not performed: compare deterministic and stochastic discrete world models on Kung Fu Master using identical real transition buffers, matched training budgets and documented capacity differences. Hold the policy-training configuration fixed and evaluate across at least five seeds. For held-out four-frame histories near opponent-spawn events, inspect sampled opponent configurations, reward prediction and subsequent original-game policy returns. Include multiple stochastic samples per history rather than judging one pixelwise match. If sharper or more diverse predicted opponents do not improve reward accuracy or real-game returns, the proposed causal link from stochastic prediction to useful control is weakened, even if the images look more plausible. e03e06e11e13e14e15
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, every original numbered figure and table, the architecture layer tables, and all pages supporting retained method, numerical, training, evaluation and reproduction details. Figures 1 and 2 were checked against their captions, Algorithm 1, Section 4 and Appendix C; the distinction between training and inference latent branches is preserved, and differing architectural widths are disclosed. All six final crops were individually inspected, including the recropped learning loop and architecture. The benchmark crop retains every table column and missing entry; the diagnostic retains axes, legend and error bars. References and related-work prose were read in the complete supplied text, but pages 3 and 11–14 were not included in the visual pass. External videos, code and separate supplements remain outside this reading.
PDF pages inspected for this edition: 1, 2, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Section 1: Introduction (pp. 1–2)
- Section 2: Related Work (pp. 2–3)
- Section 3: Simulated Policy Learning and Algorithm 1 (p. 4)
- Section 4: World Models (pp. 5–6)
- Section 5: Policy Training (p. 6)
- Sections 6.1–6.4: Experiments, sample efficiency, interaction budgets, stochasticity and ablations (pp. 6–10)
- Section 7, acknowledgments and references (pp. 10–14)
- Appendix A: Ablations (pp. 15–18)
- Appendix B: Qualitative Analysis (pp. 16, 19)
- Appendix C: Architecture Details (p. 20)
- Appendix D: Numerical Results, Tables 2–4 (pp. 21–24)
- Appendix E: Baselines Optimization (p. 25)
- Appendix F: Results at Different Numbers of Interactions (pp. 25–28)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting original PDF figures, tables and all pages supporting retained technical claims. Linked videos and code were not inspected; no experiments were reproduced.
- Version scope: the supplied title page identifies arXiv:1903.00374v5, 3 April 2024, with an ICLR 2020 conference header. Only this revision was supplied and reviewed; earlier revisions and the original conference artifact were not compared. The observed title omits the catalog title's hyphen in Model-Based; all fourteen authors agree after normalization of diacritics. This is not evidence that the editions are identical.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block, arXiv margin and abstract
The title is MODEL BASED REINFORCEMENT LEARNING FOR ATARI. Fourteen authors and six affiliations are printed. The artifact is arXiv:1903.00374v5, dated 3 April 2024, under an ICLR 2020 header; the abstract introduces SimPLe and the low-interaction Atari objective.
Go to primary source ↓e02PDF p. 2, Figure 1 and caption; p. 4, Section 3 and Algorithm 1
Real-policy trajectories expand D; supervised learning updates the world model, and reinforcement learning in that model updates the policy. Figure 1 distinguishes self-supervised observation prediction from supervised reward learning.
Go to primary source ↓e03PDF p. 4, Figure 2, caption and Section 3; p. 5, Section 4, Deterministic Model
Four frames reduce partial observability. The architecture encodes pixels, uses skip-connected convolution/deconvolution layers and action multiplication, and predicts next frame plus reward. Section 3 defines the discounted-return expectation. Figure 2 and Section 4 describe architectural variants rather than matching all Appendix C widths.
Go to primary source ↓e04PDF p. 5, Section 4, Loss functions
The loss is max(Loss,C), with C = 10 for L2 and 0.03 for softmax. Correct-pixel probability above approximately 97% stops its gradient. Emphasis on important small objects is the authors' proposed explanation.
Go to primary source ↓e05PDF p. 5, Section 4, Scheduled sampling
Training replaces input frames with previous predictions, increasing the mixing probability linearly to 100% around the middle of the first training-loop iteration.
Go to primary source ↓e06PDF p. 4, Figure 2 caption; pp. 5–6, Section 4, Stochastic Models; p. 20, Appendix C, final sentence
Target-conditioned latents are discretized with bypassed gradients, uniform noise before discretization and dropout afterward. An auxiliary LSTM predicts discrete latents at inference. The detailed model outputs 128 bits in eight-bit chunks. Gaussian latents required game-sensitive KL weights and could mismatch the prior.
Go to primary source ↓e07PDF p. 6, Section 5, Policy Training
PPO uses short rollouts, uniform resets from D and terminal value bootstrapping. The section specifies γ = 0.95, N = 50, fifteen iterations, 45K initial/15K later model-training steps, sixteen parallel agents and 15.2M simulated interactions.
Go to primary source ↓e08PDF p. 6, Section 6 and footnote 2; p. 7, Section 6.1
Preprocessing repeats actions four times and halves image dimensions. Sixteen batches of 6,400 real interactions total 102,400 steps or 409,600 frames. The main suite selects twenty-six games with non-random performance by SimPLe or Rainbow at 100K. Results average five SimPLe runs; real data also supplies PPO updates.
Go to primary source ↓e09PDF p. 25, Appendix E, Baselines Optimization
Rainbow tuning evaluates five agents per configuration on Pong through one million interactions. The listed target_update_period candidates are 50, 100, 1000 and 4000, but the reported best is 8000. PPO uses referenced Baselines defaults.
Go to primary source ↓e10PDF p. 23, Table 3, Pong, Freeway and BankHeist rows; complete headers and game list
Pong: SimPLe 12.8 (17.2), PPO_100k −20.5 (0.6), Rainbow_100k −19.5 (0.2), Rainbow_500k 19.9 (0.4). Freeway: 16.7 (15.7), 8.0 (9.8), 0.1 (0.1), respectively; random 0.0. BankHeist lists SimPLe 34.2 (29.2) versus random 15.0. The appendix table includes games beyond the main twenty-six-game plots.
Go to primary source ↓e11PDF p. 15, Table 1, caption and all rows
Counts of best / at-least-median game scores are deterministic 0/7, deterministic recurrent 3/13, SD 8/16, SD γ = 0.9 1/14, default 10/21, SD 100 steps 0/14, and SD 25 steps 4/19. Each game/configuration score is a five-experiment mean.
Go to primary source ↓e12PDF p. 15, Appendix A, Steps, Gamma, Model-based iterations and Long model training; pp. 17–18, Figures 7–8
The appendix compares model types, N = 25/50/100, discount factors and iteration counts. It describes fivefold longer model training for the best results, with shorter training for the other ablations; longer rollouts are slightly worse and model choice has the larger effect.
Go to primary source ↓e13PDF p. 16, Appendix B, Exploration, Benign errors and Failures on hard games; p. 19, Figures 10–11 and final paragraph
Failures include poor exploration in Freeway, disappearing small bullets in Atlantis/Battle Zone and global transitions in Private Eye. Some nonphysical errors remain compatible with useful policies. Kung Fu Master model frames can contain different numbers of opponents than the original game.
Go to primary source ↓e14PDF p. 9, Section 6.3 and Figure 6; p. 8, Section 6.2 and Figure 5
Four-frame observation histories leave hidden stochasticity, illustrated by unpredictable opponent spawns. Training with sticky actions often yields similar results without retuning. Section 6.2 reports loss of the advantage over PPO at larger data budgets.
Go to primary source ↓e15PDF p. 21, Appendix D, Numerical Results
Evaluation uses softmax(logits(π_i)/T), with T = 0.5 empirically best in most cases. SimPLe configurations have five runs; mean/standard deviation and median/best statistics are reported separately. The passage does not specify evaluation-episode counts.
Go to primary source ↓e16PDF p. 22, Table 2, SD long column, Pong and Freeway rows; p. 24, Table 4, Pong row, SD long median/best columns
SD long means are Pong 12.8 (17.2) and Freeway 16.7 (15.7), matching Table 3. Pong SD long has median 20.7 and best 21.0 across five runs; these are distinct from its mean.
Go to primary source ↓e17PDF p. 2, final Introduction paragraph
The revised discussion states that later tuned model-free methods in van Hasselt et al. and Kielak each outperform SimPLe in thirteen of twenty-six games, with SimPLe better in the other thirteen. It also distinguishes improved SimPLe results from the first preprint.
Go to primary source ↓e18PDF p. 20, Appendix C, opening paragraph, layer tables and final paragraph
The detailed model has about 74M parameters; inference/backpropagation take approximately 0.5/0.7 seconds on NVIDIA Tesla P100 at batches 16/2. The layer table starts with 96 channels and reaches 768; convolution dropout is 0.15. These settings differ from the schematic/example widths and dropout in the main text.
Go to primary source ↓e19PDF p. 10, Section 7, Conclusions and Future Work
The reported method applies model-free learning inside a simulator. Planning, using simulator gradients and passing model representations to the policy are future directions. Limitations include lower final scores, substantial run variance and computation; applications beyond Atari are proposed.
Go to primary source ↓e20PDF p. 6, Section 5; p. 9, Section 6.4; p. 15, Appendix A, Gamma; p. 18, Figure 8b legend
Section 5 specifies γ = 0.95, whereas the ablation text and Figure 8b label 0.99 as default. The supplied paper therefore does not use one consistent default label.
Go to primary source ↓e21PDF p. 10, first paragraph; p. 16, Appendix A, Random starts; p. 18, Figure 9 right panel and caption
Seaquest random-start performance exceeds the no-random-start curve, but the latter also uses 1,000-step rollouts. The horizontal axis is training-loop iteration; rewards are clipped to −1, 0, 1 before the plotted mean-reward comparison. The caption does not define the error-bar statistic.
Go to primary source ↓e22PDF p. 7, code-availability paragraph and footnote 3; p. 15, paragraph below Table 1; p. 25, Appendix E
The authors link Tensor2Tensor and claim experiment instructions are available. Baseline implementation links are provided, without establishing immutable software revisions or installation success.
Go to primary source ↓8.5 Primary sources
MODEL BASED REINFORCEMENT LEARNING FOR ATARI ↗
PDF · 13,544 extracted words
Source fingerprint
905b7f0d614c34187325dec4858420812a7042d38624582a8b737d5d772d1efa