World Models
1. Paper overview
In one sentence: A separately learned visual and recurrent world model makes a small controller effective, but training inside its predictions requires guarding against exploitable simulator errors. e-identitye-frameworke-car-resultse-doom-proceduree-temperature
| At a glance | What to know |
|---|---|
| Research problem | Source description How can visual reinforcement learning use a large predictive network without making reward-based optimization search all its weights? The paper assigns representation learning to a world model and restricts policy optimization to a compact controller. It also asks whether learned dynamics can replace the environment during controller training. The authors present a simplified experimental realization of earlier recurrent world-model/controller ideas. e-framework |
| Core mechanism | Source description A modular Vision–Memory–Controller agent separates visual and temporal representation learning from black-box policy optimization. e-frameworke-car-procedure |
| A key reported result | DoomTakeCover-v0 dream-to-game transfer: 1092 ± 556 Average survival time in game steps, with reported ± spread. Controller trained only in DoomRNN at tau = 1.15, then tested in the original game over 100 random rollouts; episodes capped at 2,100 steps. Table 2: virtual score 918 ± 546; random actual score 210 ± 108; cited Gym leader 820 ± 58. Exceeds the 750-step mean-survival threshold in a game simulator. Appendix A.5 separately reports 959 over 1,024 virtual rollouts; that evaluation differs from the table’s virtual score and the transfer test. e-doom-proceduree-temperaturee-doom-appendix |
| Reading caution | Author claim Controllers can visit states outside the random-policy training distribution and exploit inaccurate dynamics, including suppressing fireballs. Access to M’s hidden state makes exploitation easier; stochastic sampling mitigates rather than eliminates this failure. e-exploitation |
Core contributions
- Source description
A modular Vision–Memory–Controller agent separates visual and temporal representation learning from black-box policy optimization. e-frameworke-car-procedure
- Source description
CarRacing tests recurrent state against single-frame latent features, including a larger vision-only controller. e-car-resultse-car-ablation
- Source description
VizDoom tests latent-simulator policy transfer and uses sampling temperature to examine the tradeoff between realism and exploitability. e-doom-proceduree-temperaturee-exploitation
Figure 8. One action loop connects visual compression, recurrent memory and a small controller. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the environment box and follow the observation into V. The latent z branches toward both M and C; the recurrent state h supplies C with temporal information. C’s action returns to the environment and also enters M, because the next observation depends on what the agent does. The unindexed diagram compresses time: its caption and the page-4 pseudocode specify that C uses the current z and h, executes the action, then updates M to obtain the next state. The blue boundary encloses V and M as the world model; C remains a separate policy optimized for reward. e-frameworke-car-resultse-doom-procedure
What it supports. The architectural economy comes from placing perception and temporal representation inside V and M while keeping reward-driven search confined to C. In CarRacing, the controller can use predictive memory features directly; selecting an action does not require generating candidate future trajectories or searching a rollout tree.
Where the evidence stops. This diagram depicts the original-environment loop. Doom’s virtual training replaces incoming observations with sampled latent states, so V is not repeatedly encoding dream frames. The figure alone does not specify that alternate training protocol.
2. Motivation
2.1 The problem and the proposed response
How can visual reinforcement learning use a large predictive network without making reward-based optimization search all its weights? The paper assigns representation learning to a world model and restricts policy optimization to a compact controller. It also asks whether learned dynamics can replace the environment during controller training. The authors present a simplified experimental realization of earlier recurrent world-model/controller ideas. e-framework
2.2 What this reading follows
A useful world model need not reconstruct every visual detail, but it must preserve the information that control depends on. Ha and Schmidhuber explore that distinction through a VAE, a recurrent mixture-density predictor and a small evolved controller. Read the two experiments separately: CarRacing tests learned features while its controller trains in the original simulator; VizDoom trains its controller in a sampled latent world and then tests transfer. The most revealing result is the temperature sweep, where almost perfect dream survival can coincide with worse-than-random performance in the original game. This edition reads the supplied arXiv v4, dated 9 May 2018. e-identitye-frameworke-car-resultse-doom-proceduree-temperature
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The foundational classical-world-model classification fits the modular dynamics and controller experiments. V, M and C are separate trained components, not one joint future/action predictor. Actions condition M and are chosen by C; there is no inverse-dynamics extraction. The recorded architecture, prediction-paradigm and quadrant remain Not applicable within this catalog, rather than asserting that the paper lacks an architecture. Section 5’s action-predicting extension is proposed, not evaluated. e-frameworke-doom-proceduree-iterative
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate learning a representation from learning what to do
Begin with the random-policy dataset. Its observations teach V to compress frames, while the paired actions and successive latents teach M about transitions. This training does not require solving the control task, and CarRacing’s V and M never receive reward. Once those representations are learned, CMA-ES searches only the small controller’s parameters. At deployment, z tells C about the current frame and h carries information accumulated by the predictive recurrent network. The resulting policy is a direct mapping from features to bounded actions. The interpretation is that expensive representation capacity and a small reward-search space can coexist. The CarRacing ablations support the usefulness of that representation, but do not by themselves establish that next-frame prediction is superior to every alternative way of learning recurrent features. e-frameworke-car-datae-car-proceduree-rnne-controllere-car-resultse-car-ablation
Figure 22. The visual bottleneck determines what information the downstream controller can receive. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward from the 64 × 64 × 3 RGB image. Four stride-two convolutions reduce spatial resolution while expanding the channel count, ending at a 2 × 2 × 256 tensor. Dense branches produce the encoder parameters mu and sigma; the central sampling operation creates z. A dense expansion and four deconvolutions reconstruct the original image size. Italic labels specify activation, channel count and filter size; the final sigmoid bounds reconstructed pixels. Appendix A.1 sets the latent width to 32 for CarRacing and 64 for Doom and trains this network with reconstruction and KL losses before recurrent dynamics learning. e-vaee-discussione-car-procedure
What it supports. The controller receives compressed observations instead of the complete image tensor. This supports compact control, but reconstruction is an imperfect proxy for task relevance: the discussion reports preserved Doom wall textures alongside missing CarRacing road-tile detail. The bottleneck can therefore discard information that a reward-aware representation might retain.
Where the evidence stops. The diagram writes z = mu + sigma N(0,1), treating sigma as a sampling scale; the adjacent prose writes N(mu, sigma I). This scale-versus-covariance notation is unresolved in the PDF and should not be silently normalized into an implementation recipe.
5.2 Replace the Doom environment while preserving its policy interface
The CarRacing dream demonstration deploys a controller already trained in the original environment. Doom reverses that direction. Its recurrent model predicts both the next latent and a death event, which is enough to construct a survival task in latent space. Each controller action becomes an input to the next modeled transition; predicted death ends the episode, and survival duration supplies fitness. The VAE decoder can make this process visible, but image rendering is unnecessary for the controller’s latent-space interaction. After evolution, the original game supplies frames again, V encodes them, and the same controller acts using latent and recurrent features. The policy-transfer claim is therefore stronger than showing plausible generated video, while remaining limited to the evaluated game and the random-rollout data used to learn its simulator. e-car-dreame-doom-proceduree-rnne-doom-appendix
5.3 Treat uncertainty as part of the simulator, then measure transfer
A controller optimized against an imperfect model is rewarded for exploiting whatever that model allows. In Doom, the authors observed policies that suppressed fireballs in the learned world. Low-temperature mixture sampling can also collapse toward trajectories where those threats hardly appear. Raising tau changes the virtual environment and can make such behavior less profitable, but it also moves simulation away from faithful dynamics. Table 2 shows why the balance must be measured: the best dream scores transfer poorly, intermediate temperature improves actual survival, and further temperature increase reduces the actual mean. The reader’s inference is that model realism, policy robustness and controllability are different objectives. The sweep motivates transfer evaluation and new-data collection; it does not establish calibrated uncertainty or prove that stochastic models cannot be exploited. e-exploitatione-temperaturee-iterative
5.4 Training and inference
During training
For each task, collect 10,000 random-policy rollouts and train V before M. V receives one epoch of reconstruction L2 plus KL training. M receives 20 epochs of teacher-forced training; cached encoder means and scales are resampled into latent inputs for each batch. The PDF does not give a complete scalar MDN/death-loss weighting recipe. e-car-datae-doom-proceduree-vaee-rnn
CMA-ES then optimizes C separately, keeping V and M fixed. Population size is 64; candidate fitness averages 16 randomly seeded rollouts. CarRacing uses original-environment rewards, which V and M never receive. Doom optimizes virtual survival. The controllers have 867 and 1,088 parameters respectively. e-car-proceduree-doom-proceduree-es
The proposed iterative loop collects new trajectories and learns observation, reward, action and termination predictions. Curiosity and hierarchical skill absorption are future directions; the reported simple tasks use one collection/training iteration. e-iterative
During inference
In the original environment, encode the current frame, compute the action from current latent and memory, execute it, and update memory using that latent and action. CarRacing selects actions directly without searching hypothetical future rollouts. e-frameworke-car-results
Inside DoomRNN, M samples the next latent instead of receiving rendered frames. Termination occurs when predicted death probability exceeds 50%, with a 2,100-step episode cap. V need not encode dream observations. Deployment restores real-frame encoding and retains C; no original-Doom policy fine-tuning is reported. e-doom-proceduree-rnne-doom-appendix
5.5 Implementation flow
- Compress each observation
V is a convolutional VAE with four convolutional encoder layers and four deconvolutional decoder layers. Its latent dimension is 32 for CarRacing and 64 for Doom; pixels are scaled to [0,1]. The decoder reconstructs images but does not select actions. e-vae
- Learn temporal dynamics
M is an LSTM with a five-component, diagonal-covariance Gaussian-mixture output. It predicts the next latent conditioned on current latent, action and recurrent state. The LSTM has 256 units for CarRacing and 512 for Doom. e-rnne-framework
- Act from learned features
C maps concatenated latent and memory features to actions. Appendix A.3 adds tanh-based action bounding to the main text’s linear map. CarRacing uses the LSTM output state; Doom uses both output and cell states and converts a scalar continuous output into left/stay/right actions. e-frameworke-controller
6. Experiments & results
World Models separates visual compression, recurrent prediction and reward-driven control. A VAE and an MDN-RNN learn from random gameplay; a small controller then uses their representations. CarRacing demonstrates useful predictive features during simulator-based policy training. VizDoom demonstrates policy training inside a learned latent simulator followed by transfer to the original game. Its temperature sweep exposes the central risk: a controller can exploit model errors and obtain excellent imagined returns while failing after transfer.
6.1 Read the original evidence
Figure 24. Candidate fitness and repeated evaluation answer different questions during evolution. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as CMA-ES generations and the vertical axis as average cumulative reward. The blue, orange and green curves summarize the population mean, worst performer and best performer. Each candidate’s fitness averages 16 rollouts, with 64 candidates per generation. The red curve comes from a different calculation: every 25 generations, the best candidate is evaluated on 1,024 random rollouts. Follow that red trace against the dashed 900 line when assessing sustained task performance. A high green value alone can reflect the best candidate under its smaller rollout sample, rather than the broader evaluation used for the red trace. e-ese-car-results
What it supports. The appendix reports that the broader evaluation reaches 900.46 after 1,800 generations. This makes the policy-search effort visible alongside the small controller size. The separate final evaluation gives 906 ± 21 over 100 trials, so the two reported values describe different evaluation samples and should remain distinct.
Where the evidence stops. These curves describe population variation within the reported evolution process. They are not confidence bands across independently repeated training runs. The red evaluation schedule also differs from the final 100-trial benchmark and should not be relabeled as that test.
Table 1. Predictive memory improves the reported CarRacing result beyond the vision-only controls. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the bottom three rows, because they compare the paper’s own controller variants. The V model supplies only z; the next row adds a hidden layer to that vision-only controller; the full world model supplies z together with recurrent memory. Appendix A.4 identifies the added layer as 40 tanh units and the resulting controller as 1,443 parameters, compared with 867 in the full model. Next read the upper rows as historical comparison scores compiled from cited methods and the Gym leaderboard. The full-model headline is measured over 100 random trials on randomly generated tracks. e-car-datae-car-proceduree-car-resultse-car-ablation
What it supports. The full model reports 906 ± 21, compared with 632 ± 251 for a linear vision-only controller and 788 ± 141 after adding its hidden layer. The full model crosses the stated mean-reward solving criterion of 900. Increasing the vision-only controller’s size alone does not recover that result.
Where the evidence stops. The ± statistic is not explicitly defined in the supplied paper. Literature rows differ in preprocessing and are not controlled reruns. The ablation also does not isolate predictive training from every possible benefit of recurrent memory.
Figure 29. Successful mean transfer still includes a broad distribution of individual outcomes. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read survival time along the horizontal axis and trial frequency along the vertical axis. This histogram concerns the original DoomTakeCover-v0 environment, not latent-world training scores. Appendix A.5 identifies the controller as trained in DoomRNN at tau = 1.15 and reports 100 transfer trials. Outcomes occupy a broad span of survival times, with a conspicuous bar at the right boundary. Section 4.1 defines that boundary as the 2,100-step episode cap. Compare the shape with the reported average and with the task’s mean-survival threshold; neither a single long episode nor a short one determines whether the average criterion is met. e-doom-proceduree-doom-appendixe-temperature
What it supports. The reported mean of 1092 ± 556 exceeds the 750-step criterion after policy training inside the learned environment. The histogram makes the variability visible: crossing an average-based threshold does not imply uniformly long survival across trials. This is evidence of policy transfer between game simulators, with uneven episode outcomes.
Where the evidence stops. The rightmost outcomes are limited by the episode cap; the figure cannot reveal how long those runs would otherwise continue. It summarizes trials for a selected controller, not the reliability of independently repeating the complete training procedure.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CarRacing-v0 control Randomly generated tracks; final controller evaluated over 100 random trials in the original simulator. | 906 ± 21 Average cumulative reward, with the paper’s reported ± spread | Vision-only linear: 632 ± 251; vision-only with hidden layer: 788 ± 141; cited Gym leader: 838 ± 11. Exceeds the stated 900-average solving criterion. Within-paper ablations support recurrent features; literature rows are not a matched-compute rerun. The source does not explicitly define the ± statistic. e-car-datae-car-resultse-car-ablation |
| DoomTakeCover-v0 dream-to-game transfer Controller trained only in DoomRNN at tau = 1.15, then tested in the original game over 100 random rollouts; episodes capped at 2,100 steps. | 1092 ± 556 Average survival time in game steps, with reported ± spread | Table 2: virtual score 918 ± 546; random actual score 210 ± 108; cited Gym leader 820 ± 58. Exceeds the 750-step mean-survival threshold in a game simulator. Appendix A.5 separately reports 959 over 1,024 virtual rollouts; that evaluation differs from the table’s virtual score and the transfer test. e-doom-proceduree-temperaturee-doom-appendix |
| Doom temperature sensitivity Separate controllers trained at different virtual sampling temperatures; actual-environment evaluation over 100 random rollouts. | tau = 0.10: 2086 ± 140 virtual, 193 ± 58 actual; tau = 1.30: 732 ± 269 virtual, 753 ± 139 actual. Virtual versus actual mean survival steps, with reported ± spread | tau = 1.00 gives 868 ± 511 actual; tau = 1.15 gives 1092 ± 556 actual. Near-perfect dream survival can transfer worse than random. Increasing temperature helps only within a range; the highest tested value sacrifices mean return while reducing reported spread. e-temperaturee-exploitation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Dream return can be a misleading objective when the learned simulator is exploitable. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare each row horizontally before comparing temperatures vertically. The virtual column evaluates behavior in the learned simulator; the actual column reports transfer to the original game after training at the listed temperature tau. At 0.10 and 0.50, excellent virtual returns coexist with poor actual survival. Moving to 1.00 and then 1.15 sharply improves transfer, while 1.30 lowers the actual mean and the reported spread. The bottom rows provide actual-environment baselines rather than additional temperature conditions. The surrounding text specifies 100 random actual-environment rollouts; Appendix A.5 separately describes a 1,024-rollout virtual training evaluation, which should not replace this table’s virtual entries. e-temperaturee-exploitatione-doom-appendix
What it supports. At tau = 0.10, the model gives 2086 ± 140 virtual steps but only 193 ± 58 actual steps, below the random-policy mean of 210. At tau = 1.15, actual survival reaches 1092 ± 556. The result supports checking transfer directly instead of trusting a controller’s dream score.
Where the evidence stops. Temperature changes the simulator’s sampling distribution, not the training dataset’s coverage. The sweep shows an empirical tradeoff, not a guarantee against exploitation. No separate temperature-selection split or repeated-training uncertainty is reported, so the best row is not an unbiased estimate of hyperparameter selection.
7. Analysis & limitations
7.1 What the evidence leaves open
Controllers can visit states outside the random-policy training distribution and exploit inaccurate dynamics, including suppressing fireballs. Access to M’s hidden state makes exploitation easier; stochastic sampling mitigates rather than eliminates this failure. e-exploitation
Standalone reconstruction may preserve irrelevant Doom brick textures while losing reward-relevant CarRacing road tiles. The discussion identifies limited recurrent capacity and catastrophic forgetting, and leaves hierarchical reasoning and unified controller/model extensions untested. e-discussion
Two game tasks do not establish physical-robot transfer or broad task generalization. Trial spreads are not uncertainty across independently trained models; training-seed replication and formal significance tests are not supplied. e-car-resultse-temperaturee-doom-appendix
7.2 Questions for discussion
- Would equally sized recurrent features trained without next-latent prediction retain the CarRacing benefit?
- How should temperature be selected without repeatedly consulting the final transfer benchmark?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction requires the original environment variants, random-rollout data, VAE/MDN training and CMA-ES protocol. Appendix A.4 uses a 64-core machine and reevaluates the best CarRacing candidate every 25 generations on 1,024 rollouts; 900.46 is reached after 1,800 generations. This differs from the final 100-trial score. e-car-datae-doom-proceduree-es
The PDF leaves optimizer rates, batch sizes, exact software versions and complete seed lists unspecified. Figure 22 uses sigma as a sampling scale, whereas Appendix A.1 places it in the Gaussian covariance argument; resolve that notation before implementing. The single-GPU timing statement does not identify the GPU model. e-vaee-rnne-ese-car-data
Proposed checks: compare predictive memory with shuffled and untrained recurrent features under equal controller budgets, and repeat the temperature sweep with model selection separated from final transfer testing. e-car-ablatione-temperature
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does predictive training add useful memory beyond extra features?
Reader-proposed experiment: freeze one CarRacing VAE and compare controllers receiving z alone, z plus trained M state, z plus an equally sized randomly initialized recurrent state, and z plus temporally shuffled trained states. Match controller parameter budgets and CMA-ES rollout budgets, use the same evaluation-track seeds, and repeat training independently. Report mean return and lower-tail returns on held-out tracks. If untrained or shuffled memory matches trained M, the claim that learned temporal prediction supplies the useful control information is weakened. A consistent trained-state advantage would support that mechanism beyond the published vision-only comparison. e-frameworke-rnne-car-resultse-car-ablatione-es
Check 2: Does temperature improve transfer after separating selection from testing?
Reader-proposed experiment: freeze a Doom V/M pair and train replicated controllers at the five published temperatures with equal CMA-ES budgets. Use one set of original-game seeds for temperature selection and a disjoint set for final transfer evaluation. Keep death-probability threshold, episode cap and action discretization identical. Measure virtual and actual survival, early-failure frequency, and the occurrence of suppressed-fireball trajectories. If an intermediate temperature repeatedly reduces exploitation and improves held-out transfer, the robustness account is supported. If its advantage vanishes across training seeds or after selection is separated from testing, the single sweep overstates reliability. e-doom-proceduree-exploitatione-temperaturee-rnne-controllere-ese-doom-appendix
8.3 Reading coverage
Visual audit: Actually inspected PDF pages 1–15, including the title/author/version block, all numbered figures on these pages, both parameter-count tables, Tables 1–2, and Appendix A.1–A.5. Inspected all six final PNG crops. Figure 8’s unindexed arrows were checked against its caption, Eq. (1) and the page-4 rollout order; Figure 22’s sampling notation was checked against Appendix A.1 and its ambiguity is disclosed. Every page supporting retained method, numerical, training, evaluation and proposed-reproduction claims is included. Reference pages 16–21 were read in the supplied text chunks but were not visually inspected. Static PDF images cannot reproduce the linked interactive demonstrations; separate supplements remain unverified.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2.4 (PDF pp. 1–4)
- Sections 3–3.4: Car Racing Experiment (PDF pp. 4–6)
- Sections 4–4.5: VizDoom Experiment (PDF pp. 6–10)
- Section 5: Iterative Training Procedure (PDF pp. 10–11)
- Sections 6–7: Related Work and Discussion (PDF pp. 11–13)
- Acknowledgements and Appendix A.1–A.5 (PDF pp. 13–15)
- References (PDF pp. 16–21)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The supplied title page identifies arXiv:1803.10122v4, 9 May 2018, with the catalog title and both catalog authors. The catalog submission date is 27 March 2018; earlier revisions were not supplied, so differences between revisions were not assessed.
- All eight supplied text chunks were read. PDF pages 1–15 and all six final crops were visually inspected; reference pages 16–21 were read as text.
- The PDF contains static versions of interactive demonstrations. The linked interactive article, external references and code were not opened; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author affiliations and arXiv margin stamp
World Models; David Ha (Google Brain) and Jürgen Schmidhuber (NNAISENSE; Swiss AI Lab, IDSIA (USI & SUPSI)); arXiv:1803.10122v4, 9 May 2018.
Go to primary source ↓e-frameworkPDF pp. 2–4, Sections 2–2.4, Figures 4–8, Eq. (1) and rollout pseudocode
V compresses observations, M predicts action-conditioned latent distributions, and separately trained C selects actions. Pseudocode executes an action before updating memory for the next step.
Go to primary source ↓e-car-dataPDF p. 4, Section 3.1 and footnotes 2–3
CarRacing uses steering, acceleration and brake on random tracks; training data contain 10,000 random rollouts. The solving criterion averages 900 over 100 trials. The authors report under one hour per V/M model on one unspecified GPU.
Go to primary source ↓e-car-procedurePDF p. 5, Section 3.2, opening paragraph and unnumbered parameter table
Car latent dimension is 32; V/M do not receive reward; CMA-ES optimizes an 867-parameter controller after VAE and MDN-RNN training.
Go to primary source ↓e-car-resultsPDF pp. 5–6, Section 3.3, Table 1 (all method rows)
Full world model scores 906 ± 21 over 100 random trials; V-only 632 ± 251; V with hidden layer 788 ± 141; Gym leader 838 ± 11. The policy uses recurrent features without online future-rollout planning.
Go to primary source ↓e-car-dreamPDF p. 6, Section 3.4 and Figure 13
The already-trained CarRacing controller is demonstrated in a sampled latent environment rendered with the VAE decoder; this is not the reported CarRacing controller-training protocol.
Go to primary source ↓e-doom-procedurePDF p. 7, Sections 4.1–4.3, procedure steps 1–6 and parameter table
Doom uses 10,000 random rollouts, 64-dimensional latents, next-latent/death prediction and a 1,088-parameter controller trained for virtual survival; original-game evaluation has a 750-step mean criterion and 2,100-step cap.
Go to primary source ↓e-doom-transferPDF p. 8, Sections 4.3–4.4, Figures 16–17
Increased sampling temperature makes the virtual world less predictable; a dream-trained controller transfers to original VizDoom despite imperfect reconstruction of monsters.
Go to primary source ↓e-exploitationPDF pp. 8–9, Section 4.5 and Figure 18
The authors report controllers suppressing hallucinated fireballs and discuss hidden-state access, off-distribution trajectories, discrete mixture modes and low-temperature collapse.
Go to primary source ↓e-temperaturePDF p. 10, Table 2 (temperature rows and baselines) and adjacent Section 4.5 text
Virtual/actual scores: 0.10, 2086 ± 140 / 193 ± 58; 0.50, 2060 ± 277 / 196 ± 50; 1.00, 1145 ± 690 / 868 ± 511; 1.15, 918 ± 546 / 1092 ± 556; 1.30, 732 ± 269 / 753 ± 139. Random actual: 210 ± 108; Gym leader: 820 ± 58. Actual scores use 100 random rollouts.
Go to primary source ↓e-iterativePDF pp. 10–11, Section 5, four-step iterative procedure
The proposed loop adds new actual-environment data and predicts observations, rewards, actions and termination. One iteration sufficed for the simple reported tasks; curiosity and hierarchical learning are proposed extensions.
Go to primary source ↓e-discussionPDF pp. 12–13, Section 7
Standalone VAE training can preserve irrelevant features and lose relevant ones; recurrent capacity and forgetting constrain richer worlds. Hierarchical reasoning and One Big Net extensions are left for future experiments.
Go to primary source ↓e-vaePDF p. 13, Appendix A.1 and Figure 22
RGB inputs are 64 × 64 × 3; four stride-two convolutions and four deconvolutions surround 32/64-dimensional latents. One-epoch training uses L2 reconstruction and KL losses. Figure 22 writes z = mu + sigma N(0,1), while prose writes N(mu,sigma I), leaving scale/covariance notation ambiguous.
Go to primary source ↓e-rnnPDF p. 14, Appendix A.2 and Figure 23
MDN-LSTMs have 256/512 hidden units, five factored Gaussian components, 20 training epochs and teacher-forced inputs resampled from cached encoder parameters. Doom declares death when its predicted probability exceeds 50%.
Go to primary source ↓e-controllerPDF p. 14, Appendix A.3
Actions are bounded with tanh; Doom discretizes a continuous range into thirds. CarRacing uses LSTM output h, whereas Doom supplies both cell c and output h.
Go to primary source ↓e-esPDF p. 14, Appendix A.4 and Figure 24
CMA-ES population 64; each fitness averages 16 rollouts. Every 25 generations the best candidate is evaluated on 1,024 rollouts on a 64-core machine. At 1,800 generations the evaluation average reaches 900.46; graph distinguishes population statistics and reevaluated best-agent reward.
Go to primary source ↓e-car-ablationPDF p. 15, Appendix A.4, Figures 25–27 and intervening paragraph
Histograms show full, z-only and z-only-with-hidden-layer CarRacing evaluations. The larger z-only controller uses 40 tanh hidden units and 1,443 parameters; scores are 906 ± 21, 632 ± 251 and 788 ± 141.
Go to primary source ↓e-doom-appendixPDF p. 15, Appendix A.5, Figures 28–29
Policy training used DoomRNN rather than actual VizDoom. At temperature 1.15 the best agent achieved 959 over 1,024 virtual rollouts and 1092 ± 556 over 100 original-game trials; Figure 29 plots survival frequencies.
Go to primary source ↓8.5 Primary sources
World Models ↗
PDF · 13,090 extracted words
Source fingerprint
b0c1e30aab53efd28ddf61d661f680150918d4d03b77bae62bc52d62dbd76cce