Learning Latent Dynamics for Planning from Pixels
1. Paper overview
In one sentence: PlaNet turns pixel histories into a compact stochastic dynamical model and uses online action search to gain data efficiency, while retaining finite-horizon planning costs and task-dependent performance. e01e02e03e12e14
| At a glance | What to know |
|---|---|
| Research problem | Source description A single camera image can hide velocity, occluded objects, or other state needed for control. PlaNet must learn useful dynamics from interaction and predict rewards far enough ahead for planning, while avoiding the cost of rendering every imagined future. The benchmark includes sparse rewards, contacts, and partial observability. e02e07 |
| Core mechanism | |
| A key reported result | Cheetah Run from pixels: 662 Mean final episode return; higher is better. DeepMind control suite; PlaNet trained for 1000 episodes from camera images. Table 1 reports mean final performance over 5 seeds and 10 trajectories. D4PG: 524 after 100,000 pixel-input episodes; A3C: 214 after 100,000 proprioceptive-input episodes; CEM with true simulator: 656. The paper reports a 26% relative improvement over D4PG. Budgets and modalities differ; the finite-search simulator baseline is only an estimated reference, not a mathematical upper bound. e07e08 |
| Reading caution | Reader analysis Evidence is from simulated camera-based control. Physical deployment and robustness to substantially more varied imagery are untested here. Fixed action repeat and a finite reward horizon remain design constraints; temporal abstraction and value-function extensions are future work. e07e18 |
Core contributions
Figure 2. Memory and stochastic state play complementary roles in PlaNet's forward model. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels from left to right, then follow time horizontally within each. Squares denote deterministic variables and circles stochastic variables; solid arrows describe generation, while dashed arrows describe inference. In panel (c), the previous memory h, sampled state s, and action a feed the next memory state. That memory parameterizes a stochastic state, and both states support observation and reward prediction. The first two observations are available; the third is imagined. Equation (4) and its encoder paragraph specify q(s_t | h_t, o_t), so the combined o,r drawing should not be read as specifying reward input to the encoder. e03e04e09e12
What it supports. The figure explains how a model can retain history without forcing all uncertainty into a deterministic trajectory. The matching model comparison supports the usefulness of this design for the six tested control tasks. During action search, latent transitions and reward prediction are used without decoding images.
Where the evidence stops. This is a dependency diagram, not an action-generating policy. Actions condition transitions and are chosen by CEM outside the model. The figure alone cannot establish calibrated uncertainty or the universal necessity of either state component.
2. Motivation
2.1 The problem and the proposed response
A single camera image can hide velocity, occluded objects, or other state needed for control. PlaNet must learn useful dynamics from interaction and predict rewards far enough ahead for planning, while avoiding the cost of rendering every imagined future. The benchmark includes sparse rewards, contacts, and partial observability. e02e07
2.2 What this reading follows
Imagine controlling a walker when each observation is only a camera image. A frame does not directly reveal velocity or hidden state, and predicting a plausible picture is not enough to choose a useful action. PlaNet learns a recurrent latent model that carries memory, predicts uncertain futures, and estimates rewards. A search procedure then tests candidate actions inside that model before executing one action and observing again. This reading follows the supplied June 2019 revision from architecture to evidence: the combined recurrent/stochastic state matters, while the separately proposed latent-overshooting objective is not part of the final RSSM agent. e01e02e03e12e14
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational world-model/model-based-RL category fits the architecture. Actions are external candidates conditioning a learned forward model and selected by CEM; they are neither jointly generated with future observations nor recovered by inverse dynamics. The recorded Not applicable architecture/paradigm/quadrant fields are therefore appropriate for this catalog's WAM taxonomy. A shared RSSM does not by itself establish a One Model action-generation architecture. e03e12
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 First infer a state that can survive missing observations
PlaNet treats the image stream as partially observed. A useful internal state must retain information from preceding frames and actions, because the current picture alone need not identify the underlying dynamics. The RSSM separates this responsibility into deterministic memory h_t and stochastic state s_t. The recurrent update carries preceding memory, state, and action forward; the encoder then uses the current image to form an informed posterior over s_t. Image and reward reconstruction teach this representation what information to preserve, while the KL term trains the prior to anticipate the posterior. All observation information must pass through the sampled state, avoiding a deterministic shortcut that could reconstruct images without training useful stochastic dynamics. The small simulator-state predictors in Appendix I are later diagnostic probes, not privileged state inputs to the control agent. e03e04e16
Figure 3. Longer predictions can receive training signals without decoding every imagined image. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The notation s_i|j means a state at time i conditioned on observations through time j. In panel (a), reconstruction uses informed posterior states, and wavy KL links train one-step predictions. Panel (b) adds long transitions whose predicted states are decoded into observations, creating more image-level losses. Panel (c) instead attaches a wavy link between a multi-step prediction and the informed state at the same destination time. Follow the chain from s_1|1 through s_2|1 to s_3|1, then the KL link to its posterior target. The caption and Equation (7) identify these links as distribution-matching losses rather than environment transitions. e05e14
What it supports. Latent overshooting addresses the mismatch between training with frequent observations and planning without future observations. It can teach consistency over longer prediction distances while avoiding additional image-decoder passes. This is a training contribution with architecture-dependent empirical benefits, not an extra computation required during PlaNet action selection.
Where the evidence stops. The drawing does not mark stopped gradients: Section 4 specifies stopping posterior gradients for distances greater than one. Appendix D reports slight RSSM degradation, so the final agent's performance should not be attributed to this optional objective.
5.2 Then turn the learned predictor into a feedback controller
At a decision point, PlaNet has a filtered state belief, not a precomputed action policy. CEM initializes a Gaussian distribution over candidate action sequences, rolls sampled sequences through the latent dynamics, and scores them by predicted rewards. It repeatedly concentrates the action distribution around the elite sequences. Only the first action mean is sent to the environment; after the task-specific action repeat, a fresh image updates the belief and triggers a new search. This feedback limits how long the controller must trust an uncorrected prediction. Model fitting and interaction alternate during training, so improved plans also change the experience available for subsequent learning. The decoder is essential to the stated training objective but unnecessary for scoring plans, and the agent uses no learned value function to cover rewards beyond its planning horizon. e02e04e11e12e18
5.3 Finally separate a useful training idea from an established control gain
The standard objective trains the stochastic transition through one-step prior-to-posterior comparisons, whereas planning repeatedly applies the model without future observations. Latent overshooting addresses this mismatch by attaching losses to longer imagined trajectories, in latent space rather than image space. That motivation is plausible, but Figure 8 makes its practical value conditional: DRNN can benefit substantially while RSSM already works well and slightly worsens with the extra objective. Reader interpretation: the paper establishes a strong combined agent and a useful research direction, not a rule that every model needs multi-step losses. The same caution applies to the search ablation, where iterative CEM receives more candidate evaluations than random shooting. Reproduction should therefore distinguish predictive consistency, equal-budget action search, and achieved closed-loop return rather than treating them as interchangeable measures. e04e05e10e11e14
5.4 Training and inference
During training
Model fitting alternates with data collection: begin with five random episodes, then add one planned episode every 100 updates. Randomly sampled batches contain 50 chunks of length 50. Collection adds Gaussian action noise, printed as Normal(0, 0.3). e02e11
Jointly train the encoder, transition, image decoder, and reward predictor using reconstruction likelihoods and posterior-to-prior KL regularization. Appendix A specifies a 200-unit GRU, 30-dimensional stochastic state, 5-bit images, Adam learning rate 10⁻³, epsilon 10⁻⁴, gradient clipping norm 1000, and three free nats. The final agent omits overshooting and the earlier fixed global prior. e03e04e11
Optional overshooting averages KL penalties across prediction distances up to D, weighted by β_d. For distances greater than one, posterior gradients are stopped so imagined priors move toward informed targets. The relationship to the ordinary one-step likelihood is presented as a conjecture, not an independently established guarantee. e05e06
During inference
Default CEM settings are horizon H = 12, I = 10 optimization rounds, J = 1000 candidate sequences per round, and K = 100 elites. Action repeat is 8 for cartpole, 4 for reacher/cheetah/cup, and 2 for finger/walker. These repeats determine how planning steps map to simulator steps. e11e12
5.5 Implementation flow
- Filter the observation history
The recurrent state summarizes preceding latent states and actions. A convolutional encoder combines the current image with this deterministic state to infer a diagonal-Gaussian posterior. Observation information passes through stochastic sampling, preventing a deterministic reconstruction shortcut. e03e04
- Imagine and score
Roll candidate action sequences forward through latent transitions. Sum predicted mean rewards along one sampled state trajectory per candidate. The observation decoder supplies a training signal but is unused during planning; there is no policy or value network. e12
- Execute and correct
CEM repeatedly refits a diagonal Gaussian to elite action sequences. Execute only its current action mean, repeat that action for the task-specific interval, receive a new image, and replan. The action-sequence distribution is reset at every decision. e02e12
6. Experiments & results
PlaNet learns an action-conditioned latent dynamics model from images and rewards, then searches that model online to choose continuous actions. Its recurrent state-space model combines deterministic memory with stochastic latent states. The experiments establish sample-efficient simulated control, with uneven gains across tasks. Latent overshooting is a separate proposed training objective; the final RSSM agent omits it.
6.1 Read the original evidence
Table 1. PlaNet's data efficiency accompanies an uneven final-performance comparison. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with Modality and Episodes before reading task scores. PlaNet and D4PG use pixels, but their reported training budgets are 1000 and 100,000 episodes respectively; A3C receives proprioceptive state. The task columns report mean final returns, with the caption specifying five seeds and ten trajectories. Read the last row separately: these efficiency factors were estimated from learning curves by comparing when performance levels were reached. They are not simple ratios of the episode totals printed above. The CEM + true simulator row removes learned-dynamics error but still uses finite search, so it is a practical reference rather than a guaranteed optimum. e07e08e12
What it supports. On Cheetah Run, PlaNet scores 662 versus D4PG's 524, corresponding to the paper's reported 26% relative gain. Finger Spin reverses the comparison: PlaNet scores 700 versus 985. Across the remaining tasks, the table supports strong sample efficiency without showing uniformly superior final performance.
Where the evidence stops. Numerical uncertainty is absent from this table, and the baseline results are imported from the cited control-suite study. Different modalities and training budgets prevent interpreting these rows as a controlled comparison at equal data and compute.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Cheetah Run from pixels DeepMind control suite; PlaNet trained for 1000 episodes from camera images. Table 1 reports mean final performance over 5 seeds and 10 trajectories. | 662 Mean final episode return; higher is better | D4PG: 524 after 100,000 pixel-input episodes; A3C: 214 after 100,000 proprioceptive-input episodes; CEM with true simulator: 656. The paper reports a 26% relative improvement over D4PG. Budgets and modalities differ; the finite-search simulator baseline is only an estimated reference, not a mathematical upper bound. e07e08 |
| Other five continuous-control tasks from pixels Table 1; same PlaNet and model-free training budgets as the Cheetah comparison. Task order: Cartpole Swing Up, Reacher Easy, Finger Spin, Cup Catch, Walker Walk. | PlaNet: 821, 832, 700, 930, 951. Mean final episode return | D4PG: 862, 967, 985, 980, 968. A3C: 558, 285, 129, 105, 311. True-simulator CEM: 850, 964, 825, 993, 994. PlaNet exceeds A3C but remains below D4PG on these final means, especially Finger Spin. Table 1 gives no numerical uncertainty. Its efficiency factors are estimates from learning curves, not the ratio of final training budgets. e08 |
| Model, planner, and collection ablations Figures 4–5; six tasks; medians and 5th–95th percentiles over 5 seeds and 10 trajectories. | Qualitative: RSSM outperforms the tested GRU/SSM designs; planned collection and iterative CEM improve learning. Test-return learning curves | Random collection replaces the data-gathering policy; random shooting selects from 1000 sequences without refinement. These are agent-level comparisons. Collection changes the training distribution, and random shooting is not matched to CEM's total candidate budget. e09e10e11 |
| Latent overshooting ablation Appendix D, Figure 8; RSSM and DRNN with/without overshooting; 5 seeds and 10 trajectories. | Qualitative: substantial improvements for DRNN in some tasks, but slightly reduced RSSM performance overall as described by the authors. Median test-return curves with 5th–95th percentiles | DRNN uses separate encoder and decoder RNNs with a stochastic state sequence between them. The result separates the proposed regularizer from the mechanism needed by the final PlaNet agent; it does not establish universal gains from multi-step losses. e14 |
| One agent across six tasks Section 5 and Appendix C; padded action spaces, no task label, one episode per task per collection round. | Qualitative: the shared agent learns all tasks more slowly than separate agents. Test-return learning curves against episodes per task | Figure 6 states 5 seeds; Figure 7 states 4 seeds. Their statistics also differ: mean/standard deviation versus median/percentiles. This is joint training on observed domains, not zero-shot transfer; the seed-count discrepancy is unresolved. e13 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. The tested RSSM gives stronger and generally more stable control than either single-path model. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to compare the blue RSSM, green stochastic SSM, and red deterministic GRU within each task. Horizontal position is collected episodes; vertical position is test return. The curves summarize medians, and shaded regions span the fifth to ninety-fifth percentiles over five seeds and ten trajectories. Read both the center and the spread: sparse-reward Reacher and Cup show especially broad bands, so a high central curve need not mean uniformly successful runs. The dashed model-free lines are final references after much larger training budgets, not trajectories obtained alongside PlaNet under the same protocol. e08e09e15
What it supports. Combining memory with stochastic states produces the most effective model among these tested variants. The SSM can learn some tasks but is often slower or less reliable; the deterministic GRU performs poorly. These results motivate the RSSM architecture while leaving its precise causal advantages open to more controlled investigation.
Where the evidence stops. The Reacher A3C dashed reference does not visually align with Table 1's 285; both are retained as printed. Architecture changes and activation sensitivity also limit a universal conclusion that stochasticity alone explains the performance gap.
Figure 5. Both where the agent gathers data and how it searches that model affect control. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the blue full-agent curve as the reference. Green changes the behavior used to collect training episodes to random actions; red changes the action optimizer to choosing among 1000 sampled sequences without iterative refinement. These are different interventions: one changes the model's experience, while the other changes how candidate controls are searched. The x-axis counts collected episodes and the y-axis shows test return. As in Figure 4, lines are medians and bands are fifth-to-ninety-fifth percentiles over five seeds and ten trajectories. Compare the Walker and Cartpole panels first, then inspect sparse-reward tasks where variability complicates the picture. e10e11e12
What it supports. The authors find benefits from planned collection and iterative search across the tasks, with particularly visible gaps on Walker and Cartpole. A capable predictive model is therefore only part of this agent: the data-collection loop and the procedure using its predictions materially affect the observed control results.
Where the evidence stops. Random shooting evaluates 1000 candidates once; default CEM evaluates 1000 per iteration for ten iterations. This does not isolate refinement at equal search cost. Random collection also changes the training distribution, so it is not a fixed-data model comparison.
Figure 8. The optional multi-step loss helps a different architecture more than it helps RSSM. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare within each architecture before comparing across architectures. Blue is RSSM and green adds overshooting; red is DRNN and purple adds overshooting. DRNN is described here as two recurrent networks, one encoder and one decoder, with a stochastic state sequence between them. Read horizontal distance as collected episodes and vertical height as test return. The caption reports medians and fifth-to-ninety-fifth percentiles over five seeds and ten trajectories. Reacher and Cup illustrate potentially large DRNN gains, but wide bands show variability. Blue versus green tells a separate story: RSSM already learns effectively with the standard objective. e14e11e05
What it supports. The authors summarize overshooting as substantially improving DRNN while slightly reducing RSSM performance. This qualifies the abstract's emphasis on the objective: the final PlaNet configuration does not rely on it. The architecture and its training regularizer are distinct contributions with different levels of necessity in these experiments.
Where the evidence stops. No numerical ablation table, significance test, or complete overshooting distance/weight configuration accompanies these curves. Treat the reported effect as architecture-dependent evidence, not proof that longer-horizon regularization always improves planning.
7. Analysis & limitations
7.1 What the evidence leaves open
Evidence is from simulated camera-based control. Physical deployment and robustness to substantially more varied imagery are untested here. Fixed action repeat and a finite reward horizon remain design constraints; temporal abstraction and value-function extensions are future work. e07e18
Video examples and frozen-model state probes support useful prediction qualitatively. They do not provide a numerical pixel-accuracy benchmark or establish robot execution. Figure 11's probes require simulator targets and are separate from the control agent. e16
Architectural ablations do not prove stochasticity is universally necessary. Appendix E shows that changing ReLU to ELU improves the SSM, while RSSM is comparatively robust to that choice. e09e15
7.2 Questions for discussion
- Would multi-step prediction improvements survive a fixed-data comparison and translate into better closed-loop return?
- How much of CEM's advantage remains after matching model evaluations and decision latency?
8. Reproducibility audit
8.1 Requirements and known gaps
The authors report 10–20 training hours on one Nvidia V100 and use TensorFlow Probability. Exact software versions are not stated. Appendix A delegates convolutional/deconvolutional architecture details to prior work, so the supplied paper alone is not a complete implementation specification. e11e20
Algorithm 2 prints its Gaussian scale update as an absolute-deviation sum divided by K−1, and uses inclusive horizon indexing. Verify these conventions explicitly before implementation; do not silently replace the printed update with a standard variance estimator. e12
Reader-proposed checks: compare RSSM/DRNN with and without overshooting under a shared replay dataset, then compare CEM and random shooting at matched candidate budgets. Appendix J's true-simulator sweep motivates separating search limitations from learned-model error. e14e10e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does overshooting help prediction and control under the same data?
Reader-proposed experiment: use one fixed collection of Cheetah and Cup episodes to train RSSM and DRNN, each with and without overshooting, with matched update counts and declared distance/weight settings. Keep action repeats and downstream CEM settings identical. Evaluate held-out multi-step reward prediction and then closed-loop return across several seeds; publish distributions rather than only best runs. The fixed replay set controls the feedback between collection and model quality. An architecture-specific gain for DRNN with little RSSM benefit would support Figure 8's distinction. If prediction improves but return does not, that would falsify the stronger claim that this predictive regularizer necessarily helps planning. e05e11e14
Check 2: Does CEM refinement still help when candidate cost is matched?
Reader-proposed experiment: freeze a trained Cheetah model and compare CEM with random shooting at the same horizon, total candidate count, initial-state set, and action repeat. Match the paper's ten rounds of 1000 candidates with a single 10,000-candidate random-shooting pass; report both return and decision latency. Repeat the comparison with the true simulator, following Appendix J's separation of search settings from model learning. Declare the chosen interpretation of Algorithm 2's printed scale update. If CEM's advantage disappears at equal cost, the original comparison cannot attribute its improvement solely to iterative refinement. A difference between learned and true dynamics would implicate interaction with model error. e10e11e12e17
8.3 Reading coverage
Visual audit: Visually inspected the title/author/affiliation block; both algorithms; method equations and training rules; all twelve figures and Table 1; and all appendix evidence pages. Six original crops were inspected individually after extraction, retaining diagram labels, curve legends, axes, table headers, and the efficiency row. The table was rendered at 280 DPI and other crops at 200 DPI. Figure 2 was checked against Equation (4) and the encoder definition; Figure 3 was checked against its caption, Equation (7), and the stated posterior stop-gradient rule. The Reacher A3C reference lines and Table 1 differ visually; exact tabulated values were retained. Multi-task seed counts differ between captions, and Algorithm 2's scale update is preserved as printed. Reference pages 9–11 were read in text but not visually inspected. No external videos, code, earlier revision, or separate supplement was reviewed.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 12, 13, 14, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Latent Space Planning; Algorithm 1
- 3. Recurrent State Space Model
- 4. Latent Overshooting
- 5. Experiments
- 6. Related Work
- 7. Discussion and acknowledgements
- References
- A. Hyper Parameters
- B. Planning Algorithm; Algorithm 2
- C. Multi-Task Learning
- D. Latent Overshooting
- E. Activation Function
- F. Bound Derivations
- G. Additional Related Work
- H. Video Predictions
- I. State Diagnostics
- J. Planning Parameters
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by inspecting the supplied PDF: all twelve figures, Table 1, both algorithms, and the method and appendix evidence pages were visually reviewed. References on pages 9–11 were read as text.
- Identity/version scope: the supplied title page identifies arXiv:1811.04551v5, 4 June 2019, with ICML 2019 proceedings information. The title and seven-author list match the catalog; its BibTeX expands Timothy Lillicrap to Timothy P. Lillicrap. Earlier revisions and the separate venue artifact were not supplied for comparison.
- Only the supplied twenty-page PDF was reviewed. Linked code and videos were not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author block, affiliation footnote, arXiv margin and proceedings footer
Observed title matches the catalog. Credits: Danijar Hafner; Timothy Lillicrap; Ian Fischer; Ruben Villegas; David Ha; Honglak Lee; James Davidson. Affiliations: Google Brain, University of Toronto, DeepMind, Google Research, University of Michigan. Version: arXiv:1811.04551v5, 4 June 2019; ICML/PMLR 97, 2019.
Go to primary source ↓e02PDF pp. 2–3, Figure 1, Section 2, Eq. (1), Algorithm 1 and Experience collection
Image-based POMDP with continuous actions and scalar rewards; six domains illustrate contacts, sparse rewards and partial observability. Model fitting alternates with planned collection; repeated actions accumulate rewards and retain the last image.
Go to primary source ↓e03PDF p. 4, Figure 2, Deterministic path, Eq. (4) and following encoder paragraph
RSSM uses recurrent deterministic memory and stochastic states; observations/rewards depend on both. The encoder is q(s_t | h_t, o_t), and observation information must pass through sampling. Solid arrows are generative, dashed arrows inference.
Go to primary source ↓e04PDF pp. 3–4, Latent dynamics, Variational encoder, Training objective and Eq. (3)
Gaussian observation/reward models, filtering encoder, reconstruction likelihood and KL regularization define joint variational learning. Reward losses follow the observation formulation analogously.
Go to primary source ↓e05PDF pp. 5–6, Figure 3, Section 4, Eqs. (5)–(7) and stop-gradient paragraph
Multi-step priors can be matched to posteriors in latent space without additional image decoding. Distances are weighted by beta_d; posterior gradients stop for d > 1. The final RSSM agent does not require overshooting.
Go to primary source ↓e06PDF p. 16, Appendix F, Eqs. (8)–(10); PDF p. 5, paragraph below Eq. (6)
The appendix derives one-step and multi-step predictive bounds. The proposed relationship between multi-step and one-step likelihoods is explicitly introduced as a conjecture using data processing.
Go to primary source ↓e07PDF p. 6, Section 5, environment descriptions and Comparison to model-free methods; PDF p. 7, Table 1 caption
Six DeepMind control tasks use 64 × 64 × 3 camera observations. The text reports a 26% relative Cheetah improvement; table statistics are mean final performance over 5 seeds and 10 trajectories.
Go to primary source ↓e08PDF p. 7, Table 1, all task columns, method rows, efficiency row and caption
Order cartpole/reacher/cheetah/finger/cup/walker: PlaNet 821/832/662/700/930/951 at 1000 episodes; D4PG 862/967/524/985/980/968 and A3C 558/285/214/129/105/311 at 100,000. True-simulator CEM: 850/964/656/825/993/994. Curve-estimated efficiency factors: 250/40/500+/300/100/90. No numerical error bars accompany table means.
Go to primary source ↓e09PDF p. 6, Model designs; PDF p. 7, Figure 4 and caption
RSSM is compared with deterministic GRU and stochastic SSM. Curves show test-return medians and percentiles 5–95 over 5 seeds and 10 trajectories; the tested deterministic model performs poorly.
Go to primary source ↓e10PDF p. 6, Agent designs; PDF p. 8, Figure 5 and caption
Random collection and selection from 1000 random sequences are compared with PlaNet. The authors report benefits from planned collection and CEM refinement. Statistics are medians and percentiles 5–95 over 5 seeds and 10 trajectories.
Go to primary source ↓e11PDF p. 12, Appendix A, all three paragraphs
Specifies network sizes, bit-depth reduction, optimizer/clipping/free-nat settings, batch/chunk sizes, seed/update schedule, action noise, task-specific repeats and CEM defaults. Convolutional modules refer to prior work; earlier overshooting/global-prior components are unnecessary in the final agent.
Go to primary source ↓e12PDF p. 3, Model-based planning and Planning algorithm; PDF p. 12, Appendix B, Algorithm 2, lines 1–10
CEM samples latent trajectories, sums mean rewards, refits elite actions and returns the first mean; distributions restart each decision. Planning uses neither decoder nor policy/value network. Printed scale update uses absolute deviations/(K−1), with inclusive t:t+H indexing.
Go to primary source ↓e13PDF p. 6, One agent all tasks; PDF p. 13, Appendix C, Figures 6–7 and captions
Shared agent uses padded action spaces and infers the task from images; collection adds one episode per task per round. Learning is slower than separate agents. Figure 6 reports 5 seeds and mean/standard deviation; Figure 7 reports 4 seeds and median/percentiles.
Go to primary source ↓e14PDF p. 14, Appendix D, Figure 8 and caption
RSSM and DRNN are tested with/without latent overshooting. Caption reports substantial DRNN benefits and slight RSSM degradation. DRNN has encoder/decoder RNNs separated by stochastic states; curves use 5 seeds and 10 trajectories.
Go to primary source ↓e15PDF p. 15, Appendix E, Figure 9 and caption
ELU improves the purely stochastic model relative to ReLU, whereas RSSM is robust to the activation choice; the deterministic-model improvement mentioned in the caption is not plotted.
Go to primary source ↓e16PDF pp. 18–19, Appendices H–I, Figures 10–11 and captions
Figure 10 shows reconstructed context and open-loop Cheetah predictions from noisy test episodes. Figure 11 freezes the dynamics model and trains small probes for simulator positions, velocities and reward; evidence is plotted trajectories rather than aggregate error metrics.
Go to primary source ↓e17PDF p. 20, Appendix J, Figure 12 and caption
Cheetah planning uses the true simulator while varying horizon, proposal count, iteration count and elite fraction. More search generally helps, but very short and much longer horizons hurt; the reported best horizon is near eight steps.
Go to primary source ↓e18PDF p. 8, Section 7, future-work paragraph
Proposes learned temporal abstraction, a value function beyond the horizon, gradient-based planning and representations without reconstruction for higher visual diversity.
Go to primary source ↓e20PDF p. 6, Section 5, opening paragraph
Authors report task-dependent training times of 10–20 hours on one Nvidia V100, implementation using TensorFlow Probability, and a link for code/videos; no software version is specified.
Go to primary source ↓8.5 Primary sources
Learning Latent Dynamics for Planning from Pixels ↗
PDF · 9,539 extracted words
Source fingerprint
abac727526e6a45669d3ab9957126587e22aacf4dc9bcd60ffe5c853108e5bcc