When to Trust Your Model: Model-Based Policy Optimization
1. Paper overview
In one sentence: MBPO gains useful training data from short model rollouts anchored to real experience, trading longer imagination for more reliable local predictions. e-probleme-algorithme-modele-sace-horizon
| At a glance | What to know |
|---|---|
| Research problem | Source description Environment interactions can be expensive, whereas learned models can generate additional samples. However, a policy optimized against inaccurate predictions may exploit model errors and perform poorly under actual dynamics. MBPO asks whether limited, carefully placed model rollouts can preserve model-based sample efficiency while approaching strong model-free final performance. e-problem |
| Core mechanism | Source description The analysis separates model error from policy distribution shift. Its initial branched-rollout bound actually favors zero model steps; incorporating error under the updated policy permits nonzero rollout lengths when that error is sufficiently small. e-boundse-generalization |
| A key reported result | Ant sample efficiency: MBPO at 300,000 steps is reported to match SAC at 3,000,000 steps. Environment interactions needed for comparable average return. Online MuJoCo continuous control; standard 1000-step episodes. Figure 2 reports mean and standard deviation across five trials; no fixed dataset train/test split is specified. SAC is also MBPO's underlying policy learner. Other plotted baselines are PPO, PETS, STEVE and SLBO. This is an author-reported tenfold interaction-count comparison at comparable performance, not an exact tabulated return threshold or measured wall-clock speedup. e-protocole-ant |
| Reading caution | Reader analysis The practical algorithm is motivated by conditional theory rather than a per-update certificate. Theorem 4.3 describes updated-policy error, but its printed expectation still uses the data-policy state-distribution notation. That inconsistency remains unresolved without the referenced proof; the report preserves it rather than silently correcting the formula. e-boundse-generalizatione-theory-notatione-algorithm |
Core contributions
- Source description
The analysis separates model error from policy distribution shift. Its initial branched-rollout bound actually favors zero model steps; incorporating error under the updated policy permits nonzero rollout lengths when that error is sufficiently small. e-boundse-generalization
- Source description
The practical contribution is many short rollouts from replay states, followed by off-policy actor-critic learning. This decouples model rollout length from episode length. Experiments compare six tasks and diagnose whether gains merely reflect more gradient updates or longer predictions. e-algorithme-protocole-no-modele-horizon
Algorithm 2. Two datasets connect real interaction, learned prediction and policy optimization. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the two dataset names rather than treating every loop as an environment rollout. Line 3 fits the predictive model on the environment dataset. Line 5 executes the policy and adds real experience. Lines 7–8 then choose a real replay state uniformly and generate a separate k-step trajectory under the model and current policy. Those transitions enter the model dataset, which supplies the policy update in line 10. M counts generated rollouts and G counts gradient updates; k controls each rollout's length. The update arrow changes policy parameters, not the predictive-model parameters. Section 5 supplies the ensemble sampling and SAC details behind these compact instructions. e-algorithme-modele-sace-boundse-supplement-gap
What it supports. Short prediction chains can still produce a large training dataset because the algorithm generates many branches. The source reports 20–40 policy gradient steps per environment sample. Its useful distinction is between the length of one modeled trajectory and the total amount of model-generated training data.
Where the evidence stops. This is pseudocode, not a network architecture diagram. The page's footnote says SAC also needs Q-function updates, omitted here for clarity. Full hyperparameters are deferred to unavailable Appendix C, and the practical loop does not explicitly enforce Algorithm 1's theoretical improvement penalty.
2. Motivation
2.1 The problem and the proposed response
Environment interactions can be expensive, whereas learned models can generate additional samples. However, a policy optimized against inaccurate predictions may exploit model errors and perform poorly under actual dynamics. MBPO asks whether limited, carefully placed model rollouts can preserve model-based sample efficiency while approaching strong model-free final performance. e-problem
2.2 What this reading follows
A learned simulator can generate far more experience than an agent can afford to collect, but optimizing against its mistakes can produce a poor policy. MBPO addresses this by changing where model rollouts start and how long they last. It samples real replay states, predicts a few transitions, and uses those transitions to train soft actor-critic. The model supports learning; the actor chooses actions during execution. Read the algorithm alongside the error diagnostics, then compare the benchmark and ablation curves. Together they explain why one-step predictions can improve long-episode control without requiring accurate simulation of an entire episode. e-probleme-algorithme-modele-sace-horizon
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational model-based RL and training-optimization categories are supported: the dynamics ensemble augments training for a separate actor-critic. The paper does not describe a unified model jointly predicting future observations and actions, or inverse dynamics extracting actions from desired futures. Keeping the WAM architecture, prediction-paradigm and quadrant fields not applicable is therefore appropriate; this is an architectural assessment, not evidence of a single jointly predictive model. e-related-mechanisme-modele-algorithme-sac
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Anchor prediction where experience is available
Imagine drawing a Hopper observation from the real replay buffer. The actor samples an action, and an ensemble member predicts the next observation and reward. With k = 1, that transition already supplies a training example; the model does not need to invent the rest of the episode. Repeat this operation from many replay states and the algorithm obtains broad training coverage while keeping each prediction chain short. In the source procedure, only real data fits the dynamics ensemble, while generated data supports SAC updates. Reader interpretation: replay anchoring limits how far each branch can drift before it stops, although it cannot eliminate one-step bias or errors caused by policy shift. Algorithm 2 separates these operations explicitly, making clear why a large synthetic dataset does not imply a long model horizon. e-algorithme-modele-sace-horizon
Figure 1. More training data improves both fit and local robustness to policy shift. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column as one environment. In row a, the horizontal axis is policy KL divergence from the data-collecting policy, and the color bar identifies how much data trained the frozen predictive model. Compare both the height near zero shift and the rise as the policy moves away. In row b, the vertical axis is the estimated local error-change rate around zero shift; the horizontal axis is training size. Both rows use logarithmic error-related axes, so vertical distances represent multiplicative changes. The paper uses the falling local sensitivity to motivate its linear approximation of error under an updated policy. e-generalizatione-boundse-theory-notatione-supplement-gap
What it supports. The plots support a local generalization argument: with more data, models tend to have lower error and smaller sensitivity to nearby policy changes. This makes limited model use plausible even when a pessimistic distribution-shift bound recommends none. It does not identify a universal safe rollout length.
Where the evidence stops. The plotted loss and KL quantities are empirical proxies, not direct numerical total-variation bounds. Theorem 4.3 also calls its error updated-policy error while retaining data-policy notation in the printed expectation; the absent proof leaves that notation discrepancy unresolved.
5.2 Read the guarantee as motivation for a tradeoff
The theoretical starting point compares true discounted return with modeled return minus an error penalty. Two quantities matter: how wrong the model is and how much the new policy differs from the policy that collected its data. Full modeled trajectories allow errors to compound. Branching introduces a rollout length k, but the first bound is so pessimistic that its preferred setting is k = 0. Figure 1 motivates replacing worst-case error growth with an empirical local approximation: more data tends to reduce sensitivity to nearby policy shifts. The revised bound can then favor positive k when updated-policy model error is sufficiently low. Reader interpretation: this explains why short rollouts are worth trying; it does not make Algorithm 2's finite SAC updates a certified optimizer of the bound, or make the empirical proxy an exact error bound. e-boundse-generalizatione-algorithm
5.3 Separate prediction quality, learning efficiency and executed control
Three observations have to be read together. Figure 2 measures environment return as training interactions accumulate and supports the reported Ant sample-efficiency advantage. Figure 3 asks whether extra updates alone explain that advantage and whether longer model horizons improve it; the Hopper controls favor useful short predictions over either simple explanation. Figure 4 shows that return agreement can coexist with deteriorating state trajectories during long model rollouts. Reader interpretation: a dynamics model need not be a faithful long-horizon simulator to be valuable as a local source of actor-critic training data. During execution the actor still chooses the action from the current observation. None of these plots demonstrates physical robot performance, and none alone establishes freedom from model exploitation under arbitrary policies or unfamiliar tasks. e-protocole-ante-no-modele-horizone-diagnostice-algorithm
5.4 Training and inference
During training
Train the ensemble by maximum likelihood on real environment data. Model-generated data supports policy learning, rather than replacing the environment dataset used to fit dynamics. The practical procedure alternates model fitting, interaction, synthetic rollout generation and SAC updates; it does not specify pretrained or permanently frozen modules. e-algorithme-model
The paper reports 20–40 policy gradient steps per environment sample. Rollout length controls individual prediction chains independently of the number of generated rollouts. Complete environment-specific settings are deferred to unavailable Appendix C, so the stated range is not a recovered configuration for every task. e-sace-supplement-gap
During inference
For action execution, the current policy receives the current observation and chooses an action. Model sampling supplies training data; MBPO does not use an online receding-horizon planner to select the executed action. The reported environment is a MuJoCo simulator. The paper's use of 'real environment' contrasts simulator dynamics with learned dynamics, not simulated control with physical deployment. e-related-mechanisme-algorithme-protocole-diagnostic
5.5 Implementation flow
- Collect grounding data
Execute the current policy in the environment and add observed transitions to the environment dataset. Both transition dynamics and the reward function are treated as unknown. New interaction data supports subsequent model retraining, providing feedback when the policy visits new states. e-backgrounde-algorithm
- Fit a probabilistic ensemble
Bootstrap an ensemble of neural predictors. Each produces a Gaussian distribution over next state and reward with diagonal covariance. The paper associates each member's probabilistic output with aleatoric uncertainty and differences induced by bootstrapping with epistemic uncertainty. e-model
- Branch briefly from replay
Sample a starting state uniformly from real replay, sample actions from the current policy, and unroll the learned dynamics for k steps. Select an ensemble member uniformly for each predicted transition, allowing member changes within one rollout. Store generated transitions in the model dataset. e-modele-algorithm
- Improve the actor and critic
Use SAC's Bellman-based policy evaluation and actor improvement on generated data. Repeat generation and gradient updates between real interactions. Algorithm 2 replaces Algorithm 1's intractable policy argmax with a fixed update count; its footnote explicitly says required Q-function updates are omitted from the pseudocode for clarity. e-sace-algorithm
6. Experiments & results
Model-based policy optimization (MBPO) uses a learned dynamics-and-reward ensemble to generate extra training transitions for soft actor-critic. Each short rollout begins at a real replay-buffer state, limiting accumulated prediction error while permitting many policy updates per environment interaction. The central question is how much model usage improves learning before model bias dominates. The paper combines conditional return bounds, empirical model-generalization diagnostics and MuJoCo experiments. Its strongest explicit comparison matches SAC's Ant performance using 300,000 rather than 3 million environment steps; it does not establish an equivalent reduction in computation time.
The supplied 12-page proceedings PDF contains four numbered figures and two algorithms, but no quantitative results table or architecture block diagram. This edition therefore uses all four original figures for quantitative and diagnostic evidence and Algorithm 2 as the original method visual. It does not manufacture a table from plotted curves. Appendices A and C are referenced for proofs and configurations but are absent from this artifact; separate supplemental material availability remains unverified. e-visual-scopee-supplement-gap
6.1 Read the original evidence
Figure 2. MBPO improves interaction efficiency across the displayed continuous-control comparisons. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal axes count environment interactions and the vertical axes show average return. Read MBPO in blue against the full legend, noting that axis ranges differ across tasks. Solid curves show five-trial means; shaded regions show standard deviations. Dashed horizontal references represent convergence performance for methods whose learning extends beyond the displayed window, so they are not trajectories achieved at every displayed step. The Ant panel is the clearest explicit sample-efficiency example: its caption and the following page compare MBPO at 300,000 interactions with SAC at 3 million. Hopper and Walker2d show different learning dynamics and variability rather than copies of the Ant result. e-protocole-ante-visual-scope
What it supports. The authors report that MBPO matches SAC's Ant performance with one tenth as many environment steps. Across all six panels, the useful comparison is how quickly substantial returns appear and how those returns relate to longer-run baselines. The evidence concerns simulated control using standard 1000-step episodes.
Where the evidence stops. Do not read the Ant comparison as a tenfold wall-clock speedup or an exact return threshold. The longer SAC run is summarized by a convergence reference, and uncertainty bands are standard deviations rather than confidence intervals. No endpoint results table accompanies these curves.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Ant sample efficiency Online MuJoCo continuous control; standard 1000-step episodes. Figure 2 reports mean and standard deviation across five trials; no fixed dataset train/test split is specified. | MBPO at 300,000 steps is reported to match SAC at 3,000,000 steps. Environment interactions needed for comparable average return | SAC is also MBPO's underlying policy learner. Other plotted baselines are PPO, PETS, STEVE and SLBO. This is an author-reported tenfold interaction-count comparison at comparable performance, not an exact tabulated return threshold or measured wall-clock speedup. e-protocole-ant |
| Hopper model-data ablation Figure 3 left: SAC without model data, varying gradient updates per environment step. | SAC settings G = 10, 20 and 30 remain below MBPO's learning curve; the text reports that extra updates alone do not match MBPO sample efficiency. Average return versus environment steps | Standard SAC and MBPO are the common reference curves. Supports a contribution from model-generated data beyond increasing update frequency. Exact endpoint returns and significance tests are not tabulated. e-no-model |
| Hopper rollout-length ablation Figure 3 center and Section 6.2; fixed model horizons compared with scheduled MBPO. | Fixed k = 1 retains much of the benefit. The authors report the best schedule increasing from 1 to 15; 200-step rollouts perform worse than shorter ones, and 500-step rollouts are too inaccurate for effective learning. Average return versus environment steps | Fixed k = 1, 200 and 500, with MBPO and SAC references. Longer prediction is not automatically more useful. Schedule endpoints are stated, but scheduling epochs are absent with Appendix C. e-horizone-supplement-gap |
| Hopper value-expansion comparison Figure 3 right and Section 6.2; model value expansion implemented on SAC to reduce policy-learner confounding. | H = 1 is competitive among the H = 1, 5 and 10 value-expansion settings, while MBPO learns faster in the displayed range. Average return versus environment steps | Model predictions improve targets for real transitions, versus MBPO training with generated transitions. The authors report no improvement from preliminary attempts to combine both approaches. This is not an exhaustive test of their compatibility. e-value-expansion |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Extra updates, longer rollouts and better targets test distinct explanations for MBPO's gains. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the common blue MBPO and green SAC curves as anchors, then read each local legend independently. At left, G is the number of SAC gradient updates per environment step, with no model data. In the center, k is model rollout length; the fixed values are 1, 200 and 500. The blue reference uses the scheduled MBPO configuration, whose best Hopper schedule is described as increasing from 1 to 15. At right, H is the value-expansion horizon, where modeled rewards improve targets for real transitions. These three horizontal comparisons ask different questions; the same orange or purple color does not identify the same intervention across panels. e-no-modele-horizone-value-expansione-supplement-gap
What it supports. More SAC updates alone do not recover MBPO's sample efficiency. Single-step rollouts retain much of MBPO's advantage, whereas very long rollouts lose effectiveness. One-step value expansion is also competitive within that family. Together these controls support useful local prediction as a central ingredient, without showing that every short-horizon design is equivalent.
Where the evidence stops. These diagnostics are shown for Hopper. The source does not provide exact endpoint tables or significance tests here, and schedule timing is deferred to missing Appendix C. The reported preliminary combination of MBPO and value expansion is not an exhaustive compatibility study.
Figure 4. Long-rollout state uncertainty grows even when modeled cumulative reward broadly tracks the environment. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with panel a, which traces the joints through a 450-step hopping sequence under the environment dynamics. Panel b repeats the same action sequence under the learned model 1000 times; its colored bands summarize one standard deviation around predicted joint trajectories. The increasingly broad bands show where repeated prediction loses temporal precision. Then read panel c separately: the horizontal axis is rollout step, and the vertical axis is cumulative return. Thin blue lines are individual model rollouts, thick blue is their mean, and black is the environment result. This panel concerns return agreement, a different property from precise agreement of joint trajectories. e-diagnostice-protocole-algorithm
What it supports. The displayed model does not inflate predicted return above environment return as the authors' model-exploitation concern would suggest. Nevertheless, trajectory uncertainty grows substantially. A model can therefore remain useful for short training branches while being unreliable as a detailed simulator of an entire extended motion sequence.
Where the evidence stops. The 'real environment' here is the MuJoCo benchmark, not a physical robot. This is a diagnostic for a trained Hopper policy, not a universal no-exploitation guarantee. Replaying an action sequence also differs from evaluating closed-loop robustness to unexpected disturbances.
7. Analysis & limitations
7.1 What the evidence leaves open
The practical algorithm is motivated by conditional theory rather than a per-update certificate. Theorem 4.3 describes updated-policy error, but its printed expectation still uses the data-policy state-distribution notation. That inconsistency remains unresolved without the referenced proof; the report preserves it rather than silently correcting the formula. e-boundse-generalizatione-theory-notatione-algorithm
Figure 4 shows growing uncertainty over a 450-step Hopper sequence despite broadly tracking cumulative returns. The authors interpret model underestimation as evidence against exploitation in these runs. One task's diagnostic is not a general guarantee against exploitation, nor proof of accurate long-horizon state prediction. e-diagnostic
Evaluation concerns six simulated continuous-control tasks. It does not establish visual world modeling, physical robot success or out-of-distribution task transfer. Learning curves support interaction efficiency, while missing full configurations and hardware details prevent a compute-matched interpretation. e-protocole-sace-supplement-gap
7.2 Questions for discussion
- Does one-step MBPO retain its advantage when synthetic-transition count and gradient-update count are both controlled?
- Can a measured local model-error slope select useful rollout lengths reliably as the policy changes?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful implementation needs a probabilistic bootstrap ensemble, SAC actor and critics, separate real/model datasets, replay-state branching and per-transition ensemble sampling. The supplied body leaves ensemble size, network widths, optimizer settings, model-data retention, exact batch composition and environment-specific rollout schedules unresolved; it directs readers to Appendix C. Hardware counts and software versions are not specified here. e-modele-algorithme-sace-supplement-gap
Proposed minimal check: compare Hopper SAC with matched gradient-update counts against one-step MBPO, keeping environment budget and evaluation protocol fixed. Separately vary rollout length while holding the number of generated transitions constant. Report return, model prediction error and computation cost; these controls distinguish useful local predictions from extra optimization and unequal synthetic-data volume. e-algorithme-no-modele-horizone-diagnostic
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Control optimization frequency before crediting model data
Reader-proposed, not run: on the same Hopper environment version with 1000-step episodes, compare SAC using only real replay against k = 1 MBPO. Sweep G = 10, 20 and 30 for both, holding actor/critic architecture, minibatch size, environment budget and evaluation episodes fixed. Use at least the source's five-trial scale and record return versus interactions, total gradient updates and elapsed computation. Declare any chosen model-data mixing rule because the supplied body does not recover it. A persistent MBPO advantage at matched G would support useful model-data augmentation; disappearance of the advantage would weaken that explanation. Equalize model-query counts within MBPO runs when comparing G settings. e-protocole-no-modele-horizone-algorithme-supplement-gap
Check 2: Separate rollout length from synthetic-data volume
Reader-proposed, not run: compare Hopper MBPO with fixed k = 1, 15, 200 and 500, using identical real-data budgets and SAC update counts. Adjust the number of branches so each condition generates the same number of model transitions; sample all starts uniformly from real replay. Treat fixed k = 15 as a new control, not the paper's increasing schedule. At matched checkpoints, evaluate policy return in the environment and compare model predictions with environment trajectories under the same actions, recording error growth and model-versus-environment return gaps. If longer horizons remain worse at equal transition volume and their prediction errors grow, that supports the compounding-error explanation. If ranking changes, data volume or branch coverage helped drive the original comparison. e-algorithme-horizone-diagnostice-supplement-gap
8.3 Reading coverage
Visual audit: All 12 supplied PDF pages were rendered and visually inspected, including the title and authors, Algorithms 1–2, the theoretical equations, all four numbered figures, experiment and reproduction details, and the reference-only ending. Five final original crops were individually inspected. The Figure 3 crop was widened to retain its complete rightmost axis label and viewed again. Legends, axes, pseudocode update directions and the Algorithm 2 Q-update omission were checked against captions and surrounding text. Figure 1's KL-versus-TV distinction and Theorem 4.3's distribution notation discrepancy are disclosed. No quantitative table or architecture block diagram occurs in the supplied artifact; no referred appendix or external supplement was available for inspection.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related work
- 3 Background
- 4 Monotonic improvement with model bias
- 4.1 Monotonic model-based improvement
- 4.2 Interpolating model-based and model-free updates
- 4.3 Model generalization in practice
- 5 Model-based policy optimization with deep reinforcement learning
- 6 Experiments
- 6.1 Comparative evaluation
- 6.2 Design evaluation
- 7 Discussion
- Acknowledgements
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The complete supplied 12-page NeurIPS 2019 proceedings artifact was read. Its title and four authors match the catalog; the title page identifies University of California, Berkeley. No separate revision number or revision date is printed, and no comparison with other editions was supplied.
- Appendices A and C are referenced for proofs and hyperparameters but are absent from this PDF, which ends with references. Separate supplemental material availability has not been fully verified; no supplement was supplied or read.
- Text extraction does not reconstruct figure images and corrupts some mathematical symbols. All 12 PDF pages were visually inspected to check layout, equations and figures; five original crops were inspected separately.
- Code was not supplied or inspected, and no experiments were reproduced. External references and other paper versions were not opened.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block and conference footer
The observed title matches the catalog. Authors are Michael Janner, Justin Fu, Marvin Zhang and Sergey Levine, affiliated with University of California, Berkeley; the footer identifies NeurIPS 2019.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section 1
The paper frames model usage as a tradeoff between efficient synthetic-data generation and model bias, and proposes short rollouts to decouple model and task horizons.
Go to primary source ↓e-backgroundPDF p. 3, Section 3 and Algorithm 1, lines 1–4
States, actions, discounted return and unknown dynamics are defined; reward is also learned. Real state/action/next-state/reward tuples train the predictive model by maximum likelihood.
Go to primary source ↓e-related-mechanismPDF p. 2, Section 2, first paragraph and discussion of Dyna-style model use
The probabilistic ensemble is used to learn a policy rather than for receding-horizon planning; model rollouts can supply extra training examples.
Go to primary source ↓e-boundsPDF p. 3, Section 4.1; p. 4, Theorems 4.1–4.2, Eqs. (1)–(2), and Section 4.3 first paragraph
True return is lower-bounded by model return minus an error penalty. Model error and policy shift are controlled using total variation; policy KL is used empirically. The initial branched-rollout bound is maximized at k = 0.
Go to primary source ↓e-generalizationPDF p. 4, Section 4.3; p. 5, Figure 1(a–b), caption, local-error approximation and Theorem 4.3, Eq. (3)
Frozen models trained on different amounts of policy data are evaluated under shifted policies. Larger datasets reduce local error sensitivity in Hopper and Walker2d. An updated-policy error approximation allows positive rollout lengths when error is sufficiently small.
Go to primary source ↓e-theory-notationPDF p. 5, paragraph preceding Theorem 4.3 and the theorem's printed definition of epsilon_m'
The prose identifies error under the current or updated policy, while the theorem's displayed expectation retains s sampled from pi_D,t. The proof is referred to Appendix A, Theorem A.2.
Go to primary source ↓e-modelPDF p. 6, Section 5, Predictive model paragraph and Gaussian equation
Bootstrap ensemble members output diagonal-covariance Gaussians over next state and reward; a member is selected uniformly for each transition. The text distinguishes aleatoric output uncertainty and epistemic parameter uncertainty.
Go to primary source ↓e-algorithmPDF p. 6, Algorithm 2, lines 1–10; Section 5, Model usage; footnote 1
The model fits real data; real actions add environment data; uniformly sampled replay states start k-step policy-driven model rollouts; model data supports gradient updates. Fixed updates replace an argmax. Q-function updates are necessary but omitted from the pseudocode.
Go to primary source ↓e-sacPDF p. 6, Section 5, Policy optimization and final Model usage paragraph
SAC alternates Bellman-based Q evaluation with actor improvement. Many short model rollouts enable 20–40 policy gradient steps per environment sample; complete hyperparameters are deferred to Appendix C.
Go to primary source ↓e-protocolPDF p. 7, Figure 2, all six panels and caption; Section 6.1
MuJoCo tasks are InvertedPendulum, Hopper, Walker2d, Ant, HalfCheetah and Humanoid. MBPO is compared with SAC, PPO, PETS, STEVE and SLBO using standard 1000-step episodes. Curves show five-trial means with standard-deviation shading; dashed lines indicate convergence performance.
Go to primary source ↓e-antPDF p. 7, Figure 2 Ant panel and caption; p. 8, Section 6.1 continuation immediately below Figure 3
The authors explicitly state that Ant performance at 300,000 MBPO environment steps matches SAC performance at 3 million steps. The plot displays the shorter training window and an asymptotic SAC reference.
Go to primary source ↓e-no-modelPDF p. 8, Figure 3 left panel and caption; Section 6.2, No model
Hopper SAC variants use G = 10, 20 and 30 updates per environment step without model data. Increasing updates speeds learning somewhat but does not match MBPO sample efficiency.
Go to primary source ↓e-horizonPDF p. 8, Figure 3 center panel and caption; Section 6.2, Rollout horizon
Fixed k = 1, 200 and 500 are compared. Single-step rollouts retain much of MBPO's benefit; the best reported Hopper schedule grows linearly from 1 to 15. The text describes 200 steps as usable but worse than shorter horizons and 500 as too inaccurate.
Go to primary source ↓e-value-expansionPDF p. 8, Figure 3 right panel and Section 6.2, Value expansion; p. 9, continuation above Model exploitation
SAC is augmented with H-step model value expansion for H = 1, 5 and 10. One-step expansion is competitive, and preliminary combination with MBPO did not improve performance. The paper acknowledges confounds in comparisons using different policy learners.
Go to primary source ↓e-diagnosticPDF p. 9, Figure 4(a–c), caption, and Section 6.2, Model exploitation
A 450-step Hopper sequence is replayed under the learned model 1000 times. Prediction bands denote one standard deviation. Thin blue curves show individual modeled cumulative returns, thick blue their mean, and black environment returns. The authors observe accumulating state uncertainty and generally underestimated model returns.
Go to primary source ↓e-supplement-gapPDF pp. 4–5, proof references; p. 6, final Section 5 paragraph; p. 8, Section 6.2; pp. 10–12, ending reference section
Proofs point to Appendix A and full hyperparameters and schedules point to Appendix C. The supplied artifact contains neither appendix and ends with references on p. 12. Those referred details cannot be recovered from this artifact.
Go to primary source ↓e-visual-scopePDF pp. 1–12; Algorithms 1–2 on pp. 3 and 6; Figures 1–4 on pp. 5, 7, 8 and 9
The supplied paper presents its mechanism as equations and pseudocode, empirical comparisons as plots, and no quantitative results table or architecture block diagram. Its four numbered figures cover generalization, benchmarks, ablations and rollout diagnostics.
Go to primary source ↓8.5 Primary sources
When to Trust Your Model: Model-Based Policy Optimization ↗
PDF · 6,463 extracted words
Source fingerprint
675f5b1fda804f3bcdd846eec04b7aeef08767888ea18778b0e722a37697b9f7