Temporal Difference Learning for Model Predictive Control
1. Paper overview
In one sentence: A reward-oriented latent model and terminal Q function make short-horizon continuous-action planning effective, with task-dependent costs in computation and transfer. identityproblemplanningtoldlosseslocomotionmulti-settings
| At a glance | What to know |
|---|---|
| Research problem | Source description Finite-horizon MPC is myopic, while longer rollouts increase computation and model error. The paper asks whether TD learning can supply long-term value and train a compact model around reward-relevant dynamics, avoiding the burden of reconstructing observations. Its setting is online reinforcement learning with continuous actions and environment reward feedback. problemplanning |
| Core mechanism | |
| A key reported result | Walker Walk time to solve: TD-MPC: 0.47 h to solve; 5.60 h/500k steps. Hours to threshold; hours per 500k environment steps (lower is better). State-based DMControl; target average return 940; one RTX3090; mean of five runs. LOOP: 7.72 h and 18.5 h; SAC: 0.41 h and 1.41 h; MPC:sim: 0.91 h to solve. The source reports about 16× faster solving than LOOP, while SAC has slightly shorter solving time and cheaper fixed-budget training. No timing uncertainty is tabulated. wall-time |
| Reading caution | Reader analysis Evidence comes from DMControl and Meta-World simulation. Ground-truth MPC has no terminal value and different sampling budgets, so its weaker results do not isolate model accuracy. Pixel baselines mix published protocols. locomotionmulti-settingsbaseline-protocolpixel-results |
Core contributions
Figure 1. Short imagined trajectories guide a controller that repeatedly returns to real environment feedback. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the observation on the upper left. Encoding produces z₀, and the blue branches represent alternative latent rollouts scored using reward and value. The red path leads to an action and then the environment; the green path returns feedback. The schematic places the action arrow after zH, but Algorithm 1 and Section 3 specify that only the first planned action is executed, followed by replanning. Below, compare episode return against environment steps within each task. Red is TD-MPC, blue is SAC, and the black dashed line is MPC with a ground-truth simulator. Shading denotes 95% confidence intervals over five runs. planninglocomotionbaseline-protocol
What it supports. The six locomotion panels show TD-MPC learning substantially stronger control than SAC under the plotted budgets, including the 38-dimensional Dog action space. The method combines imagined short-term consequences with a long-term value estimate; these curves demonstrate the complete controller's behavior, rather than directly measuring latent prediction accuracy.
Where the evidence stops. MPC:sim has no terminal value and uses a different search budget. Its comparison does not isolate learned versus exact dynamics. These are simulator returns, and the action-arrow placement should not be read as execution of the terminal action.
2. Motivation
2.1 The problem and the proposed response
Finite-horizon MPC is myopic, while longer rollouts increase computation and model error. The paper asks whether TD learning can supply long-term value and train a compact model around reward-relevant dynamics, avoiding the burden of reconstructing observations. Its setting is online reinforcement learning with continuous actions and environment reward feedback. problemplanning
2.2 What this reading follows
Consider a controller that must choose many joint commands but cannot afford to simulate a long future at every decision. TD-MPC evaluates short candidate trajectories in a learned latent space, then attaches a value estimate for what happens after the rollout ends. Its distinctive choice is to train that latent model through reward, temporal-difference value and consistency objectives rather than observation reconstruction. The figures below explain how training creates this planning space, where additional computation helps, and what the experiments actually establish. The evidence comes from the supplied ICML 2022 proceedings paper and its appendices; all reported control experiments are simulated. identityproblemplanningtoldlosseslocomotionmulti-settings
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The foundational model-based-RL and planning classification is supported. Candidate actions drive latent future prediction; a separate sampling optimizer selects control actions, aided by a policy. This is not joint future/action generation or inverse dynamics. Shared TD training alone does not justify a One Model label, so the recorded architecture/paradigm/quadrant Not applicable entries are reasonable within this catalog. taxonomy
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Learn what planning needs to predict
TOLD begins from the observation encoder hθ, but its training target is not a reconstructed observation. A replay action advances the latent state through dθ, and reward and Q heads read that evolving state. Errors from several rollout steps backpropagate through the dynamics, so the learned representation is shaped by downstream reward and value estimation. The consistency term supplies a further constraint: the predicted next latent should agree with the target encoder's representation of the actual next observation. This connects imagination to observed transitions while avoiding a decoder. The target parameters move slowly to stabilize supervision. Policy learning is separate within this procedure: Equation (11) adjusts only the policy parameters to increase Q, rather than letting that objective freely reshape the representation. toldlossespolicy
Figure 2. Train a latent rollout to preserve the information needed for rewards and values. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal blue path from z₀ through z₁ and z₂. Only the first observation is encoded by the online hθ; subsequent online latents come from dθ with replay actions. The gray hθ− arrows encode later observed states to form consistency targets. They are supervision for the blue predictions, not replacement inputs to the imagined rollout. Beneath each latent, the trophy, apple and joystick denote value, reward and policy outputs. Equations (7)–(10) train the model with temporally weighted reward, TD and consistency losses. Equation (11) separately trains the policy to increase Q, with stop-gradient on its latent input. toldlossespolicyplanning
What it supports. The representation is trained for control-relevant prediction without reconstructing pixels or full states. Multi-step gradients connect reward and value errors back through the latent dynamics. The policy contributes both inexpensive TD-target action selection and candidate trajectories for planning; its presence does not turn training into action imitation.
Where the evidence stops. The graphic is schematic: it omits the explicit loss nodes and policy stop-gradient. Gray future observations are available as training targets only. During control, the model must roll forward from the current observation without access to those future observations.
5.2 2. Use a critic to finish a short plan
At a decision, TD-MPC does not ask its dynamics model to predict an entire successful episode. It samples continuous action sequences, simulates a short latent rollout, and adds discounted predicted rewards. At the end it queries Q for the remaining return. The terminal critic therefore supplies information that a short reward-only rollout would miss. A diagonal Gaussian represents the search distribution, with return-weighted elite sequences determining subsequent means and variances. Additional trajectories come from the learned policy and compete on the same score. After optimization, only the first action is executed. A new observation then resets the latent starting point for the next search, while a shifted previous mean warm-starts the action distribution. The horizon-five default is a practical choice whose sensitivity is tested, not a general optimality guarantee. planningsettingsbudget
5.3 3. Separate a strong controller from a universal model
Reader analysis: the main experimental claim should be assessed at the controller level. Strong Humanoid and Dog returns show that the learned model, critic, policy proposals and optimizer work together. They do not identify which module alone accounts for the advantage over a simulator baseline that has neither the same search budget nor a terminal value. The regularizer comparison gives more direct evidence about representation learning, although its alternative heads and coefficients differ. Transfer adds another boundary: retaining the encoder is useful between Walk and Run, but retaining fixed dynamics is substantially less effective. Finally, Table 2 separates interaction efficiency from computation per interaction. A reproduction should report both return versus environment steps and elapsed time to a declared threshold, rather than compressing these into one efficiency claim. locomotionbaseline-protocolablationtransferwall-time
5.4 Training and inference
During training
Replay sampling is prioritized by value loss. Only the first online observation encoding initializes each recurrent rollout; future target encodings supervise latent consistency. All three model losses backpropagate through time. The policy objective updates only policy parameters, using stop-gradient on its latent input. Target networks move slowly rather than remaining permanently frozen. lossespolicysettings
Table 4 sets H=5, discount 0.99, temporal weight 0.5 and reward/value/consistency coefficients 0.5/0.1/2. Adam uses task-dependent learning rates and batches. Over 25k decision steps, the exploration floor falls from 0.5 to 0.05 and the horizon grows from 1 to 5. Pixel inputs use three 84×84 RGB frames with ±4-pixel shifts. hyperparameterssettingspixels
During inference
Use 512 Gaussian candidates, 5% additional policy trajectories and 64 elites. Table 4 refines the main-text summary: planning uses 12 iterations for Humanoid, 8 for Dog and pixels, and 6 otherwise. A trained agent can trade return against planning budget or use the faster, generally weaker policy alone. settingsbudgetlatency
5.5 Implementation flow
- Encode and imagine
Encode the current observation with hθ. Recur through deterministic, action-conditioned dynamics dθ; reward and Q heads score predicted latents. State inputs use an MLP encoder, pixels a CNN. Multi-modal experiments sum separate encodings; MT10 appends a one-hot task vector. toldpixelsimplementationmulti-settings
- Search continuous action sequences
Sample a time-dependent diagonal Gaussian plus policy-generated trajectories. Rank by discounted predicted rewards and terminal Q, then refit mean and variance using exponentially weighted elites. Warm-start the next search with a shifted mean, retaining a large initial variance. planning
- Close the feedback loop
Execute only the first action, receive the next observation and reward, and replan. The policy proposes actions and supplies TD maximization; it is not an inverse-dynamics decoder from an imagined goal frame. planningpolicy
6. Experiments & results
TD-MPC couples short-horizon continuous-action planning with a reward-oriented latent dynamics model and a learned terminal Q function. Training shapes predicted latents through reward, value and consistency losses; control executes the first optimized action and replans from feedback. The experiments support strong simulated-control performance with a task-dependent computation tradeoff, not a general-purpose environment simulator or demonstrated physical deployment.
6.1 Read the original evidence
Table 2. Sample efficiency can offset planning cost when measuring time to a useful policy. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the task-group headers before comparing methods. The upper data row measures hours until average return reaches 940 for Walker Walk or 800 for Humanoid Stand. The lower row instead measures hours per 500k environment steps. Both are lower-is-better metrics, but they answer different questions: how quickly a target is reached versus how expensive a fixed interaction budget is. Results are means of five runs on one RTX3090. The dash for MPC:sim in the fixed-budget row is unreported; it should not be read as zero. The crop preserves both task groups and all numeric entries. wall-time
What it supports. On Walker Walk, TD-MPC reaches the threshold in 0.47 hours versus LOOP's 7.72, while taking 5.60 versus 18.5 hours per 500k steps. SAC reaches it in 0.41 hours with lower fixed-budget cost. Humanoid Stand solving times are also close: 9.39 hours for TD-MPC and 9.31 for SAC.
Where the evidence stops. The table prints no timing uncertainty. The thresholds are task-specific and the measurements concern one GPU setting. These results support a favorable time-to-threshold tradeoff against LOOP, not a claim that TD-MPC is cheaper than SAC per interaction.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Walker Walk time to solve State-based DMControl; target average return 940; one RTX3090; mean of five runs. | TD-MPC: 0.47 h to solve; 5.60 h/500k steps. Hours to threshold; hours per 500k environment steps (lower is better) | LOOP: 7.72 h and 18.5 h; SAC: 0.41 h and 1.41 h; MPC:sim: 0.91 h to solve. The source reports about 16× faster solving than LOOP, while SAC has slightly shorter solving time and cheaper fixed-budget training. No timing uncertainty is tabulated. wall-time |
| Image-based DMControl at 100k environment steps Table 1; task-dependent action repeat; ten runs; baselines partly imported from prior work. | Finger Spin: 943±59; Cheetah Run: 222±88. Episode return, mean ± standard deviation | DrQ: 901±104 and 344±67, respectively. Pixel performance is competitive but task-dependent. The means do not establish significance. EfficientZero entries elsewhere in the table use discretization and extra gradient updates. pixel-resultspixels |
| Latent-regularizer ablation 15 state-based DMControl tasks through 500k environment steps; five runs with 95% confidence bands. | Latent consistency has the highest plotted aggregate curve; strong separations appear on Quadruped Walk and Acrobot Swingup. Episode-return learning curves | No regularizer, reconstruction and the source-labelled contrastive objective. This supports the chosen regularizer in the tested configurations, not uniform superiority: Cheetah Run favors reconstruction. Alternative heads and loss scales also change. ablation |
| Quadruped Walk planning-budget sensitivity Fully trained agents; vary one inference parameter while fixing the other; five-run means. | Three iterations retain near-default performance relative to six; horizon one substantially reduces return. Episode return versus horizon or iteration count | Default horizon 5 and six iterations; policy-only evaluation. Computation can be reduced selectively. The source claims a 50% planning-cost reduction from halving iterations; the figure does not supply uncertainty or prove equivalence. budget |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 10. The regularizer changes control performance, with benefits that vary across tasks. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the Average panel, then inspect tasks whose curves separate strongly, such as Acrobot Swingup and Quadruped Walk. The red curve uses latent state consistency, brown omits regularization, green reconstructs states, and orange uses the source-labelled contrastive alternative. All variants retain reward and value prediction, making the extra representation objective the focus of this comparison. The horizontal scale reaches 500k environment steps; the vertical scale is episode return and differs across tasks. The curves are five-run means with 95% confidence bands. Appendix D describes the contrastive alternative's projection and predictor heads, BatchNorm, and increased loss coefficient. ablationlosses
What it supports. Latent consistency has the strongest aggregate curve and substantial gains on several demanding tasks. It is not the highest curve everywhere: reconstruction performs better on Cheetah Run, while simpler Cartpole tasks show little separation. The useful conclusion is robustness across this tested set, rather than a universal ordering of representation losses.
Where the evidence stops. The alternative objectives also change architecture and loss scaling, including coefficient 100 versus 2 for the contrastive variant. This is a comparison of implemented alternatives, not a perfectly isolated change in loss form with identical capacity and optimization.
Figure 6. With fixed learned weights, search depth and search refinement affect return differently. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Both panels evaluate fully trained Quadruped Walk agents. On the left, the blue line varies rollout horizon while holding iterations at the default; on the right, the green line varies iterations while holding horizon fixed. Orange stars mark the training configuration, horizon five and six iterations. Gray crosses show the learned policy without planning and are reference levels rather than horizon-zero samples. The axis retains the paper's label CEM iterations, while Section 3 defines an MPPI-style importance-weighted elite update. Read the curves as an inference-budget experiment: the agents are not retrained separately for every displayed point. budgetplanninglatency
What it supports. Reducing iterations from six to three preserves a return close to the default, whereas a horizon of one gives much weaker control on this task. At adequate budget, planning exceeds the jointly trained policy. The two knobs therefore offer different practical tradeoffs even though both affect planning computation.
Where the evidence stops. The figure reports five-run means without uncertainty bars. Similar means do not establish statistical equivalence. Appendix C shows task-dependent sensitivity, and the separate latency experiment in Appendix H uses Quadruped Run, so its milliseconds should not be assigned directly to this plot.
Figure 7. Reward-oriented representations transfer between related tasks more readily than frozen dynamics. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each panel begins a Run task after either random initialization or prior training on the corresponding Walk task. Red trains from scratch, purple finetunes all parameters, cyan freezes the encoder hθ, and gray freezes both hθ and dynamics dθ. The snowflake in the legend means frozen parameters. All finetuning is online, so new interactions continue to update the unfrozen components; this is not zero-shot evaluation. Appendix A keeps hyperparameters unchanged across these conditions. Follow the curves over the new task's environment steps, and distinguish fast initial improvement from the final return. Shading gives 95% confidence intervals over five runs. transferboundaries
What it supports. Full finetuning accelerates learning, and freezing only the encoder retains much of that advantage. Freezing dynamics as well causes a substantial loss, particularly in Quadruped Run. This supports the authors' interpretation that the encoder preserves useful shared information while the learned transition representation contains more task-specific structure.
Where the evidence stops. The transfer pairs share an environment domain and closely related locomotion rewards. They do not establish transfer to unrelated tasks, and the Run-task horizontal axis does not include the cost of earlier Walk training.
7. Analysis & limitations
7.1 What the evidence leaves open
Evidence comes from DMControl and Meta-World simulation. Ground-truth MPC has no terminal value and different sampling budgets, so its weaker results do not isolate model accuracy. Pixel baselines mix published protocols. locomotionmulti-settingsbaseline-protocolpixel-results
Finger Turn Hard exposes weaker exploration than SAC/LOOP. Walk-to-Run transfer helps, especially with a frozen encoder, but freezing dynamics also degrades learning; transfer to unrelated tasks remains unestablished. boundariestransfer
MT10 average success differs from the maximum-per-task metric in Table 8. Only 24 individual Meta-World curves are shown, and v2 results cannot be directly compared with v1. multi-settingsboundaries
Two appendix details remain internally inconsistent: Figure 12 calls IQM a median while Appendix K calls it a mean; Figure 11 places policy latency near 1 ms while the text says nearly six times faster than approximately 20-ms planning. No precise policy speedup is adopted here. aggregatelatency
7.2 Questions for discussion
- How much of short-horizon performance depends on accurate terminal values rather than latent dynamics accuracy?
- Would the encoder still transfer when the new task rewards features irrelevant to the original task?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires benchmark variants, action repeats, replay/prioritization, exploration schedules, task-specific batches and planner budgets from Tables 4–7. For state Quadruped tasks, use repeat 4; for Walker use 2. Timing comparisons require the reported RTX3090 setting. hyperparameterssettingswall-time
The printed implementation includes twin Q heads but leaves compute_td unexpanded. Its range(H) loop differs from the inclusive horizon in Eq. (7)/Algorithm 2. Resolve transition indexing, target construction and target-momentum conventions before claiming exact replication; the paper is not an executable specification. implementationlosseshyperparameters
A useful proposed minimum test fixes a trained Quadruped Walk checkpoint and compares three versus six planning iterations under identical evaluation starts. Separately, compare latent consistency against c3=0 with matched training settings and multiple seeds to test the regularizer mechanism. These checks are proposals, not reproduced results. budgetablationhyperparameters
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test the claimed iteration saving on a fixed controller
Reader-proposed check: train state-based Quadruped Walk using the documented default horizon five and six planning iterations, then freeze each of at least five independently trained checkpoints. Evaluate policy-only, one, three and six iterations on paired initial states, retaining the same population, elite count and policy fraction. Keep action repeat four and horizon five fixed. Record return distributions and synchronized decision latency on a reported GPU, with warm-start handling identical across conditions. Predefine a practical return-equivalence margin. The claimed saving is supported only if three iterations fall within that margin of six and reduce measured latency; a reproducible return drop would falsify equivalence for this setup. budgetsettingshyperparameterslatency
Check 2: Isolate the latent-consistency contribution
Reader-proposed check: on state-based Quadruped Walk and Cheetah Run, compare c3=2 against c3=0 while retaining the same encoder, dynamics, twin critics, reward/value losses, planner and data budget. Use at least five seeds and keep initialization, action repeat and exploration schedules paired where possible. Measure return curves, elapsed training time, and held-out multi-step reward/value prediction errors. First resolve the paper's inclusive-horizon versus range(H) indexing and document one consistent choice for both arms. A stable benefit only on Quadruped Walk would support task-dependent usefulness; no return or prediction benefit across either task would weaken the proposed regularization explanation under this controlled setup. ablationlossesimplementationhyperparameters
8.3 Reading coverage
Visual audit: All 20 pages of the supplied proceedings PDF were visually inspected, including the title and affiliation block, Algorithms 1–2, Eqs. (1)–(11), Figures 1–15 and Tables 1–8. The six final crops were individually inspected with their original labels and legends intact. Claim support includes uncropped Table 1 on p. 7, architecture and pseudocode on p. 14, hyperparameters on p. 16, action repeats and timing on p. 17, and aggregate metrics and Meta-World details on pp. 18–20. Figure 1's terminal-position action arrow is interpreted using Algorithm 1's first-action execution rule; Figure 2's gray arrows are target supervision, with policy stop-gradient specified by Eq. (11). Budget plots retain the source's CEM label despite the MPPI formulation in Section 3. Source ambiguities in horizon indexing, IQM naming and policy latency are disclosed in the base report. No external videos, code or separate supplements were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Preliminaries
- 3. TD-Learning for Model Predictive Control
- 4. Task-Oriented Latent Dynamics Model
- 5. Experiments
- 6. Related Work
- 7. Conclusions and Future Directions
- Acknowledgements and References
- A. Model Generalization
- B. Comparison to Prior Work
- C. Variable Computational Budget
- D. Latent Dynamics Objective
- E. Exploration by planning
- F. Implementation Details
- G. Extended Description of Baselines
- H. Inference Time
- I. Meta-World
- J. Multi-Modal RL
- K. Additional Metrics
- L. Task Visualizations
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All supplied text, including appendices, was read; all 20 PDF pages and Figures 1–15/Tables 1–8 were visually inspected, addressing the extraction-only figure limitation.
- Identity/version note: the inspected ICML 2022/PMLR 162 proceedings PDF has the exact catalog title. It prints Nicklas Hansen; Xiaolong Wang; Hao Su, with Wang and Su marked equal contributors. The catalog instead gives Nicklas A Hansen; Hao Su; Xiaolong Wang. This is a name-form/order discrepancy, not evidence of a different work. No numbered revision or revision history is established by this artifact, and no alternate edition was compared.
- No external code, project videos or separate supplements were inspected; no experiments were reproduced.
- The source displays individual curves for only 24 of 50 Meta-World tasks. Missing curves and untabulated point estimates have not been reconstructed.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author line, affiliation and proceedings footnotes
The title matches the catalog. The printed author order is Nicklas Hansen, Xiaolong Wang, Hao Su; Wang and Su share an equal-contribution marker. All authors are affiliated with UC San Diego. The artifact identifies ICML 2022, PMLR 162.
Go to primary source ↓problemPDF pp. 1–2, Abstract and Section 1
Long-horizon planning is expensive and inaccurate learned dynamics compound errors. TD-MPC combines short model rollouts with a learned terminal value and avoids observation reconstruction.
Go to primary source ↓planningPDF p. 3, Section 3, Eqs. (3)–(5) and Algorithm 1; p. 4, policy-guided trajectory optimization
MPPI fits a time-dependent diagonal Gaussian using return-weighted elite trajectories, including policy proposals. Return combines predicted rewards and terminal Q. Only the first action is executed; the mean is shifted to warm-start replanning. Exploration uses a variance floor.
Go to primary source ↓toldPDF p. 4, Section 4, Eq. (6), Figure 2 and caption
TOLD contains an observation encoder, action-conditioned latent dynamics, reward predictor, Q function and policy. Training starts from one online encoding and compares recurrent latent predictions against future observations encoded by the target network.
Go to primary source ↓lossesPDF p. 4, Eqs. (7)–(10); p. 5, Section 4 and Algorithm 2
Training sums temporally weighted reward, TD-value and latent-consistency squared errors. Gradients flow through recurrent latent predictions. Target encodings and target values use slow-moving parameters. Eq. (7) and Algorithm 2 use an inclusive t through t+H index.
Go to primary source ↓policyPDF p. 5, Computing TD-targets, Eq. (11)
The policy maximizes Q to approximate action maximization in the TD target, avoiding an inner planning problem. Its objective uses stop-gradient on the latent input and updates only policy parameters.
Go to primary source ↓settingsPDF pp. 5–6, implementation details; p. 16, Table 4
Planning uses horizon 5, population 512, 64 elites and 5% policy trajectories. Table 4 specifies 12 iterations for Humanoid, 8 for Dog and pixels, and 6 otherwise. Exploration and horizon schedules span 25k decision steps.
Go to primary source ↓pixelsPDF p. 6, implementation details; p. 14, Appendix F, encoder architecture; p. 17, Table 7
Pixel input uses three stacked 84-by-84 RGB frames with four-pixel shifts and a four-layer CNN encoder. Action repeat differs by task in the 100k benchmark; the Dreamer benchmark uses repeat 2 throughout.
Go to primary source ↓locomotionPDF p. 1, Figure 1 and caption; p. 6, Figure 3 and caption; p. 7, task list
The high-dimensional Humanoid and Dog curves compare TD-MPC, SAC and MPC:sim, using five runs and 95% confidence intervals. Figure 3 reports 15 state-based tasks and a mean panel. Humanoid and Dog have 21- and 38-dimensional action spaces.
Go to primary source ↓pixel-resultsPDF p. 7, Table 1, Finger Spin and Cheetah Run rows, TD-MPC and DrQ columns, and full caption
At 100k environment steps, Finger Spin returns are 943±59 for TD-MPC and 901±104 for DrQ; Cheetah Run returns are 222±88 and 344±67. Uncertainty is standard deviation over ten runs. Some baselines are sourced from prior papers; EfficientZero uses discretized actions and 20k additional gradient steps.
Go to primary source ↓multi-settingsPDF p. 8, Figure 5 and discussion; p. 14, Appendix F; p. 17, Appendices I–J and Table 8; p. 18, Appendix J; p. 19, Figure 14 caption
Experiments include 50 goal-conditioned Meta-World v2 tasks, MT10 with a one-hot task input, and two quadruped tasks combining proprioception and an egocentric camera. Modal encodings are summed. Figure 14 displays only 24 of 50 task curves. Table 8 reports maximum per-task success for SAC, distinct from Figure 5 average success.
Go to primary source ↓budgetPDF p. 8, Figure 6, caption and computational-budget paragraph; pp. 12–13, Appendix C and Figure 8
Fully trained agents are evaluated with varying horizon or planning iterations while holding the other setting fixed. Quadruped Walk retains near-default return with three instead of six iterations; horizon one performs worse than the policy. Additional tasks show differing sensitivity.
Go to primary source ↓wall-timePDF p. 9, Table 2 and Training wall-time paragraph
On one RTX3090, mean time to reach return 940 on Walker Walk is 0.47 hours for TD-MPC, 7.72 for LOOP, 0.41 for SAC and 0.91 for MPC:sim. Hours per 500k steps are 5.60, 18.5 and 1.41 for TD-MPC, LOOP and SAC. Humanoid Stand threshold 800 is reached in 9.39 hours by TD-MPC versus 9.31 by SAC; five runs, no uncertainty printed.
Go to primary source ↓transferPDF p. 12, Appendix A, Figure 7 and caption
Walk-to-Run online finetuning is compared with training from scratch on Walker and Quadruped, keeping hyperparameters unchanged. Freezing the encoder nearly matches full finetuning; additionally freezing latent dynamics degrades convergence. Curves show five-run means and 95% confidence intervals.
Go to primary source ↓ablationPDF pp. 12–13, Appendix D; p. 15, Figure 10 and caption; p. 16, no-consistency-regularization paragraph
The 15-task loss comparison retains reward/value learning and varies the regularizer among none, reconstruction, a SimSiam-based objective labelled contrastive, and latent consistency. The alternative adds projection/predictor heads, BatchNorm and coefficient 100 instead of 2. The latent-consistency average is highest, without winning every task.
Go to primary source ↓implementationPDF p. 14, Appendix F, architecture listings and training pseudocode
The architecture includes two Q networks; the state encoder has a 256-unit hidden layer and other MLPs have two 512-unit hidden layers. Pseudocode loops over range(H), but leaves compute_td and target-network updating as unexpanded helpers.
Go to primary source ↓hyperparametersPDF p. 16, Table 4; p. 17, Table 7
Table 4 specifies discount 0.99, temporal weight 0.5, loss coefficients 0.5/0.1/2, Adam, 5,000 seed steps, prioritized replay, task-dependent learning rates/batches, and target updates every two steps. Action-repeat values are 2 for Humanoid/Dog/Walker/Finger, 8 for Cartpole, 4 for other DMControl tasks, and 1 for Meta-World.
Go to primary source ↓latencyPDF p. 16, Appendix H; p. 17, Figure 11 and Appendix H continuation
Inference timing on one RTX3090 is approximately 20 ms per decision at the default Quadruped Run budget and approximately 12 ms at horizon one. The figure plots policy latency near 1 ms, whereas the prose calls the policy nearly six times faster than planning; these are not reconciled in the supplied paper.
Go to primary source ↓boundariesPDF p. 8, comparison and generalization discussion; p. 12, Appendix A; p. 17, Appendix I; p. 19, Figure 14 caption
The authors report weaker sample efficiency on Finger Turn Hard than SAC/LOOP and expect less benefit from unrelated-task sharing than general-purpose models. Transfer tests use related tasks; Meta-World v2 results are not directly comparable to v1.
Go to primary source ↓baseline-protocolPDF p. 6, Baselines; p. 15, Appendix G, MPC:sim paragraph
MPC:sim has ground-truth dynamics but no terminal value and uses horizon 10, 200 candidates, four iterations and 20 elites. This differs from TD-MPC in planning budget as well as model and value components.
Go to primary source ↓taxonomyPDF pp. 3–5, Algorithms 1–2 and Eqs. (6)–(11); p. 13, Table 3, TD-MPC row
The architecture is a collection of encoder/dynamics/reward/value/policy modules with sampling-based continuous control. Dynamics predicts future latents conditioned on candidate actions; actions come from optimization aided by a policy. Joint training does not establish a single joint future/action generative model.
Go to primary source ↓aggregatePDF p. 18, Figure 12 and Appendix K
Aggregate results use five seeds after 500k environment steps and stratified percentile-bootstrap confidence intervals. The caption expands IQM as interquantile median, while Appendix K calls it interquantile mean; this terminology differs within the source.
Go to primary source ↓8.5 Primary sources
Temporal Difference Learning for Model Predictive Control ↗
PDF · 13,369 extracted words
Source fingerprint
1027118649680e5ba1bbb93d9a7b95ddf47f3059e6c14809616c0bb7668b2cd6