Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
1. Paper overview
In one sentence: Ensemble disagreement makes long imagined rollouts useful for offline locomotion learning, but successful control still depends on penalty calibration and diverse data. e-identitye-dynamicse-policye-diagnostice-mixture
| At a glance | What to know |
|---|---|
| Research problem | Source description Offline robotic learning cannot repair its world model by collecting new transitions as the policy changes. Partial observation and narrow behavior-policy coverage therefore make long imagined rollouts vulnerable to compounding error. The paper asks whether autoregressive dynamics, epistemic uncertainty, and stable policy optimization can together make fixed-data learning useful on physical quadrupeds and humanoids. e-problem |
| Core mechanism | Source description RWM-U extends RWM with Gaussian ensemble prediction heads behind a shared recurrent extractor, separating predicted observation noise from disagreement associated with limited data. e-dynamicse-uncertainty |
| A key reported result | ANYmal D hardware velocity tracking with offline data mixtures: 0.91 ± 0.03 Normalized policy performance; higher is better. Physical evaluation after offline training on 800K simulated Mixed transitions plus 200K real transitions; each Table 1 mixture totals 1M. Sim-only offline: 0.82 ± 0.02; collecting policy: 0.79 ± 0.02; online model-free policy trained in simulation: 0.88 ± 0.01. The best tested mixture exceeds these reported means. This is a normalized score, not a success percentage. The ± statistic, normalization formula, and hardware evaluation count are not defined; statistical significance is not established. e-mixture |
| Reading caution | Author claim The authors acknowledge that uncertainty cannot eliminate missing coverage or long-horizon error. Real data omit risky but informative experience; complementary simulation remains useful even in a pipeline whose policy optimization is entirely offline. e-limits |
Core contributions
- Source description
RWM-U extends RWM with Gaussian ensemble prediction heads behind a shared recurrent extractor, separating predicted observation noise from disagreement associated with limited data. e-dynamicse-uncertainty
- Author claim
MOPO-PPO combines uncertainty-penalized imagination with PPO and demonstrates offline-trained policies on ANYmal D and Unitree G1. The authors frame this as the first uncertainty-penalized long-horizon offline MBRL deployment on physical hardware; priority is their claim. e-policye-hardwaree-design
Figure 2. Prediction and uncertainty enter a separate policy-learning loop. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow each column downward from its observation history through RWM-U to the Gaussian prediction heads. The curves represent member distributions: their individual spread and the separation of their means encode different uncertainties. Below them, the predicted observation and modified reward enter MOPO-PPO. The policy block at right connects back to the models; these connections are consistent with action conditioning in Eq. (2), although actions are not individually labeled in the graphic. Across columns, real observations give way to predicted observations in the rolling history. Read the lower arrows as information used for policy learning, not as a depiction of a single network jointly outputting actions and futures. e-dynamicse-uncertaintye-policye-architecture
What it supports. The architecture separates learning a predictor from learning a controller. Mean predictions support imagined transitions, while disagreement influences the reward used to train the controller. Algorithm 1 places model training before the policy-update loop, making the fixed model a source of synthetic experience rather than an online collector of corrective data.
Where the evidence stops. The diagram does not define uncertainty scalarization or all history indices. Eqs. (4)–(5) retain an unexplained k in one-step action-history notation. The mean/variance computation is clear; those implementation details remain unresolved.
2. Motivation
2.1 The problem and the proposed response
Offline robotic learning cannot repair its world model by collecting new transitions as the policy changes. Partial observation and narrow behavior-policy coverage therefore make long imagined rollouts vulnerable to compounding error. The paper asks whether autoregressive dynamics, epistemic uncertainty, and stable policy optimization can together make fixed-data learning useful on physical quadrupeds and humanoids. e-problem
2.2 What this reading follows
A robot policy can discover high rewards inside a learned simulator by exploiting the simulator’s mistakes. RWM-U addresses that problem by predicting both future proprioception and disagreement among ensemble heads; MOPO-PPO uses the disagreement to reduce the reward of unreliable imagined transitions. Read this paper as a study of where to trust learned dynamics. Its evidence moves from a forecasting diagnostic to penalty sweeps, simulation comparisons, and physical locomotion. The ANYmal D data-mixture table is particularly revealing: a moderate real-data contribution improves the reported score, while replacing still more simulation experience with real trajectories degrades it. This edition reviews the verified January 2026 arXiv v3 preprint. e-identitye-dynamicse-policye-diagnostice-mixture
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system classification is supported by a dynamics GRU/ensemble distinct from policy/value MLPs. Actions come from a policy trained through model-generated experience, rather than inverse dynamics or joint future/action prediction. Other mechanisms and Outside quadrants therefore fit the described mechanism. Policy post-training & WM-RL is supported on its world-model RL aspect; the algorithm does not establish post-training of a pretrained generalist policy. e-architecturee-policye-spaces
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Separate the dynamics target from the robot’s action
Start with the two observation spaces in Tables S2 and S5. RWM-U models proprioceptive evolution, including torques, from observation/action history; the policy additionally receives a velocity command and its previous action. Its output is a joint-position target. This separation explains how future prediction becomes useful for control without turning a future observation into an action through inverse dynamics. During model training, the predictor consumes its own earlier predictions and incurs multi-step discrepancy against the recorded trajectory. Privileged quantities such as contacts supply auxiliary prediction supervision. During policy learning, the already-trained model supplies imagined transitions, and PPO updates a different network. At physical deployment the learned policy receives robot feedback and emits targets; Algorithm 1 does not describe online model repair or deployment-time planning. e-spacese-dynamicse-policye-architecturee-problem
5.2 Ask what disagreement changes in the optimization
The important intervention occurs in the reward used to train the policy. Eq. (4) produces a mean observation prediction and an epistemic disagreement estimate; Eq. (5) subtracts a weighted version of that estimate from task reward. Figure 4 then supplies a causal diagnostic: some policies look excellent inside imagination but evaluate poorly, while very cautious policies barely move. My reading is that uncertainty acts as a learned trust cost on synthetic experience, not as a guarantee that every predicted transition is correct. The distinction between eight-step model training and 100-step policy rollouts makes this especially consequential: the policy is optimized on sequences longer than those directly supervised by the forecast objective. Figure 3 motivates the trust signal, but does not prove calibrated error bounds at every horizon. e-uncertaintye-policye-ablatione-diagnostice-model-traininge-policy-training
5.3 Interpret the data mixture as a coverage experiment
Table 1 holds total transition count fixed while changing the source of experience. The 800K/200K mixture gives the best reported score, but replacing more simulated trajectories with real trajectories makes performance worse. The authors’ explanation is that collected real trajectories are realistic yet avoid risky, informative states. Appendix A.4 makes the collection bias concrete: velocity commands are filtered through a safety boundary, with an origin-seeking fallback. My interpretation is that realism and coverage are coupled in this experiment, so the table cannot establish an intrinsically optimal real-data fraction. It also cannot tell whether adding real data while preserving all simulation examples would help. Reproduction should first resolve how the larger real-data columns relate to the stated 200K-transition collection, then measure coverage and control quality separately. e-mixturee-limitse-collection
5.4 Training and inference
During training
Model settings are history M=32, forecast N=8, ensemble B=5, forecast decay alpha=1, batch 1024, learning rate 1e-4, and 2500 iterations. Table S10 lists 6M transitions and approximately one hour of model training; this is distinct from the 1M-transition mixtures in Table 1. e-model-traininge-mixture
Policy learning uses 4096 imagined environments and 100 steps per iteration at 0.02 seconds, with lambda=1, learning rate 0.001, five epochs, four mini-batches, discount 0.99, and PPO clip 0.2. Both training tables list five seeds. e-policy-traininge-model-training
During inference
At deployment, the separate policy maps current robot observations and commands to joint-position targets: 12 for ANYmal D and 29 for G1. Observations provide execution feedback, but deployment is not an online data-collection/update loop in the reported offline protocol. No deployment-time model search or joint future/action generator is described. e-spacese-policye-hardwaree-problem
5.5 Implementation flow
- Represent robot dynamics
World-model observations comprise base linear/angular velocity, projected gravity, joint position, velocity, and torque. Policy inputs differ: they include commands and previous actions. Contacts and related privileged quantities provide auxiliary prediction supervision, rather than defining an inverse-dynamics action decoder. e-spaces
- Predict autoregressively
A shared GRU processes history, and ensemble heads predict Gaussian observation distributions. Predicted observations feed later predictions. The appendix specifies GRU hidden shape 256,256 and 128-wide ReLU heads; the policy and value networks are separate three-layer, 128-wide ELU MLPs. e-dynamicse-architecture
- Estimate trust in imagination
Use the mean across head means for the next observation and their variance for epistemic uncertainty. Within-head dispersion describes aleatoric uncertainty. The paper calls this long-horizon uncertainty propagation, but the displayed computation re-estimates disagreement along autoregressive predictions; it does not specify a full trajectory-distribution propagation algorithm. e-uncertaintye-diagnostic
- Optimize a policy with penalized rewards
Algorithm 1 first trains the model, then repeatedly seeds imagination with dataset observations, generates rollouts using the current policy, subtracts uncertainty penalties, and updates the policy with PPO. Task rewards separately encode velocity tracking and robot-specific motion/contact costs. No additional environment data enter this learning loop. e-policye-rewardse-problem
6. Experiments & results
RWM-U learns action-conditioned robotic dynamics from fixed logs and uses disagreement among prediction heads to discourage a separately trained PPO policy from exploiting unreliable imagined transitions. Its strongest evidence combines locomotion comparisons, a penalty sweep, and physical ANYmal D data-mixture results. The practical tradeoff is between exploiting useful model predictions and becoming too conservative to move; realism alone also fails when real data replace diverse simulation experience. The reviewed source is the January 2026 arXiv v3 preprint.
6.1 Read the original evidence
Figure 5. Locomotion with richer data is the clearest simulation advantage. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First choose a robot task, then compare algorithms within one dataset group. Random data come from a randomly initialized policy; Medium and Expert use partially and fully trained PPO policies; Mixed combines data from different training stages. The dark-blue bars are MOPO-PPO. Its most pronounced separation from the alternatives appears in Expert and Mixed locomotion. Table S9 explains the model-based controls: MBPO and MBPO-PPO use RWM with SAC and PPO respectively; MOPO and MOPO-PPO use RWM-U with those optimizers. This crossing helps locate benefits associated with both uncertainty-aware modeling and policy optimization, while CQL provides the model-free comparison. e-benchmarke-baselines
What it supports. The figure supports strong MOPO-PPO results on the two locomotion tasks, especially when data support richer behavior. Manipulation is less decisive: several bars are close, and competing bars exceed MOPO-PPO in some groups. The supported conclusion is task- and dataset-dependent, narrower than the surrounding prose’s claim of consistent wins.
Where the evidence stops. These are simulation episodic rewards, not physical success rates. Exact bar values and error-bar definitions are absent. The named model-based baselines are the paper’s matched RWM/RWM-U implementations, not necessarily their original published configurations.
Figure 7. Physical examples show the behavioral cost of both penalty extremes. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right along the time labels, then compare rows within the same robot. The upper three ANYmal D rows use lambda values 0.2, 2.0, and 1.0; the lower G1 rows use 0.1, 5.0, and 1.0. The green arrow indicates the commanded velocity target. Small inset robot views accompany the physical scenes and help relate commands to the motion shown. The authors interpret the weak-penalty sequences as unstable, the strong-penalty sequences as overly cautious, and the lambda=1 sequences as effective locomotion. Keep the robot-specific coefficients intact rather than assigning the quadruped sweep values to the humanoid. e-hardwaree-mixture
What it supports. This is physical execution evidence for both quadruped and humanoid policies trained through offline imagination. The selected sequences make the penalty tradeoff concrete: an optimization setting can fail through instability or through insufficient movement. They complement the reward plots by showing what those qualitatively different policy failures look like.
Where the evidence stops. Selected frames cannot establish a deployment failure rate, long-term reliability, or a quantitative G1 advantage. The caption interprets the behaviors, but repeated-trial statistics for these sequences are not provided; Table 1 quantifies only ANYmal D.
Table 1. A moderate real-data mixture gives the best tested hardware score. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top two rows together: each of the first four columns contains one million transitions, with real trajectories progressively replacing simulated trajectories. The final two columns are comparison policies, not additional mixtures. The best reported entry is 0.91±0.03 for 800K simulation plus 200K real transitions. Compare it with 0.82±0.02 for simulation-only offline learning, 0.79±0.02 for the collecting policy, and 0.88±0.01 for the online model-free policy trained in simulation. Moving further right within the mixture columns lowers the score to 0.78±0.02 and then 0.59±0.04. These normalized values should be read as scores, not percentages. e-mixturee-limitse-collection
What it supports. The best tested mixture improves the reported mean over both listed reference policies and the offline simulation-only condition. More real data is not automatically better: the authors argue that realism helps domain transfer while simulated experience preserves informative behavioral coverage. Because the total size is fixed, increased realism and decreased simulation coverage change together.
Where the evidence stops. The table leaves the ± statistic and hardware trial count undefined. A.4 describes 200K collected real transitions, but does not explain the 400K/600K real-data columns. Neither statistical significance nor a resolved dataset-construction protocol follows from this table.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| ANYmal D hardware velocity tracking with offline data mixtures Physical evaluation after offline training on 800K simulated Mixed transitions plus 200K real transitions; each Table 1 mixture totals 1M. | 0.91 ± 0.03 Normalized policy performance; higher is better | Sim-only offline: 0.82 ± 0.02; collecting policy: 0.79 ± 0.02; online model-free policy trained in simulation: 0.88 ± 0.01. The best tested mixture exceeds these reported means. This is a normalized score, not a success percentage. The ± statistic, normalization formula, and hardware evaluation count are not defined; statistical significance is not established. e-mixture |
| Real-data proportion sensitivity on ANYmal D Same Table 1 hardware comparison, replacing simulation transitions with real transitions at fixed total size. | 600K Sim/400K Real: 0.78 ± 0.02; 400K Sim/600K Real: 0.59 ± 0.04. Normalized policy performance | 800K Sim/200K Real: 0.91 ± 0.03. More real data does not monotonically improve performance. The authors attribute deterioration to restricted real-world coverage; the comparison also removes simulation data, so it does not isolate real-data quality alone. e-mixturee-limits |
| Offline simulation performance across robot tasks and dataset qualities Reach-Franka and two velocity-tracking tasks; Random, Medium, Expert, and Mixed datasets; CQL plus four model-based implementations. | Graphical results show the largest MOPO-PPO advantages on Expert/Mixed locomotion; exact bar values are not supplied. Normalized episodic reward in Figure 5 | Table S9 crosses RWM/RWM-U with SAC/PPO; uncertainty-unaware methods are evaluated offline. The plot supports strong locomotion gains, not a win in every group: manipulation includes near ties and groups with taller competing bars. These are simulation rewards, separate from physical Table 1. e-benchmarke-baselines |
| Uncertainty penalty sensitivity in locomotion Figure 4 sweeps penalties for ANYmal D and G1, comparing imagined training reward with final evaluation on real dynamics. | The reported useful setting is lambda=1.0; weaker and stronger settings transfer worse. Imagination reward, epistemic uncertainty, and evaluation reward | Small penalties permit high imagined reward with poor evaluation; excessive penalties suppress motion. This diagnoses a tunable trust tradeoff, not a universal optimal coefficient. Figure 7 supplies qualitative physical examples with robot-specific penalty values. e-ablatione-hardware |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 3. Disagreement tracks the broad growth of rollout error. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The left stack plots base linear velocity v in meters per second, angular velocity omega in radians per second, and joint positions q in radians. Solid traces are observations and dashed traces are predictions. The vertical marker at t=32 identifies the transition from recorded history to autoregressive prediction. On the right, compare the gray prediction-error curve with dark-blue epistemic uncertainty and light-blue aleatoric uncertainty. Section 5.1 specifies 50 Hz data, a 32-step history, an eight-step training forecast, and five ensemble members. The plotted rollout extends beyond the training forecast horizon. This tests whether the uncertainty signal remains informative as predicted inputs increasingly replace measured ones. e-diagnostice-dynamicse-uncertainty
What it supports. The epistemic curve follows the broad error trend during the displayed long rollout, whereas the aleatoric curve remains low. That observation motivates penalizing ensemble disagreement during policy learning. It provides a useful diagnostic connection between prediction reliability and the proposed reward penalty, rather than directly measuring the quality of an executed policy.
Where the evidence stops. The source supplies no correlation coefficient or calibrated coverage. Section 5.1 plots aleatoric uncertainty as mean predicted standard deviation, while Eq. (3) specifies Gaussian variance; the three curves should not be treated as identically defined error measures.
Figure 4. High imagined reward alone does not identify a useful policy. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat each robot’s two plots as a pair. Its left plot shows imagination reward during optimization and separate dots at the eval position for the final evaluation on real dynamics. Its right plot shows epistemic uncertainty during learning. Match lines by the lambda legend, noting that the two robots use different sweep values. The blue lambda=1 curve combines substantial imagined reward with a high evaluation dot. Several weaker-penalty curves reach high training rewards yet end at much lower evaluation dots. Very strong penalties lower uncertainty further but also suppress useful behavior. The comparison therefore depends on reading reward, uncertainty, and evaluation together. e-ablatione-hardwaree-mixture
What it supports. The sweep supports a middle ground between exploiting model errors and refusing useful motion. The reported lambda=1 setting works well in both plotted tasks, while low uncertainty alone does not guarantee good control. Together with the hardware sequences, this makes penalty strength a central practical parameter of the method.
Where the evidence stops. The caption calls the dots evaluation on real dynamics; they are distinct from imagination and should not be merged with Table 1’s hardware mixture scores. Error-bar conventions and an evaluation-free procedure for selecting lambda are not specified.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge that uncertainty cannot eliminate missing coverage or long-horizon error. Real data omit risky but informative experience; complementary simulation remains useful even in a pipeline whose policy optimization is entirely offline. e-limits
Figure 3 supports a qualitative association between disagreement and error, without a correlation coefficient, calibrated coverage, or proof against shared ensemble bias. Shared recurrent features make unmeasured common-mode error a relevant reader concern. e-diagnostice-dynamics
The physical data-mixture table covers ANYmal D only; G1 hardware evidence is qualitative. Broad claims about outperforming every baseline exceed some Figure 5 groups, and the appendix’s rationale for PPO does not prove all off-policy methods unsuitable for robotics. e-mixturee-hardwaree-benchmarke-design
7.2 Questions for discussion
- Does disagreement still predict error when all heads share the same biased recurrent representation?
- Would retaining simulation failure coverage remove the deterioration attributed to larger real-data mixtures?
- How should lambda be chosen without repeatedly using hardware evaluation feedback?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the trajectory datasets, specified observation/auxiliary targets, reward construction, and architectures. The reported platform is PyTorch 2.4.0/CUDA 12.6 on an NVIDIA RTX 4090. The preprint promises code upon acceptance; that statement alone does not establish an available release. e-spacese-rewardse-architecturee-model-traininge-repro
Clarify bootstrap resampling, the Gaussian training loss, observation normalization, variance scalarization, initialization histories, and evaluation statistics. Table S11’s buffer size |D|=1000 is not reconciled with offline transition counts. These omissions prevent an exact implementation from the text alone. e-dynamicse-uncertaintye-policye-model-traininge-policy-traininge-mixture
Collection uses command rejection against an odometry/LiDAR safety boundary and an origin-seeking fallback. A.4 reports about one hour and 200K transitions, but Table 1 includes 400K and 600K real transitions without explaining their derivation. A reproduction should resolve this discrepancy before constructing the mixtures. e-collectione-mixture
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does correctly aligned disagreement improve control?
Reader-proposed check, not an executed experiment: train one RWM-U on a fixed ANYmal D dataset, then hold its weights, PPO settings, initialization histories, and 100-step rollout horizon fixed. Compare no penalty, the aligned lambda=1 disagreement penalty, and a penalty shuffled among imagined transitions while preserving its marginal distribution. Repeat across seeds and evaluate only after training in the same independent dynamics environment. Record imagined reward, evaluation reward, falls, and disagreement versus prediction error by horizon. If shuffled penalties match the aligned penalty, the specific claim that local disagreement identifies untrustworthy transitions would be weakened even if generic regularization still helps. e-policye-diagnostice-ablatione-model-traininge-policy-training
Check 2: Separate real-data realism from lost simulation coverage
Reader-proposed check, not an executed experiment: first resolve the 200K versus larger real-data counts and document unique trajectories. Recreate the four fixed-size Table 1 mixtures with matched sampling and seeds. Add a control that holds simulation data fixed while adding real data, plus a matched-size simulation-only control. Use identical rewards, architectures, and policy-training budgets. Measure coverage of contacts, failures, and recovery states alongside normalized evaluation return and failure counts, with an explicit uncertainty statistic. If degradation disappears when informative simulation coverage is retained, that supports the authors’ diversity explanation; if it persists, investigate domain mixing and model fitting rather than attributing it solely to missing risky states. e-mixturee-limitse-collectione-spacese-rewardse-architecturee-model-traininge-policy-training
8.3 Reading coverage
Visual audit: The title/version/author page and every page supporting retained scientific claims were rendered and visually inspected. This includes Figures 1–7 and S8, Algorithm 1, Eqs. (1)–(5), Table 1, Tables S2–S11, limitations, reproducibility statements, and Appendix A.1–A.6. All six final original crops were individually viewed with labels and legends retained. Figure 2 arrows were cross-checked against Eqs. (2), (4), (5), its caption, and Algorithm 1; its unlabeled action connections do not resolve the source’s action-history indexing ambiguity. Reference-only pages 12–13 were read in the complete text pass. Separate videos, code, and supplements were not inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–7 (including all subsections)
- Ethics Statement; Reproducibility Statement; Acknowledgments; References
- Appendix A.1–A.6, including task spaces, rewards, architectures, training, collection, and online/offline distinction
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Official arXiv title and Chenhao Li, Andreas Krause, Marco Hutter match the catalog. This is an author preprint version; equivalence to the camera-ready version has not been established.
- Identity notes: the title and all three authors match. The inspected artifact is arXiv:2504.16680v3, dated 8 January 2026 and marked Preprint, under review. The catalog names an ICLR 2026 Workshop source; that edition was not supplied or compared.
- All six supplied text chunks were read. The extraction-only figure limitation was addressed by inspecting the PDF and original crops. All scientific-content pages were visually inspected; reference-only pages 12–13 were read as text.
- No code, project website, separate supplement, or videos were inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title and author block; arXiv margin stamp
The title matches the supplied observed title. Authors are Chenhao Li, Andreas Krause, and Marco Hutter, affiliated with ETH Zurich, Switzerland. The artifact is marked Preprint, under review and arXiv:2504.16680v3, 8 January 2026.
Go to primary source ↓e-problemPDF pp. 1–3, Abstract and Sections 1 and 3.1; pp. 19–20, Section A.6
The fully offline setting fixes the dataset before model and policy learning, preventing corrective environment interaction during training. Partial observation, restricted coverage, and compounding prediction errors motivate uncertainty-aware imagination.
Go to primary source ↓e-dynamicsPDF p. 4, Sections 3.2 and 4.1, Eqs. (1)–(3)
Autoregressive prediction uses observation/action history and prior predictions. A shared recurrent feature extractor feeds bootstrap observation heads that predict Gaussian means and variances. Training averages multi-step discrepancy over ensemble members; the exact discrepancy and bootstrap procedure are not expanded here.
Go to primary source ↓e-uncertaintyPDF p. 5, Figure 2 and Eq. (4); p. 4, Section 4.1
The mean of ensemble means supplies the next observation; variance across those means estimates epistemic uncertainty. Within-member variance represents aleatoric uncertainty. The schematic routes predictions and penalized rewards to MOPO-PPO and the policy back to the model. Eq. (4) retains an unexplained k in its one-step action-history subscript.
Go to primary source ↓e-policyPDF p. 5, Section 4.2 and Eq. (5); p. 6, Algorithm 1, lines 3–8
Train the model on the fixed dataset, initialize imagination from dataset observations, roll out with the current policy, subtract lambda times uncertainty from reward, and update the policy using PPO. The model-training step precedes the policy loop. Eq. (5) does not specify how observation-wise variance is reduced to a scalar reward penalty.
Go to primary source ↓e-diagnosticPDF p. 6, Figure 3 and Section 5.1; p. 7, continuation of Section 5.1
ANYmal D forecasting uses 50 Hz data, history 32, forecast training horizon 8, and five ensemble members. The plotted rollout switches to predictions at t=32. Epistemic uncertainty follows the broad prediction-error trend; aleatoric uncertainty is plotted as mean predicted standard deviation and stays low. No numerical correlation coefficient or calibration coverage is supplied.
Go to primary source ↓e-ablationPDF p. 7, Figure 4 and Section 5.2
Penalty sweeps compare imagination reward and uncertainty with final evaluation dots on real dynamics. The reported useful setting is lambda=1.0; small penalties produce poor transfer despite high imagined rewards, while large penalties inhibit motion. The two robot panels use different sweep values.
Go to primary source ↓e-benchmarkPDF pp. 7–8, Section 5.3 and Figure 5
Simulation benchmarks cover Reach-Franka, Velocity-ANYmal-D, and Velocity-Unitree-G1 with Random, Medium, Expert, and Mixed datasets. Figure 5 shows especially large MOPO-PPO advantages for Expert/Mixed locomotion, but does not show it winning every manipulation/data group. Exact bar values and error-bar definitions are not tabulated.
Go to primary source ↓e-hardwarePDF p. 9, Figure 7 and caption; p. 10, Section 5.4, paragraph after Table 1
Physical deployment sequences contrast ANYmal D penalties 0.2, 2.0, and 1.0, and G1 penalties 0.1, 5.0, and 1.0. The authors describe unstable behavior with weak penalties, little motion with excessive penalties, and effective locomotion at 1.0. These are selected sequences, not a failure-rate table.
Go to primary source ↓e-mixturePDF p. 8, Section 5.4; p. 10, Section 5.4 and Table 1, all columns
ANYmal D normalized hardware scores are 0.82±0.02, 0.91±0.03, 0.78±0.02, and 0.59±0.04 for Sim/Real counts 1M/0, 800K/200K, 600K/400K, and 400K/600K. Collecting and online model-free policies score 0.79±0.02 and 0.88±0.01. Simulation data are Mixed data under heavy randomization. The table does not define the ± statistic, normalization formula, or hardware trial count.
Go to primary source ↓e-limitsPDF p. 10, Section 6
The authors retain dataset-coverage and long-horizon error limitations. Real trajectories lack risky but informative transitions, making simulation a complementary source of behavioral diversity.
Go to primary source ↓e-spacesPDF p. 14, Section A.1.1 and Tables S2–S4; p. 15, Table S5
World-model observations include base velocities, projected gravity, joint positions, velocities, and torques. Contacts and related privileged quantities provide auxiliary prediction supervision. Policies additionally receive velocity commands and previous actions; their actions are joint-position targets, with 12 dimensions for ANYmal D and 29 for G1.
Go to primary source ↓e-rewardsPDF pp. 14–16, Section A.1.2; p. 15, Table S6
Task rewards combine velocity tracking with motion, effort, contact, orientation, and robot-specific shaping terms. Reward weights differ between robots, including action-rate, foot-clearance, and joint-deviation terms.
Go to primary source ↓e-architecturePDF p. 16, Sections A.2.1–A.2.2 and Tables S7–S8
The model uses a GRU base with hidden shape 256,256 and MLP heads of width 128 with ReLU. Policy and value function are separate MLPs with three 128-wide hidden layers and ELU.
Go to primary source ↓e-baselinesPDF p. 17, Section A.2.3 and Table S9
The comparison crosses RWM versus RWM-U with SAC versus PPO: MBPO=RWM/SAC, MBPO-PPO=RWM/PPO, MOPO=RWM-U/SAC, and MOPO-PPO=RWM-U/PPO. These are the paper’s controlled implementations.
Go to primary source ↓e-model-trainingPDF p. 17, Section A.3 and Table S10
Reported software/hardware are PyTorch 2.4.0, CUDA 12.6, and an NVIDIA RTX 4090. Model training lists 2500 iterations, learning rate 1e-4, weight decay 1e-5, batch 1024, B=5, M=32, N=8, alpha=1.0, 6M training transitions, approximately one hour, and five seeds.
Go to primary source ↓e-policy-trainingPDF p. 18, Table S11
MOPO-PPO uses 4096 imagination environments, 100 steps per iteration at 0.02 seconds, lambda=1.0, 2500 iterations, learning rate 0.001, five learning epochs, four mini-batches, KL target 0.01, discount 0.99, clip range 0.2, entropy coefficient 0.005, and five seeds. Buffer size is listed as |D|=1000 without explaining its relation to the offline transition count.
Go to primary source ↓e-collectionPDF p. 17, Section A.4; p. 18, Figure S8 and caption
Collection randomly samples velocity commands and rejects predicted final poses outside an odometry-based rectangular boundary; a fallback drives toward the origin and joystick override can recenter the region. Odometry fuses legged odometry and LiDAR. Approximately one hour yields 200K transitions; this section does not explain the larger real-data counts in Table 1.
Go to primary source ↓e-reproPDF p. 11, Ethics and Reproducibility Statements
The source says failure events are modeled to terminate unsafe rollouts and code will be open-sourced upon acceptance. The statement is a release intention, not evidence of an inspected implementation.
Go to primary source ↓e-designPDF pp. 18–19, Section A.5; pp. 19–20, Section A.6
The authors motivate PPO through practical stability in parallel locomotion and distinguish long-horizon, fixed-data learning from online methods that collect corrective experience. These arguments do not establish universal superiority over off-policy algorithms.
Go to primary source ↓8.5 Primary sources
Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots ↗
PDF · 8,417 extracted words
Source fingerprint
b0906b92791f4bf4c1b0758c1a53ce1ab1fb62baee1e556995b34c8fb7e74ddb