World Model-based Perception for Visual Legged Locomotion
1. Paper overview
In one sentence: WMP turns a sensory world model into recurrent context for a separate locomotion policy, improving difficult terrain traversal while trading perception-update speed against onboard computation. e02e04e06e10e16
| At a glance | What to know |
|---|---|
| Research problem | Source description A forward-facing camera sees terrain before the feet reach it, so locomotion requires memory as well as immediate proprioception. Teacher–student methods introduce imitation error and depend on privileged representations: the paper's sparse ground scandots miss precise Tilt boundaries and overhead Crawl obstacles. WMP asks whether learning to predict sensory observations can supply a useful control representation directly. e02e03e04 |
| Core mechanism | Source description WMP combines an RSSM perception module with a faster actor–critic controller in one simultaneous training stage. A stop-gradient separates policy optimization from world-model representation learning, despite the authors' description of the overall framework as end-to-end. e04e05e06 |
| A key reported result | Physical A1 traversal at difficult obstacle settings: WMP: 85 cm Gap 40%; 55 cm Climb 60%; 28 cm Tilt 20%. Success rate ↑. Figure 6: onboard execution, ten trials at each tested terrain difficulty; rates read from plotted markers. Student and WMP w/o Prop: 0% at the 85 cm Gap and 55 cm Climb. WMP w/o Prop: 0% at 28 cm Tilt; Student is not applicable to Tilt. These are executed robot outcomes. Reported maximum traversable sizes coexist with frequent failures; ten trials and absent uncertainty intervals limit reliability claims. e10e15e16 |
| Reading caution | Reader analysis Real-world depth predictions use an initial observation and a supplied action sequence without intermediate depth. Figure 5 is qualitative: it shows useful passage geometry despite shape mismatch, but supplies no prediction-error metric or dataset-scale accuracy estimate. The t-SNE plot likewise does not establish that memory is a sufficient state for control. e13e14 |
Core contributions
- Source description
WMP combines an RSSM perception module with a faster actor–critic controller in one simultaneous training stage. A stop-gradient separates policy optimization from world-model representation learning, despite the authors' description of the overall framework as end-to-end. e04e05e06
- Reader analysis
The experiments combine modality ablations, temporal-design diagnostics, qualitative real-world predictions, and executed locomotion. The authors attribute transfer to useful historical representations; these experiments support the approach but do not isolate predictive learning from every alternative memory objective. e10e11e12e14e16
Figure 1. The privileged representation can omit the geometry that determines whether a body fits through an obstacle. Original paper, p. 1 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from the simulated scene on the left to its depth image on the right. In Tilt, red scandots sample the ground sparsely, while the green marker draws attention to distance from a barrier boundary. In Crawl, the overhead obstacle matters even though the ground beneath it remains traversable. The right column makes these visible surfaces available through depth. The accompanying footnote states a 10 cm horizontal terrain-mesh resolution in this simulation. This figure motivates the information supplied to a learner; it is not itself a comparison of learned predictions or successful control policies. e02e10e15
What it supports. A privileged teacher is only as informative as its representation. Here, a ground-based description is poorly suited to narrow or overhead clearances. Learning directly from depth avoids requiring the actor to imitate a teacher whose selected inputs omit those features, while introducing the harder task of learning useful visual memory.
Where the evidence stops. The illustrated limitation concerns this sparse scandot setup. It does not establish that every privileged representation fails on Tilt or Crawl, or that depth alone guarantees sufficient perception under real sensor noise.
2. Motivation
2.1 The problem and the proposed response
A forward-facing camera sees terrain before the feet reach it, so locomotion requires memory as well as immediate proprioception. Teacher–student methods introduce imitation error and depend on privileged representations: the paper's sparse ground scandots miss precise Tilt boundaries and overhead Crawl obstacles. WMP asks whether learning to predict sensory observations can supply a useful control representation directly. e02e03e04
2.2 What this reading follows
A quadruped's forward camera sees an obstacle before its feet reach it. WMP addresses that delay by learning a recurrent model of depth and proprioception, then passing its memory to a faster controller. The distinction between prediction and control matters: the world model learns from simulated trajectories, and PPO also learns from the simulator, without imagined rollouts. These visuals connect the representation choice to simulation ablations and actual A1 execution. They also qualify the headline obstacle sizes: occasional traversal of the hardest gap or climb is demonstrated, but the ten-trial success curves show that robust execution remains a separate challenge. e02e04e06e10e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The core Dual-system / Other mechanisms / Outside quadrants assessment is supported: an action-conditioned RSSM predicts sensory state, while a separate PPO actor generates actions from detached context. This is neither joint future/action generation nor inverse dynamics, and simultaneous training does not make it One Model. The broad Navigation label covers terrain locomotion rather than demonstrated route planning. Latent prediction applies, but the source describes reconstruction and KL training, not a JEPA objective. e03e04e05e06e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn a predictive memory before asking it to support control
The observation contains depth and proprioception, while the full simulated state also contains privileged quantities. WMP learns an RSSM in which recurrent state h carries history and stochastic state z incorporates the current observation. Reconstruction encourages these states to retain sensory information. The KL term aligns the observation-conditioned posterior with a prior that must work from recurrent context alone. Consequently, the model can attempt future sensory prediction when only actions are supplied. The actor, however, reads h rather than decoded images or z directly. Reader interpretation: prediction acts as a training signal for useful memory, while the practical control interface is a compact recurrent feature. Figure 5 demonstrates selected prediction examples; Figure 6 separately evaluates whether the resulting controller executes the task. e03e04e05e06e14e16
Figure 2. The policy acts between world-model updates using remembered context and fresh body measurements. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the purple recurrent state h_t. Its upward arrow supplies the policy, which also receives current proprioception and emits an action. Follow the horizontal route across several policy calls to h_{t+k}: Equation (4) specifies that this update uses h_t, z_t, and all intervening actions. Below, encoder E combines depth and proprioception with recurrent context to form z_t; decoder D reconstructs observations. The actor receives h, not z directly. Equations (6)–(7) add stop-gradient operations absent from this simplified drawing. Thus the arrow from memory to policy carries features, without allowing the PPO objective to update the world model through that connection. e04e05e06e07e11
What it supports. The architecture separates the timescale of perception from action. At the selected setting, the world model updates every five policy steps while the controller runs at 50 Hz. Predictive training supplies the representation, but action selection comes from the separately trained actor and its current proprioceptive feedback.
Where the evidence stops. The drawing omits the prior distribution, critic, and gradient notation. Read it with Equations (4)–(7); neither its decoder branch nor its forward arrows establish imagined-rollout policy training or inference-time planning.
5.2 Keep simultaneous training separate from imagined control
WMP learns the world model and controller in one stage, but the optimization routes differ. The RSSM receives reconstruction and KL losses. PPO trains the actor and critic using trajectories sampled in Isaac Gym, and the source explicitly rejects using world-model rollouts for that training. The actor consumes detached h plus current proprioception; the critic adds privileged information such as scandots and contact forces. Thus removing the teacher–student imitation stage does not remove all privileged training information. The held h can serve several policy steps because proprioception still arrives at the faster action rate. Reader interpretation: this is a modular perception-and-control arrangement whose objectives are separated by stop-gradient, even though the authors call the whole framework end-to-end. It supports the catalog's Dual-system assessment. e03e05e06e07e08
5.3 Use three different experiments for three different claims
Table I establishes comparisons under a shared simulator and reward setup, including the advantage over Student on Gap and the sensitivity to sensor removal. Figure 4 asks a different question: how frequently context should update and how much history training should expose to the model. Figure 6 then tests physical traversal, where depth filtering, latency, and onboard computation enter the system. Keeping those protocols separate prevents a high simulator return from being read as a real-robot success rate. Reader interpretation: the combined evidence supports useful visual memory, but stronger causal attribution needs a matched recurrent baseline with a different learning objective. It also needs repeated evaluation: the hardest reported gap and climb endpoints remain far from certain success, despite extending the demonstrated traversal range. e10e11e12e15e16
5.4 Training and inference
During training
The RSSM minimizes observation reconstruction loss plus a beta-weighted KL penalty aligning posterior and prior. Fixed trajectory segments train temporal memory. The policy and model learn simultaneously with separate objectives; PPO gradients do not train the RSSM through its policy input. e05e06e12
Training uses legged_gym and Isaac Gym with 4,096 simulated A1 robots across six terrains, a difficulty curriculum, physical randomization, and 100 ms depth latency. Actions run at 50 Hz; k = 5 gives a 0.1-second model interval. The selected training segment is 6.4 seconds. Rewards combine velocity tracking, obstacle-related penalties, and an AMP style term. Compared methods use the same environment and rewards. e07e09e10e11e12
During inference
Physical deployment runs onboard Jetson NX. An Intel D435i supplies 424 × 240 depth at 60 Hz; spatial/temporal filtering, cropping, and downsampling produce 64 × 64 inputs with 100 ms latency. PD gains are Kp = 40 and Kd = 1.0. The policy reads the RSSM context and current proprioception; no candidate-action planning procedure is specified. e06e15
5.5 Implementation flow
- Maintain predictive memory
Every k policy steps, a GRU advances the deterministic state using the previous recurrent state, stochastic state, and intervening actions. CNNs process depth; MLPs process proprioception. The observation-conditioned posterior and observation-free prior share the recurrent context. A decoder reconstructs observations from recurrent and stochastic states. e04
- Condition a faster controller
The actor uses current proprioception and the held recurrent state throughout each interval. It receives neither the stochastic state directly nor privileged inputs. The critic additionally receives privileged information. Both consume detached recurrent features, and PPO trains them on simulator data; imagined world-model rollouts are explicitly unused. e06
- Execute joint targets and receive feedback
The policy's action is added to the standing joint configuration to obtain target positions. A PD controller converts position and velocity errors to torques, with target joint velocities set to zero. New proprioception and periodically acquired depth close the sensor–action loop. e04e06e08
6. Experiments & results
World Model-based Perception (WMP) trains a recurrent sensor-prediction model alongside a separate locomotion policy in simulation. The model supplies remembered context to the controller; PPO uses simulator trajectories rather than imagined rollouts. Simulation comparisons and Unitree A1 trials support improved terrain traversal, while the hardest physical tests still have substantial failure rates.
6.1 Read the original evidence
Table I. WMP closes much of the reported teacher–student gap and benefits from both sensor modalities. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read a terrain column vertically, keeping the upper return block separate from lower tracking error. Higher return and lower error are preferred. The caption states that values average 100 trajectories across difficulties; boldface excludes Teacher from the ranking, and N/A means inapplicable rather than zero. Gap offers a direct Teacher–Student–WMP comparison. Tilt and Crawl instead test WMP against modality ablations because the selected teacher inputs are unsuitable there. Blind removes depth from the world model. WMP w/o Prop removes its proprioception, while the actor retains the direct proprioceptive route shown in Equation (6). These are distinct interventions. e05e06e10
What it supports. For Gap, WMP's return is 32.37 ± 8.24, close to Teacher's 32.80 ± 7.06 and above Student's 27.07 ± 12.38. On Tilt, removing world-model proprioception lowers mean return from 34.73 to 30.78. The table supports useful multimodal memory, with effects that vary substantially by terrain.
Where the evidence stops. The source leaves the ± statistic undefined and reports no significance tests. An input-removal ablation also changes the learned reconstruction problem, so these rows cannot attribute the gain exclusively to one loss term or to physical-parameter inference.
Figure 5. A simulation-trained model predicts selected real trajectories, including passage geometry that survives appearance mismatch. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each terrain group, compare the Real row with the Model row at the same column. Time labels advance from k through 20k, with larger gaps between later displayed samples. The small photographs identify the physical obstacles; the grayscale sequences show sensed and predicted depth. Section V-B specifies an initial observation and the trajectory's action sequence, with no intermediate depth images supplied for prediction. In Crawl, compare the opening's position and angle even where the obstacle surface differs. The continuation on page 6 explicitly acknowledges this shape mismatch and emphasizes the geometry of the space the robot can traverse. e06e14
What it supports. These examples show that a simulation-trained predictor can preserve useful structure in selected real trajectories. The strongest supported observation is qualitative agreement in task-relevant geometry, particularly the traversable opening. This is consistent with useful recurrent features, but it does not quantify general prediction accuracy or prove why the controller transfers.
Where the evidence stops. The action sequence is supplied to the predictor; it is not generated by visual planning in this experiment. No aggregate depth-error metric, uncertainty estimate, or trajectory-selection protocol accompanies the montage, so visual plausibility cannot substitute for executed success.
Figure 6. Physical execution improves, but maximum demonstrated obstacle sizes still carry substantial failure rates. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend in the Stair panel for all five plots: red circles are WMP, blue downward triangles Student, purple stars Blind, and orange diamonds WMP w/o Prop. Compare methods at the same horizontal tick. Difficulty rises with larger Stair, Gap and Climb dimensions, but with smaller Tilt width and Crawl height; those latter axes decrease toward the right. The caption states ten trials per setting. Read red endpoints as success probabilities estimated from these trials, rather than as binary capability labels. Student is absent from the Tilt/Crawl comparison because it is inapplicable, as explained in the simulation discussion. e10e15e16e17
What it supports. At the 85 cm Gap, WMP reaches 40% success while Student and WMP w/o Prop reach zero. At the 55 cm Climb, the corresponding WMP rate is 60%, again against zero for those two baselines. The 28 cm Tilt endpoint is only 20%, illustrating the difference between demonstrated traversal and dependable traversal.
Where the evidence stops. These are real onboard A1 executions, but ten trials provide limited precision and no confidence intervals are shown. Terrain-specific success curves do not establish reliability on arbitrary outdoor routes, other robot bodies, or unseen obstacle distributions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Simulated gap traversal Table I: gaps from 0–90 cm; 100 trajectories across difficulties; shared training environment and rewards. | WMP: return 32.37 ± 8.24; tracking error 0.26 ± 0.13. Return ↑; planar velocity tracking mean squared error ↓ | Teacher: 32.80 ± 7.06 and 0.23 ± 0.21; Student: 27.07 ± 12.38 and 0.35 ± 0.28; Blind return: 9.80 ± 9.51. WMP approaches the privileged teacher on reported means. The paper does not define the ± statistic or provide a significance test; these are neither success percentages nor proof of optimality. e10 |
| Proprioception ablation on simulated Tilt Table I: Tilt widths 32–28 cm, 100 trajectories across difficulties. The ablation removes proprioception from the world model, not the actor's direct proprioceptive input. | WMP: 34.73 ± 3.13; 0.02 ± 0.01. Return ↑; tracking error ↓ | WMP w/o Prop: 30.78 ± 7.22; 0.08 ± 0.10. Teacher and Student are N/A. This supports a contribution from multimodal world modeling. The ablation changes both the model's input and prediction task, so it does not isolate proprioceptive reconstruction alone. e06e10 |
| Physical A1 traversal at difficult obstacle settings Figure 6: onboard execution, ten trials at each tested terrain difficulty; rates read from plotted markers. | WMP: 85 cm Gap 40%; 55 cm Climb 60%; 28 cm Tilt 20%. Success rate ↑ | Student and WMP w/o Prop: 0% at the 85 cm Gap and 55 cm Climb. WMP w/o Prop: 0% at 28 cm Tilt; Student is not applicable to Tilt. These are executed robot outcomes. Reported maximum traversable sizes coexist with frequent failures; ten trials and absent uncertainty intervals limit reliability claims. e10e15e16 |
| Temporal design diagnostics Figure 4 and Section V-B: separate sweeps of model interval and world-model training segment length. | Return generally decreases as k increases from 2 to 30; the training-length curve peaks at the tested 64 × 0.1 s setting. Exact curve ordinates are not tabulated. Average simulation return ↑ | The authors select k = 5 given approximately 40 ms acquisition/computation time and select a 6.4-second training length. The plots motivate temporal design choices but provide no error bars or controlled separation of compute cost from representation quality. e11e12 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. Perception frequency and the length of training history affect different parts of the temporal design. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat these as separate sweeps. The left horizontal axis is k, the number of policy timesteps between model updates; moving right makes recurrent context update less frequently. The right axis measures trajectory-segment length in units of 0.1 seconds, so its 64 tick means 6.4 seconds. Both vertical axes show average return, but their ranges differ substantially, so the apparent slopes are not directly comparable. The source links the left sweep to onboard acquisition/computation cost and the right sweep to learning useful historical information. The lines connect tested configurations; they do not establish the behavior of every intermediate setting. e07e11e12
What it supports. Small update intervals generally perform better in simulation. The authors choose k = 5 to accommodate about 40 ms of acquisition/computation, and choose 6.4-second segments at the highest plotted point of the length sweep. Longer training history helps over much of the range, but the final longer segment does not improve the mean.
Where the evidence stops. Neither curve includes error bars or seed counts. These sweeps motivate the reported settings; they do not establish a universal optimum or independently measure how latency, compute budget, and optimization difficulty contribute to the observed changes.
7. Analysis & limitations
7.1 What the evidence leaves open
Real-world depth predictions use an initial observation and a supplied action sequence without intermediate depth. Figure 5 is qualitative: it shows useful passage geometry despite shape mismatch, but supplies no prediction-error metric or dataset-scale accuracy estimate. The t-SNE plot likewise does not establish that memory is a sufficient state for control. e13e14
Teacher/Student comparisons exclude Tilt and Crawl because their scandot representation is unsuitable. Outdoor evidence consists of snapshots and reported traversals, without quantified success rates. The conclusion proposes mixed real/simulated training and touch as future work, so those capabilities remain untested here. e10e17e18
7.2 Questions for discussion
- Would a capacity-matched recurrent perception model without the predictive KL objective retain WMP's advantage?
- How much of the deployment tradeoff comes from model-update frequency versus delayed depth arrival?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the terrain curriculum, physical randomization, AMP reference motions, reward weights, RSSM/PPO configuration, and deployment filtering. The PDF specifies platform and timing choices but does not give complete optimizer settings, model dimensions, training budget, seed count, training GPU configuration, software versions, or filter parameters; some details are deferred to cited work. e04e05e07e09e15
Before implementing the reward, resolve Equation (9): the printed exponential contains a positive squared-error term divided by sigma, while the prose describes velocity tracking and gives no sigma value or sign. Do not silently substitute a conventional negative exponent. Proposed controlled checks should also preserve depth latency and the actor's direct proprioception when changing model inputs. e06e07e09e10
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate proprioceptive input from its reconstruction objective
Reader-proposed check, not run: compare full WMP, the reported world-model proprioception-removal ablation, and a model that still encodes proprioception but omits only its reconstruction loss. Hold actor proprioception, RSSM capacity, PPO budget, rewards, curriculum, delay, and seeds fixed. Evaluate matched Gap and Tilt difficulties using return and tracking error. If preserving input while removing reconstruction preserves full performance, the claim that predicting proprioception drives the benefit weakens; degradation with preserved input would support a role for that objective. Report seed-level uncertainty rather than reinterpreting the paper's undefined ± values. e04e05e06e07e09e10
Check 2: Disentangle update frequency from delayed depth
Reader-proposed check, not run: retrain otherwise matched WMP configurations at k = 2, 5, and 10, crossing each with simulated depth delays of zero and 100 ms. Keep the policy at 50 Hz, training length at 6.4 seconds, and environment interactions and reward settings matched. Compare Gap/Tilt return and tracking error across seeds. If the smaller-k advantage survives realistic delay, faster recurrent updates provide an independent benefit; if it largely disappears, the idealized interval sweep overstates the deployment gain. Before any hardware comparison, measure acquisition-to-action latency and missed deadlines on the specified onboard platform. e07e09e10e11e12e15
8.3 Reading coverage
Visual audit: All seven supplied PDF pages were rendered and visually inspected, and all four text chunks were read individually. Page 1 verifies title, eight authors, affiliations, v1 date and Figure 1; page 2 supports the problem formulation and simultaneous-training description. Page 3 supplies Figure 2, RSSM/loss equations and detached actor/critic inputs; page 4 supplies Table I, simulation configuration, action execution and reward equations. Page 5 supplies t-SNE, both timing diagnostics and depth-prediction examples; page 6 supplies their interpretation, hardware/filtering details, physical success curves, outdoor snapshots and conclusion. Page 7 contains the complete reference list and no appendix. Figure 2's forward arrows were checked against Equation (4), and its omitted stop-gradient notation was supplied from Equations (6)–(7). Equation (9)'s printed reward-sign ambiguity remains explicit in the base report. Every final crop was inspected for readability, complete scientific labels, and faithful framing. Separate supplements, video and code remain outside this reading; the ICRA 2025 edition was not compared.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, author block, abstract
- PDF pp. 1–2: I. Introduction
- PDF p. 2: II. Related Work; III. Preliminaries
- PDF pp. 2–4: IV. Method, including A. World Model Learning, B. Policy Learning, C. Training Details
- PDF pp. 4–6: V. Experimental Results, including A. Simulation Comparison, B. Empirical Study, C. Real-world Evaluation
- PDF p. 6: VI. Conclusion
- PDF p. 7: References, entries 1–46
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The preceding acquisition limitation was addressed by visually inspecting all seven PDF pages and all six final original crops.
- Separate supplemental material availability has not been fully verified.
- No separate supplement or video was supplied or reviewed. Code was not inspected and experiments were not reproduced.
- Version boundary: the title page identifies arXiv:2409.16784v1 [cs.RO], 25 September 2024. Its exact title and all eight authors match the supplied catalog. The catalog records ICRA 2025; the venue edition was not supplied, so changes between it and this preprint are unverified. This report reviews the supplied v1 only.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin stamp
Exact title and eight authors match the catalog; the observed artifact is arXiv:2409.16784v1, 25 Sep 2024. Affiliations are Shanghai Jiao Tong University and ByteDance Research, China.
Go to primary source ↓e02PDF p. 1, Section I; Figure 1 and caption; scandot footnote
Forward vision requires remembering terrain. Sparse scandots miss precise Tilt boundaries and overhead Crawl obstacles; the simulation terrain mesh has 10 cm horizontal resolution.
Go to primary source ↓e03PDF p. 2, Section III, Equations (1)–(3); opening of Section IV
Locomotion is formulated as a POMDP. Observation combines proprioception and depth; simulator state also includes privileged information. Model and policy are learned simultaneously.
Go to primary source ↓e04PDF p. 3, Figure 2, Section IV-A and Equation (4)
RSSM recurrence consumes previous h, z, and intervening actions every k steps. Observation-conditioned posterior, prior and decoder are specified; GRU, CNN and MLP components are described.
Go to primary source ↓e05PDF p. 3, Section IV-A, Equation (5) and adjacent explanation
World-model learning combines observation reconstruction and beta-weighted posterior/prior KL on trajectory segments; further RSSM training details are deferred to Dreamer references.
Go to primary source ↓e06PDF p. 3, Section IV-B, Equations (6)–(7) and final paragraph
Actor and critic receive stop-gradient h and current proprioception; critic also receives privileged information. PPO uses simulator data, explicitly without imagined model rollouts.
Go to primary source ↓e07PDF pp. 3–4, Section IV-C, Environment
legged_gym/Isaac Gym supports 4,096 A1 instances on six terrains; curriculum, physical randomization, 50 Hz actions and 100 ms depth delay are described.
Go to primary source ↓e08PDF p. 4, Section IV-C, State and Action Space and Equation (8)
Proprioception has 45 dimensions; depth is 64 × 64 with 58° × 58° view. Twelve-dimensional actions offset standing joint positions; a PD controller uses zero target joint velocity.
Go to primary source ↓e09PDF p. 4, Section IV-C, Reward Function, Equations (9)–(11)
Tracking, obstacle penalties and an AMP discriminator-based style reward are described. Equation (9) prints a positive squared-error exponent divided by sigma; its value/sign and complete reward settings are not supplied.
Go to primary source ↓e10PDF p. 4, Table I and Section V-A; PDF p. 5, Section V-A continuation
Table averages 100 trajectories over difficulties, defines N/A and excludes Teacher from boldface ranking. Gap and Tilt entries support the reported results. Blind removes depth; w/o Prop removes world-model proprioception. Teacher/Student are inapplicable to Tilt/Crawl. The ± statistic is not defined.
Go to primary source ↓e11PDF p. 5, Figure 4 left and Section V-B, Model Interval
Interval sweep covers k = 2–30; smaller intervals generally yield higher return. Around 40 ms acquisition/computation motivates k = 5, equivalent to 0.1 s.
Go to primary source ↓e12PDF p. 5, Figure 4 right and Section V-B, Training Length
Trajectory-length sweep is labeled in 0.1-second units; its highest plotted return is at 64 units. Authors describe one second as acceptable and select 6.4 seconds elsewhere.
Go to primary source ↓e13PDF p. 5, Figure 3 and Section V-B, Recurrent State Visualization
t-SNE displays terrain-associated recurrent-state clusters; authors note overlap between Slope and Climb and interpret separation as useful terrain representation.
Go to primary source ↓e14PDF p. 5, Figure 5 and Real World Prediction; PDF p. 6, opening continuation
Initial observation and action sequences generate predictions without intermediate depth images. Gap, Climb, Tilt and Crawl examples are qualitative; the text acknowledges Crawl shape mismatch while describing preserved passage position/angle.
Go to primary source ↓e15PDF p. 6, Section V-C, first paragraph
Onboard Jetson NX and Intel D435i are specified, with 60 Hz 424 × 240 depth, spatial/temporal filtering, 64 × 64 processing, 100 ms delay and PD gains 40/1.0. Five terrain types are evaluated; Slope is excluded as too easy.
Go to primary source ↓e16PDF p. 6, Figure 6, Gap/Climb/Tilt panels, legend and caption; Section V-C
Ten trials underlie success rates. WMP markers show 0.4 at Gap 85 cm, 0.6 at Climb 55 cm and 0.2 at Tilt 28 cm. Student and w/o Prop are zero at those Gap/Climb endpoints; w/o Prop is zero at Tilt 28 cm.
Go to primary source ↓e17PDF p. 6, Figure 7 and outdoor-evaluation paragraph
Outdoor snapshots accompany reported stair, platform, grass and gravel traversal, including platforms up to 45 cm; no outdoor trial count or success-rate table is given.
Go to primary source ↓e18PDF p. 6, Section VI, Conclusion
The authors attribute benefits to extracting historical perceptual information and propose mixed simulated/real training and tactile perception as future work.
Go to primary source ↓8.5 Primary sources
World Model-based Perception for Visual Legged Locomotion ↗
PDF · 5,945 extracted words
Source fingerprint
2df0291956fd21ce758dd3fa3d69e768de0906034e1b4d04e8650817e0632ed8