X-MOBILITY: End-To-End Generalizable Navigation via World Modeling
1. Paper overview
In one sentence: A recurrent world model pretrained on random actions supplies a route-conditioned imitation policy with useful navigation state, while its forecasting and transfer evidence remain bounded by the tested scenes. e01e03e06e08e09e11e12e15e16e19
| At a glance | What to know |
|---|---|
| Research problem | Source description A navigation policy must act under partial observability and generalize beyond expert demonstrations. The paper uses a learned probabilistic state instead of a manually specified navigation state, while assuming the imitation teacher approximates an optimal POMDP policy. e02 |
| Core mechanism | |
| A key reported result | Closed-loop navigation in simulation: Warehouse: 96, 37.7, 0.206. Random obstacles: 68, 40.21, 0.186. Success rate SR (%) ↑; weighted trip time WTT (s; trip time divided by success rate) ↓; average absolute angular acceleration AA (rad/s²) ↓. Ten warehouse scenarios, including five unseen-environment cases; 50 separately generated random-obstacle corridor scenes. BC: 92/40.4/0.219 and 56/42.72/0.219; Nav2: 58/68.41/0.606 and 34/74.94/0.391. The full model beats these baselines in both aggregates. Warehouse results pool seen and unseen cases; trial-allocation wording is ambiguous. e10e12 |
| Reading caution |
Core contributions
Figure 2. Observation updates and action-conditioned forecasts share a latent interface with the policy. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at panel (a): DINOv2 image features and the robot-state MLP produce the observation embedding. In panel (b), the dashed observation input is explicitly marked for the estimator only. Compare panels (c) and (d): both take previous history and action, but only estimation receives the observation. Their GRUs update history from the sampled state. Panel (e) concatenates previous history and current state into the decoder latent. Follow its separate arrows to RGB reconstruction, semantic decoding and self-attention. The route travels through VectorNet into that attention block before action decoding. These arrows agree with the observation dependence in Eqs. (1) and (3). e03e04e05e06e07e17
What it supports. The architecture supports both updating a belief from evidence and rolling it forward without a new image. Direct action decoding uses the belief together with route guidance. RGB and semantic heads provide representation-learning signals; the policy can run without producing those images.
Where the evidence stops. Eq. (2) has an anomalous nested history subscript; this explanation follows the surrounding text and Figure 2's previous-history label. The diagram supplies no candidate-action search or inference-time optimization procedure.
2. Motivation
2.1 The problem and the proposed response
A navigation policy must act under partial observability and generalize beyond expert demonstrations. The paper uses a learned probabilistic state instead of a manually specified navigation state, while assuming the imitation teacher approximates an optimal POMDP policy. e02
2.2 What this reading follows
X-MOBILITY asks how a navigation robot can learn from experience that contains no expert route or path. Its answer separates learning what actions do from learning which commands a teacher would choose. Random-action simulation trains a probabilistic recurrent world model; teacher demonstrations then train the policy alongside it. The six source excerpts below connect this information flow to the data requirements, offline errors, closed-loop ablations, future-state diagnostics and real-robot trials. Read the results by evaluation setting: improved imitation error, accurate navigable-surface prediction and successful physical navigation establish different things. This edition covers the supplied July 2025 arXiv v3, including its model appendix. e01e03e06e08e09e11e12e15e16e19
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The separate world-model and route-conditioned policy networks support the recorded Dual-system assessment despite joint second-stage training. Forward latent prediction plus direct imitation fits Other mechanisms and Outside quadrants, rather than inverse dynamics or joint future/action generation. Navigation and latent prediction are supported; this is not specifically a JEPA objective. e03e05e06e07e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn from commands without learning to imitate them
Begin with the distinction between an executed command and an expert target. The random-action dataset contains commands, images, speed and semantic labels, so it records what happened after motion even though it offers no route or expert path. X-MOBILITY first disables the policy and uses that experience for world modeling. Observation-conditioned estimation supplies a state distribution; an action-conditioned predictor learns to match it through KL divergence. RGB reconstruction and semantic decoding force the latent state to retain observable content. The second stage introduces Nav2 demonstrations and trains the policy alongside the world model. This is staged supervised learning, not a reported reinforcement-learning optimization of the POMDP reward. As a reader interpretation, the useful separation is between learning consequences from broad experience and selecting commands from narrower demonstrations. e02e03e05e06e08e09
Table I. Random actions retain transition supervision even when expert navigation targets are missing. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally to distinguish missing expert guidance from missing action information. Both datasets have an image, speed, a semantic label and the action command that produced experience. Only teacher records have the local route and the target path. The shapes show seven semantic channels at the input image resolution, twenty two-dimensional route positions and five two-dimensional path positions. The six-component action command contains desired linear and angular velocities. Section III.A describes 160K random-action frames and 100K teacher frames. During the first training stage the policy is disabled, so the absent route and path do not prevent learning observation reconstruction and action-conditioned dynamics. e04e07e08e09e19
What it supports. This table explains the practical value of decoupling world modeling from imitation: unsuccessful or unguided motion can still teach transitions. However, these random-action records include simulator semantic labels. Their usefulness does not establish that the same recipe works on arbitrary unlabeled robot video.
Where the evidence stops. The speed row is scalar, consistent with the forward-speed encoder; nearby dataset prose also mentions angular velocity. Preserve that specification discrepancy. Randomization distributions and exact split assignments are not supplied.
5.2 Carry memory into control without requiring an imagined search
At a new observation, the estimator combines the prior history, the previous command and the current encoded image and speed. A sampled state joins the prior history to form the latent supplied to the decoders, while the GRU prepares updated history. Route localization and VectorNet provide the goal-related feature that this latent alone does not contain. Self-attention fuses the two streams, and the action head returns desired velocities plus an optional path. The source compares carrying history through an episode with resetting it at every cycle, finding much weaker success for resets. The reader inference is that memory matters for deployed feedback. It does not follow that the robot searches imagined futures online: the observation-free predictor and its five-step diagnostic establish forecasting capability separately from the direct policy. e03e04e06e07e13e15
5.3 Use three evidence levels to judge generalization
First inspect prediction quality. The diffusion-policy experiment improves two offline errors but worsens linear-speed error, and semantic forecasting retains navigable surfaces more accurately than several obstacle classes. Next inspect executed navigation in simulation. Here the random-obstacle ablations make memory, semantic supervision and pretraining relevant, while warehouse results show that an auxiliary loss can trade efficiency for robustness elsewhere. Finally inspect physical execution. Nova Carter succeeds without fine-tuning in the listed lab trials, but the denominators remain small and the table contains no competing real-robot methods. Cross-embodiment examples add simulated demonstrations rather than a quantitative physical comparison. My interpretation is that the paper supports a promising navigation representation and a bounded transfer result; it leaves open how reliably those advantages persist across dynamic obstacles, camera placements and different control constraints. e11e12e13e14e15e16e18
5.4 Training and inference
During training
Isaac Sim supplies 160K random-action and 100K Nav2-teacher frames, sampled at 5 Hz in four warehouses. Random-action records retain commands and semantic labels but omit route/path supervision. e09
Pretrain the world model for 100 epochs, then train policy and world model together for 100 more. Training uses five-frame samples, batch size 32, eight H100 GPUs and AdamW at 1e-5. RGB diffusion is disabled during policy training; the source describes joint encoder training rather than a frozen visual backbone. e05e08
During inference
Initialize history once, update it with incoming observations, fuse the route, and issue velocity commands. Observation-free rollouts are separately evaluated; the reported controller does not specify candidate-action search through imagined futures. Platform controllers or locomotion policies realize the commands. e06e07e13e18
5.5 Implementation flow
- Encode observations
DINOv2 image features and an MLP speed embedding form the observation. The estimator combines this with previous history and action to sample a Gaussian state; a GRU propagates memory. e03e04
- Make the state informative
Concatenated history and state condition RGB diffusion and perspective semantic decoding. Noise MSE and multi-resolution semantic cross-entropy train representations; neither reconstruction alone establishes successful control. e03e05
- Learn action-conditioned prediction
A distinct predictor omits current observations. KL matching to estimator beliefs teaches it to forecast latent consequences; this is a forward dynamics model. e06
- Decode navigation
VectorNet embeds the localized route, self-attention combines it with the latent state, and MLP heads imitate velocity commands and an optional path. Localization remains required; only commands feed state transitions. e07
6. Experiments & results
X-MOBILITY learns recurrent navigation beliefs from synthetic random-action experience, then couples them to route-conditioned imitation. Semantic and RGB decoding shape the representation; a separate predictor forecasts latent consequences without new observations. Stronger closed-loop simulation results coexist with modest obstacle-prediction accuracy and limited real-robot evidence (e03, e05, e06, e08, e12, e14, e15, e16).
6.1 Read the original evidence
Table II. Path accuracy improves with diffusion while linear-command accuracy deteriorates. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each column measures a different output on the dataset test split: A-MAE is angular-speed error, L-MAE is linear-speed error, and P-MAE is path error. Lower values are better within a column; the table does not state explicit units, so compare methods without combining these errors. Read the final row against the default X-MOBILITY row. The diffusion variant lowers angular and path errors, yet raises linear-speed error substantially. Also notice that MILE has the smallest linear-speed error in the table despite weak closed-loop performance elsewhere. The DP footnote identifies an action-policy variant; it should not be confused with the separate RGB diffuser. e05e10e11e12
What it supports. The diffusion-policy variant changes A-MAE from 0.0747 to 0.0306 and P-MAE from 0.1156 to 0.0389, but changes L-MAE from 0.0158 to 0.1115. The authors report inconsistent commands and collision difficulties in closed loop, illustrating why offline prediction metrics alone cannot select a controller.
Where the evidence stops. No numerical diffusion-policy success rate is supplied. The proposed single-step denoising explanation is an author hypothesis, not an isolated experiment; baseline encoder and route-processing differences also complicate causal comparisons.
Table III. The strongest generalization evidence comes from closed-loop ablations on random obstacles. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two scenario groups separately. Within each, success rate increases upward in desirability, while weighted trip time and average absolute angular acceleration should decrease. Weighted trip time divides trip time by success rate; it is not an ordinary duration among successful runs. The upper block compares systems; the lower block removes pretraining, semantic decoding or continuous history. Start with the random-obstacle success column: the full model reaches 68%, compared with 44%, 36% and 16% for those removals. Then return to warehouses: removing semantics preserves 96% success and improves both time and smoothness. That reversal is central to interpreting the auxiliary task. e08e10e12e13
What it supports. The full system exceeds BC's random-obstacle success by 12 percentage points, 68% versus 56%. Semantic decoding contributes to robustness in that group, while the warehouse metrics reveal a tradeoff. The history ablation has the largest observed random-obstacle success loss, from 68% to 16%.
Where the evidence stops. Warehouse scores pool seen and unseen environments. The 100-run description does not fully specify trial allocation; no uncertainty is reported. Pretraining changes both data exposure and optimization, so its row does not isolate dynamics learning.
Table V. Physical transfer succeeds in the reported lab trials, with small sample counts. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first two columns identify obstacle configuration and lighting; the original header spells the latter 'Lightning'. Read SR as successful trials over attempted trials, rather than a percentage printed directly. The two single-obstacle rows each report ten successes in ten attempts. The multiple-obstacle rows each report eight successes in nine attempts. AA is average absolute angular acceleration, so smaller values indicate smoother motion by this metric. Section V.A identifies the platform as Nova Carter and says deployment used no fine-tuning. Figure 7 supplies the corresponding lab context; these rows are distinct from the simulated warehouse benchmark. e16e18
What it supports. The table supports zero-shot transfer to these real-robot configurations. Multiple-obstacle success is unchanged across the reported lighting conditions, but angular acceleration rises from 0.205 to 0.273 rad/s² in darkness. Single-obstacle acceleration is 0.267 in both conditions, matching the identical reported success counts.
Where the evidence stops. These few trials include no physical baseline comparison or uncertainty intervals. Cross-embodiment examples elsewhere use simulation and platform-specific controllers; they do not extend this table into a measured real-world multi-robot benchmark.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Closed-loop navigation in simulation Ten warehouse scenarios, including five unseen-environment cases; 50 separately generated random-obstacle corridor scenes. | Warehouse: 96, 37.7, 0.206. Random obstacles: 68, 40.21, 0.186. Success rate SR (%) ↑; weighted trip time WTT (s; trip time divided by success rate) ↓; average absolute angular acceleration AA (rad/s²) ↓ | BC: 92/40.4/0.219 and 56/42.72/0.219; Nav2: 58/68.41/0.606 and 34/74.94/0.391. The full model beats these baselines in both aggregates. Warehouse results pool seen and unseen cases; trial-allocation wording is ambiguous. e10e12 |
| Generalization ablations Same Table III benchmark; random obstacles and warehouse conditions retained separately. | Random obstacles: full 68; no pretraining 44; no semantics 36; no history 16. Mission success rate (%) | Warehouse: full 96; corresponding ablations 88, 96, 72. Semantic decoding helps the harder distribution but does not raise warehouse SR; removing it improves warehouse WTT from 37.7 to 33.7 s. Pretraining also adds data and optimization. e08e13 |
| Open-loop imitation and diffusion-policy variant Dataset test split; Table II reports no explicit MAE units. | Default: 0.0747 / 0.0158 / 0.1156; diffusion: 0.0306 / 0.1115 / 0.0389. Angular-speed / linear-speed / path mean absolute error (A-MAE / L-MAE / P-MAE) ↓ | MILE: 0.0751 / 0.0106 / 0.1192; BC: 0.0870 / 0.0165 / 0.1176. Diffusion improves angular/path error but worsens linear error. Its closed-loop instability is described qualitatively; offline accuracy is not sufficient evidence of navigation quality. e10e11 |
| Semantic representation and forecasting Table IV segmentation; Figure 6 separately evaluates five prediction steps. | Table IV full model: navigable 0.97, forklift 0.16, cone 0.18, pallet 0.42, fence 0.32, sign 0.15. Class IoU ↑ | Without pretraining: 0.97, 0.13, 0.14, 0.38, 0.27, 0.12 in the same order. Obstacle IoUs remain modest. Forecast curves generally decline with horizon. Table IV does not state a prediction horizon; its scores should not be treated as five-step averages. e14e15 |
| Zero-shot real-robot navigation Nova Carter, single/multiple obstacles under normal/dark lighting; no fine-tuning. | Single obstacle: 10/10 and 0.267 in each lighting condition. Multiple: 8/9 in each; AA 0.205 normal, 0.273 dark. Successful trials; AA (rad/s²) | No real-robot baseline comparison is supplied. Supports transfer in the tested lab configurations, with small samples and no reported uncertainty. e16 |
| Embedded inference Jetson AGX Orin; reported inference modes. | Policy only: 38.6 / 41.96; 594 MB. P50 / P95 latency (ms); GPU memory (MB) | Policy plus semantics: 55.55 / 55.72; 804 MB. Measures these inference modes, not the cost of running the RGB diffuser. e17 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 6. Good surface prediction coexists with declining and much lower obstacle-class IoU. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend before comparing trajectories. Blue circles and orange squares use pretraining; green diamonds and red triangles omit it. Blue and green use dataset actions, whereas orange and red use learned-policy actions. The horizontal axis counts prediction steps. Every panel has its own vertical scale, so equal-looking slopes do not imply equal errors across classes. Navigable-surface curves cluster near the top of their narrow range. Obstacle-class curves start much lower and generally decline. Compare blue with orange, or green with red, to inspect the additional error associated with learned actions; compare the pretrained and unpretrained pairs to inspect the effect of pretraining. e06e14e15
What it supports. The diagnostic supports useful short-horizon latent forecasting, particularly for navigable surfaces, and shows remaining error for obstacles. Pretraining generally improves obstacle predictions, while later learned-policy predictions tend to lose additional accuracy. These are decoded semantic forecasts, not measurements of collision-free future execution.
Where the evidence stops. No uncertainty bands are shown. High navigable IoU cannot substitute for obstacle accuracy or planning success. Figure 5's separately labeled times should not be assigned to this five-step axis.
7. Analysis & limitations
7.1 What the evidence leaves open
The benchmark does not isolate every cause: adapted MILE uses a different visual encoder and route encoding. Attention pictures and pretraining gains cannot independently establish that learned dynamics causes the navigation improvement. e11e13e14
Cross-embodiment evidence is simulated and qualitative. Additional dynamic-obstacle data, robot-specification encoding and RL fine-tuning are future work, limiting claims about broad deployment. e18
The source has specification inconsistencies: Eq. (2)'s history subscript differs from surrounding notation; Table I and the encoder describe scalar forward speed, while dataset prose mentions linear and angular velocities. These require clarification for exact reproduction. e03e04e09
7.2 Questions for discussion
- Does semantic supervision improve obstacle avoidance after matching additional training data and optimization?
- How much of the gain requires a predictive prior beyond recurrent state estimation?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the Isaac Sim/Nav2 data pipeline, camera/route conventions, semantic supervision and staged training. Appendix I gives history/state sizes 1024/512, mixed precision, weight decay 0.01, OneCycleLR with pct_start 0.2, and action/path/semantic/RGB/KL loss weights 10/5/1/10/0.001. e07e09e19
Exact dataset split assignments, randomization distributions, success tolerances and software versions are not specified in the reviewed setup or appendix. Resolve the 100-run allocation before matching benchmark totals; do not infer confidence intervals from rounded aggregate success rates. e09e10e19
Proposed checks: compare semantic supervision under matched data/compute and paired scenes; test recurrent-history resets using identical checkpoints and routes. Report success and smoothness separately from offline prediction quality. e13e14e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does semantic supervision survive matched-data controls?
Reader-proposed experiment, not performed: keep the random-action pretraining data, teacher split, encoder initialization, network capacity, batches and total updates fixed. Compare the normal semantic loss with zero semantic-loss weight while still executing the head to match computation. Use the appendix's remaining loss weights unchanged. Evaluate paired warehouse and random-corridor scenes with identical start/goal draws, and report success, WTT, AA and class IoU across several training seeds. The discriminating observation is a reproducible random-obstacle success gain after these controls. If the advantage disappears, the published ablation alone is insufficient to attribute robustness to semantic supervision. e08e09e10e13e14e19
Check 2: How much continuous history is necessary?
Reader-proposed experiment, not performed: use one trained checkpoint and the same paired navigation episodes, changing only whether history and latent state reset every control cycle, every five observations, or only at episode start. Keep routes, observation timing, stochastic sampling seeds and the action head fixed. Compare success and angular acceleration, and inspect turn overshoots and collision locations. The five-observation condition probes the relation to five-frame training samples. If continuous history consistently outperforms both reset schedules, it supports a benefit from longer context. If periodic resets match it, short retained context may explain the result without requiring long-horizon memory. e03e07e08e10e13
8.3 Reading coverage
Visual audit: All eight PDF pages were rendered and visually inspected, including the title/author/version block, Figures 1–8, Tables I–VIII, equations, evaluation text and Appendix I. All six final original crops were separately viewed at their final dimensions; table headers, the DP footnote, prediction axes and the complete legend are retained. Figure 2's observation-only estimator input and history/state arrows were checked against Eqs. (1)–(3) and their prose; the anomalous Eq. (2) subscript is disclosed. The real-robot table's lighting-header typo and scalar-speed specification discrepancy are preserved and explained. No separate supplements, source code or other editions were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and I. Introduction (pp. 1–2)
- II.A–F. Problem, belief, estimation, prediction, action policy and multi-stage training (pp. 2–4)
- III.A–C. Dataset, training and evaluation (pp. 4–5)
- IV.A–C. Navigation, ablations and world-state prediction (pp. 5–6)
- V.A–B. Sim2Real and cross embodiment (pp. 6–7)
- VI. Future Work (p. 7)
- References (pp. 7–8)
- Appendix I. Model Description (p. 8)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All four supplied text chunks were read individually. The extraction's missing figure images were resolved by inspecting all eight PDF pages and six final original crops.
- Identity scope: title and all eight authors match; the observed source is arXiv:2410.17491v3, 22 July 2025 (e01). The catalog names ICRA 2025, but a separate proceedings edition and earlier revisions were not supplied or compared; equivalence between editions is unverified.
- No separate supplements were supplied. Project links and code were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author affiliations and arXiv margin stamp
Title and all eight catalog authors match. The artifact is arXiv:2410.17491v3 [cs.RO], dated 22 July 2025. Wei Liu and Huihua Zhao contributed equally. All authors are affiliated with NVIDIA; Chenran Li also with UC Berkeley, and Joydeep Biswas also with UT Austin.
Go to primary source ↓e02PDF pp. 1–2, Introduction and Sections II.A–B
Navigation is formulated as a POMDP with a learned Gaussian belief. A teacher policy is assumed to approximate the optimal policy; state estimation, prediction and action policy have distinct roles.
Go to primary source ↓e03PDF p. 2, Section II.C, Eqs. (1)–(2); p. 3, Figure 2(b–e)
The estimator conditions on previous history and action plus current observation embedding. MLPs parameterize a Normal distribution, sampling produces the latent state, and a GRU updates history. Decoders use the concatenation of previous history and sampled current state. Eq. (2) prints an anomalous nested history subscript, while surrounding prose and Figure 2 label the previous history h_{t-1}.
Go to primary source ↓e04PDF pp. 2–3, Section II.C.2 and Figure 2(a)
A pretrained DINOv2 encodes the front image using concatenated class and average-pooled patch tokens into a 768-dimensional embedding; linear forward speed has a 32-dimensional MLP embedding. Their concatenation forms the observation embedding.
Go to primary source ↓e05PDF p. 3, Section II.C.3 and Figure 2(e)
Latent diffusion RGB reconstruction uses a state-conditioned UNet and noise MSE. A StyleGAN semantic decoder predicts perspective-view segmentation, with cross-entropy at each upsampling resolution. Encoders, decoders and estimator are trained together.
Go to primary source ↓e06PDF p. 3, Section II.D, Eq. (3) and Figure 2(d)
The predictor uses history and action without observation embeddings; KL divergence encourages its belief distribution to match the observation-conditioned estimator. Future prediction is motivated by possible long-horizon planning applications.
Go to primary source ↓e07PDF pp. 3–4, Section II.E and Figure 2(e)
Localization transforms a global route to the robot frame; a nearby segment is encoded by VectorNet into 64 dimensions. Self-attention fuses route and latent state. MLP heads predict six linear/angular velocity components and an optional five-pose 2D path under L1 imitation loss. Only action commands enter state transitions; the path can shape latent states and support safety checking.
Go to primary source ↓e08PDF p. 4, Sections II.F and III.B
World modeling is pretrained with logged random actions while the policy is disabled, then trained jointly with the policy on teacher data. Each stage lasts 100 epochs; samples have five frames; batch size is 32 over eight H100 GPUs; AdamW uses learning rate 1e-5. RGB diffusion is disabled in the policy stage.
Go to primary source ↓e09PDF p. 4, Section III.A and Table I
Isaac Sim Nova Carter data comprise 160K random-action and 100K Nav2-teacher frames at 5 Hz across four warehouses with randomized appearance/layout. Both sources include images, speed, semantics and action commands; only teacher data include route and path. Table I specifies scalar speed, 320 by 512 images, seven semantic channels, 20 route positions and five path positions. Nearby prose describes speed as linear and angular, unlike the scalar field and forward-speed encoder.
Go to primary source ↓e10PDF pp. 4–5, Section III.C
Open-loop MAE uses the dataset test split. Closed-loop evaluation describes ten warehouse scenarios, five in unseen environments, and 50 random cluttered corridor scenes. The text states 100 runs and five trials per designed warehouse scenario, leaving the allocation incompletely explicit. Metrics are success rate, trip time divided by success rate, and average absolute angular acceleration.
Go to primary source ↓e11PDF p. 5, Table II and Sections IV.A, IV.B.4; p. 6, continuation of IV.B.4
Table II gives A-MAE/L-MAE/P-MAE: MILE 0.0751/0.0106/0.1192; BC 0.0870/0.0165/0.1176; X-MOBILITY 0.0747/0.0158/0.1156; diffusion-policy variant 0.0306/0.1115/0.0389. BC shares the observation encoder; adapted MILE uses perspective semantics and ResNet-18. Learned baselines are retrained on the same dataset. Diffusion-policy closed-loop inconsistency is described without a numerical success row; the authors propose single-step denoising instability as a possible explanation.
Go to primary source ↓e12PDF p. 6, Table III, upper comparison rows
Warehouse SR/WTT/AA are X-MOBILITY 96/37.7/0.206, BC 92/40.4/0.219, MILE 34/128.32/0.274 and Nav2 58/68.41/0.606. Random-obstacle values are X-MOBILITY 68/40.21/0.186, BC 56/42.72/0.219, MILE 26/113.49/0.212 and Nav2 34/74.94/0.391. Units are percent, seconds and rad/s².
Go to primary source ↓e13PDF p. 5, Sections IV.B.1–3; p. 6, Table III, ablation rows
Without pretraining, warehouse/random-obstacle SR are 88/44%; without semantic decoding 96/36%; without history tracking 72/16%; full model 96/68%. No-semantic warehouse WTT/AA are 33.7/0.203 versus full 37.7/0.206. Resetting history every cycle is compared with initializing once and recurrently updating. The authors associate semantic decoding with generalization and recurrent history with smoother control.
Go to primary source ↓e14PDF p. 5, Figure 4 and Section IV.B.2; p. 6, Table IV
The attention example compares pretrained DINOv2 with semantic-decoding fine-tuning. Table IV full-model IoUs for navigable/forklift/cone/pallet/fence/sign are 0.97/0.16/0.18/0.42/0.32/0.15, versus no-pretraining 0.97/0.13/0.14/0.38/0.27/0.12. The background training class is not reported as a table column.
Go to primary source ↓e15PDF p. 6, Section IV.C and Figures 5–6
Latent predictions are decoded to semantic images. Figure 6 reports mean IoU over five prediction steps, with and without pretraining and with dataset versus learned actions. Navigable IoU stays high; obstacle-class values are lower and generally decline with horizon. Learned-policy curves fall slightly below corresponding dataset-policy curves later in prediction. Figure 5 labels illustrative images at 0, 0.2, 1 and 2 seconds; those are distinct from the five-step plot.
Go to primary source ↓e16PDF pp. 6–7, Section V.A; p. 7, Figure 7 and Table V
A Nova Carter is deployed without fine-tuning. Table V reports single-obstacle success 10/10 under both normal and dark lighting, AA 0.267 in both; multi-obstacle success 8/9 in both conditions, AA 0.205 normal and 0.273 dark. No real-robot baseline rows or uncertainty intervals are shown.
Go to primary source ↓e17PDF p. 6, Section V.A; p. 7, Table VI
On Jetson AGX Orin, policy-only inference has P50 38.6 ms, P95 41.96 ms and GPU memory 594 MB; policy plus semantic decoding has 55.55 ms, 55.72 ms and 804 MB. The table does not benchmark RGB diffusion inference.
Go to primary source ↓e18PDF p. 7, Sections V.B and VI, Figure 8
Cross-embodiment deployments in Isaac Sim use Nova Carter, a forklift, Unitree Go2 and Unitree G1, with suitable controllers or Isaac Lab pretrained locomotion policies. Performance depends on camera placement and dynamics. Robot-specification encoding, RL fine-tuning and more dynamic-obstacle scenes are future work; no per-platform quantitative benchmark is provided.
Go to primary source ↓e19PDF p. 8, Appendix I, Tables VII–VIII
Table VII lists module sizes including a 962M-parameter RGB diffuser. Table VIII specifies 16-mixed precision, weight decay 0.01, OneCycleLR with pct_start 0.2, history/state dimensions 1024/512, four VectorNet layers, and policy state dimension 2048. Action/path/semantic/RGB/KL loss weights are 10/5/1/10/0.001, KL balancing alpha 0.75. RGB diffusion uses LMS discrete, scaled-linear beta schedule 0.00085–0.012, 1000 training and 50 inference timesteps.
Go to primary source ↓8.5 Primary sources
X-MOBILITY: End-To-End Generalizable Navigation via World Modeling ↗
PDF · 6,055 extracted words
Source fingerprint
d59486e088f09246a49fb6772e459c0815ed8b7c0d0e14e8bc1d459626c5b1fc