UniNav: A Unified World-Action Diffusion Model for Visual Navigation
1. Paper overview
In one sentence: UniNav learns video, geometry and waypoints in one transformer, while a separately trained Fast variant trades inference-time visual foresight for lower waypoint-prediction latency. e3e5e7e8e11e13e15
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors seek to combine the direct waypoint generation of navigation policies with the visual anticipation of world models, while avoiding separate imagined rollouts for each candidate trajectory. They also ask whether videos lacking waypoint labels can improve navigation through shared visual and geometric supervision. e2e7 |
| Core mechanism | |
| A key reported result | Offline local waypoint prediction: RECON: Full 0.314; SaCSoN: Full 0.464; GO Stanford: Fast 0.687; SCAND: Full 0.463. ATE, lower is better; units are not specified. Four navigation datasets; navigation-only training; two-step UniNav sampling. Exact evaluation splits and sample counts are not supplied. Best baseline ATE respectively: NWM 0.326, ViNT 0.616, NavDP 0.840 and NWM 0.497. The text reports relative reductions of 3.7%, 24.7%, 18.2% and 6.8%. The best UniNav variant leads ATE on each dataset, but the winning variant changes. These are measured offline trajectory errors, not navigation success rates. e9e11e15 |
| Reading caution | Author claim The authors explicitly limit the present system to short-horizon local waypoint prediction and primarily offline evaluation; long-horizon planning and broader closed-loop deployment remain future work. e15 |
Core contributions
- Source description
The separately trained Fast variant uses asymmetric attention to retain video supervision during training while eliminating future-video generation during inference. e8
Figure 2. Shared token processing couples visual prediction with waypoint generation. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the blue training panel. Image sequences feed a geometry model and a VAE, producing geometry and visual tokens; waypoint tokens join history, goal and the gray tokens labeled Registry Token. Section 3.2 calls these learnable register tokens. The bracket feeds one world-action transformer, with video, camera and trajectory losses. Then follow the yellow panel: random trajectory and image states enter alongside the conditioning tokens, and repeated updates refine both outputs. This panel illustrates joint generation, not candidate-trajectory scoring. For Fast, consult Section 3.4: its separately trained asymmetric attention mask, which is not drawn here, blocks non-future tokens from reading noisy future-video tokens. e3e4e5e6e7e8e17
What it supports. The shared transformer provides direct architectural support for joint world–action prediction. Geometry is a denoising target extracted during training, not future information available to a deployed navigator. Full exposes a visual forecast alongside its waypoint sequence; Fast retains visual supervision during training while omitting future-video generation at inference.
Where the evidence stops. The inference sketch omits camera tokens, whereas Equation (3) and Section 3.4 include them in Full's joint denoising. Their initialization is unspecified. The figure therefore cannot justify removing camera tokens or supplying clean future geometry at inference.
2. Motivation
2.1 The problem and the proposed response
The authors seek to combine the direct waypoint generation of navigation policies with the visual anticipation of world models, while avoiding separate imagined rollouts for each candidate trajectory. They also ask whether videos lacking waypoint labels can improve navigation through shared visual and geometric supervision. e2e7
2.2 What this reading follows
An image-goal navigator must connect what it sees now to how it should move next. UniNav trains that connection by jointly denoising future images, local waypoints and camera features extracted by a frozen geometry model. Its Full variant produces both visual forecasts and trajectories. Fast uses asymmetric attention during separate training so it can omit future-video tokens at deployment. The most revealing evidence is the interaction between geometry supervision and extra unlabeled video: more video helps only with geometry denoising in the reported ablation. Read the benchmark gains as offline prediction results; the supplied paper does not establish closed-loop navigation success. e3e5e7e8e11e13e15
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The core Q1 classification is supported: one shared denoising transformer processes future-video and waypoint targets under joint flow matching. This is direct joint prediction, not inverse dynamics applied to generated images. Frozen representation encoders do not constitute separate world and action predictors. Fast retains joint training but omits future-image prediction at inference. Navigation and Joint video-action modeling are supported; 3D multiview modeling is less explicit, since temporal geometry features do not establish a dedicated multiview reconstruction output or simultaneous multi-camera architecture. e3e4e5e6e7e8
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn future geometry into a target rather than an oracle input
Start from the information available at deployment: a short image history and a goal image. During training, future frames are also available, so the frozen VAE can encode their appearance and the frozen geometry model can extract future-view camera features from history plus future images. UniNav adds noise to those targets and to labeled waypoints, then asks a shared transformer to predict their flow velocities. The camera features therefore provide a learning signal without being supplied clean at deployment. Equation (4) defines each interpolated noisy state; Equation (8) combines the available losses. Table 2 clarifies why this distinction matters: clean future geometry gives the strongest row, but it is explicitly an oracle. Denoising is the practical alternative described by the source. e3e4e5e7e12
5.2 Understand how unlabeled video can influence waypoint learning
For a navigation sample, video, waypoint and camera losses are all available. For a video-only sample, the trajectory availability indicator is zero, so no action target is invented. The remaining objectives still update the shared representation. The authors describe camera denoising as a geometric connection between visual dynamics and navigation actions. Table 3 supplies a concrete test of that claim: adding video without camera denoising worsens ATE, while adding it with denoising improves both errors. A reader's interpretation is that representation sharing alone is insufficient in this experiment; the auxiliary target matters. However, the paper does not match out every possible exposure or optimization difference. The result motivates a controlled interaction test rather than a general rule that more unlabeled video improves navigation. e7e10e13
5.3 Separate visual supervision from visual computation at inference
Full lets future-video, waypoint and camera targets participate in the complete denoising sequence. Fast is trained separately with a directional attention restriction: every non-future token is blocked from attending to noisy future-video tokens, while future-video tokens can attend to the other tokens. This preserves video supervision during training but removes the stated dependence of waypoint denoising on future-video representations. At inference, Fast discards future-video tokens and avoids their denoising and VAE decoding. This is not a later conversion of predicted images into actions. Table 4 then measures the resulting accuracy–latency choices, including the one-step operating point. The reported speed belongs to waypoint prediction on a GPU; the paper leaves broader closed-loop deployment and long-horizon planning to future work. e6e8e14e15
5.4 Training and inference
During training
Video and camera velocities use squared-error losses; waypoint velocities use SmoothL1. Navigation samples activate all three losses. Video-only samples deactivate the trajectory loss and require no pseudo-trajectories. e7
Training initializes Wan2.1-T2V-1.3B and fine-tunes its transformer plus new modules, keeping the VAE and geometry model frozen. AdamW learning rates are 5×10^-6 and 5×10^-5 for pretrained and new modules respectively. Training uses four RTX 4090 GPUs, two epochs, batch size one per GPU and four accumulation iterations. e5e10
Loss weights are 1 for video, 5 for trajectory and 0.01 for camera supervision. The implementation paragraph calls the trajectory weight λ_act, whereas the objective uses λ_traj. Images are resized to 256×192. Video-only sources include ScanNet, DL3DV, CityWalker and LAVN, but the main benchmark comparison excludes this additional data. e9e10e11
During inference
Full iteratively refines its predicted modalities from noise, conditioned on history and goal, using two sampling steps by default. Section 3.4 includes camera denoising, but Figure 2 omits camera tokens from its inference sketch; their initialization is not explicitly described. e3e8e10e17
Fast removes future-video tokens and skips future-frame VAE decoding. Its visual prediction objective supplies training supervision; it does not generate visual rollouts for inference-time trajectory scoring. e8
The described output is a waypoint sequence. A low-level tracking controller, replanning cadence and complete robot feedback loop are not specified in the supplied method, so executable-format predictions should not be equated with demonstrated robot execution. e3e8e15
5.5 Implementation flow
- Encode visual context
A frozen video VAE encodes history, goal and training future frames into patchified latent tokens. History and goal tokens condition prediction; future tokens are denoising targets. e4
- Represent motion and geometry
MLPs embed noisy three-component waypoints and noisy camera features. A frozen geometry model extracts normalized future-view features from history plus future images. These learned geometric targets are not supplied ground-truth camera poses. e3e4e5
- Share the denoising backbone
Camera, future-video, waypoint, history, goal and register tokens share one transformer and modality-specific heads. Visual tokens retain pretrained positional encoding; camera, waypoint and register tokens use identity encodings. e6e7
- Choose the inference dependency
Fast is trained to block every non-future token from attending to noisy future-video tokens; future-video tokens can attend to the remaining sequence. Dropping future-video tokens preserves this trained information path. e8
6. Experiments & results
UniNav adapts a pretrained video transformer to jointly predict local waypoints, future visual latents and camera-geometry tokens. Geometry denoising also makes extra video-only data useful in the reported RECON ablation. A separately trained Fast variant removes future-video tokens at inference. The evidence establishes offline trajectory-error and latency trade-offs, while closed-loop goal-reaching remains untested in the supplied results.
The supplied PDF has suitable architecture, quantitative and ablation visuals, all represented here. However, Sections 4.3–4.4 defer numerical visual-prediction metrics, alternative prediction designs and robot-collected-data evaluations to an Appendix absent from this PDF. Figures 3–4 are qualitative; they do not provide a quantitative consistency or closed-loop success test. Those missing experiment types cannot be illustrated from the supplied material. e9e15e16e18e19
6.1 Read the original evidence
Table 1. Best ATE belongs to a UniNav variant on every dataset, but the winning variant changes. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each dataset's ATE and RPE columns separately; lower is better in both. Section 4.2 specifies navigation-only training for this comparison, with no extra video-only data, and two sampling steps for UniNav. On RECON, SaCSoN and SCAND, Full has the lowest ATE. On GO Stanford, Fast leads instead. The rightmost averages summarize the table but should not replace those dataset-specific comparisons. Also inspect RECON's RPE column: NWM remains best there. The method grouping distinguishes action-only baselines from world-model approaches, but it does not imply identical inference workloads or prove that every baseline was retrained under a fully documented common protocol. e9e10e11e15
What it supports. The best UniNav ATEs are 0.314, 0.464, 0.687 and 0.463, against best baseline values 0.326, 0.616, 0.840 and 0.497, respectively. These support the paper's ATE claim. RECON RPE gives a different ordering: NWM reaches 0.115, compared with Full's 0.121 and Fast's 0.122.
Where the evidence stops. Exact train/test partitions, sample counts, ATE/RPE units and run-to-run uncertainty are unspecified in the supplied experiment description. These are offline errors, not success rates; the table does not establish collision avoidance or goal completion.
Figure 3. Selected examples pair a changing egocentric view with a predicted trajectory. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right. Three context images precede a dashed divider; six predicted images follow. The two upper rows are labeled RECON and the two lower rows SCAND. At the far right, the preserved legend identifies blue GT and red Pred trajectory curves. Compare the relative shape of those curves while watching the scene viewpoint change across the forecast. The building and tree scenes make the intended combination of visual and motion outputs concrete. Crucially, the image strip contains predictions rather than paired prediction/ground-truth future-image rows, and the trajectory plots do not provide numbered axes for measuring displacement. e18e16e15
What it supports. The figure illustrates the form of joint output: image sequences accompany trajectories whose plotted shapes are close to the shown references in these selected examples. It makes visual foresight inspectable, but the evidence is qualitative. Neither exact image fidelity nor calibrated agreement between visual motion and waypoint motion can be measured from these panels.
Where the evidence stops. The paper supplies no selection protocol for these examples or quantitative image–waypoint consistency test. Numerical visual-quality results are deferred to an absent Appendix. Apparent coherence here is not evidence that a robot executed the predicted paths.
Table 4. Fast lowers model latency while its accuracy changes nonmonotonically with sampling steps. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each UniNav block to vary denoising steps within a model, then compare Full and Fast at the same step count. The bold two-step entries mark the default setting. Latency is in seconds on a single NVIDIA RTX 4090; NFE counts function evaluations. The one-step Fast row is the precise counterpart to the abstract's rounded 0.1-second claim. Compare both errors when reducing steps, rather than assuming that fewer updates preserve every metric. The NWM row reports a much larger evaluation count, so it describes a different planning workload; its large latency should not be treated as a pure per-transformer-call speed comparison. e8e14e15
What it supports. Two-step Fast takes 0.212 seconds versus Full's 0.389, with ATE 0.316 versus 0.314. One-step Fast takes 0.109 seconds at ATE 0.318. More steps are not uniformly better: Fast's ATE is 0.300 at 10 steps, 0.313 at 20 and 0.331 at 50. ViNT remains faster at 0.015 seconds, with higher ATE.
Where the evidence stops. These are reported model timings on one GPU, without a complete robot-control timing protocol or latency variability. They do not establish end-to-end closed-loop speed or safety. Full and Fast are separately trained variants, not one checkpoint toggled for this comparison.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Offline local waypoint prediction Four navigation datasets; navigation-only training; two-step UniNav sampling. Exact evaluation splits and sample counts are not supplied. | RECON: Full 0.314; SaCSoN: Full 0.464; GO Stanford: Fast 0.687; SCAND: Full 0.463. ATE, lower is better; units are not specified | Best baseline ATE respectively: NWM 0.326, ViNT 0.616, NavDP 0.840 and NWM 0.497. The text reports relative reductions of 3.7%, 24.7%, 18.2% and 6.8%. The best UniNav variant leads ATE on each dataset, but the winning variant changes. These are measured offline trajectory errors, not navigation success rates. e9e11e15 |
| Relative pose accuracy RECON main comparison with two-step UniNav sampling | Full 0.121; Fast 0.122. RPE, lower is better | NWM achieves 0.115. UniNav's RECON ATE advantage does not extend to the best RPE; the claim of improvement should remain metric-specific. e11 |
| Geometry-supervision ablation RECON; camera-token design comparison | Camera-token denoising: 0.314 / 0.121. ATE / RPE, lower is better | Without camera tokens: 0.321 / 0.125. Clean future-camera conditioning: 0.299 / 0.102. Denoising provides a modest observed improvement. Clean conditioning is an oracle using ground-truth future images and is unavailable in practical inference. e12 |
| Transfer from video-only data RECON; camera denoising enabled or disabled, with or without additional video-only training | With camera denoising, additional videos change 0.314 / 0.121 to 0.302 / 0.109. ATE / RPE, lower is better | Without camera denoising, additional videos change 0.321 / 0.125 to 0.330 / 0.125. The observed interaction supports geometry supervision as a useful component of video transfer in this setting. It does not establish universal benefits from more video data or isolate all effects of training exposure. e13 |
| Accuracy–latency trade-off RECON; latency measured on one NVIDIA RTX 4090 | One-step Fast: 0.109 s and 0.318 / 0.123. Latency in seconds; ATE / RPE | Two-step Fast: 0.212 s and 0.316 / 0.122; two-step Full: 0.389 s and 0.314 / 0.121. ViNT is faster at 0.015 s, with errors 0.659 / 0.215. One-step Fast nearly preserves its two-step offline accuracy while reducing latency. It is not the fastest listed policy, and model latency alone does not establish closed-loop performance. e14e15 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Camera denoising improves the reported errors; clean conditioning is an oracle. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the no-camera row as the reference. The bottom row adds camera-token denoising, making camera features an auxiliary prediction target rather than a clean input from the future. Compare its two error columns to the first row to assess the practical design's observed effect. The middle row answers a different question: how helpful would future geometry be if it were already known? The caption explicitly labels that setting an oracle, because DA3 camera tokens are extracted from ground-truth future frames. Thus the smallest errors in the table must be read with their information-access condition, not as a deployable alternative. e5e7e12
What it supports. Denoising changes RECON ATE/RPE from 0.321/0.125 without camera tokens to 0.314/0.121. Clean future-camera conditioning reaches 0.299/0.102, showing that privileged future geometry can be useful. The deployable contribution is the smaller gain obtained while treating those features as targets instead of oracle observations.
Where the evidence stops. Clean conditioning requires future frames unavailable at practical inference. No uncertainty is reported for the modest denoising gain, and these trajectory metrics do not independently measure camera-pose, depth or metric reconstruction accuracy.
Table 3. The observed benefit from extra video depends on camera denoising. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this as a two-factor comparison. The left column switches camera denoising off or on; within each block, the data column switches from navigation data alone to additional video-only sequences. The unlabeled signed row is the change in error, not another trained model. Lower error is better, so a positive change indicates deterioration. Equation (8) explains the supervision: a video-only sample contributes no trajectory loss; when camera supervision is enabled, it supplies video and geometry targets. Compare the within-block changes before comparing absolute scores. This separates the observed value of extra video from the baseline improvement associated with adding geometry denoising. e7e9e10e13
What it supports. Without camera denoising, extra video increases ATE from 0.321 to 0.330 and leaves RPE at 0.125. With denoising, it decreases ATE/RPE from 0.314/0.121 to 0.302/0.109. The opposite ATE changes support a useful interaction between geometry supervision and video transfer in this RECON experiment.
Where the evidence stops. Video mixture proportions, matched training exposure and uncertainty are not documented. The table supports an interaction in this setting; it does not prove a universal benefit from unlabeled video or isolate geometry from every optimization-budget effect.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors explicitly limit the present system to short-horizon local waypoint prediction and primarily offline evaluation; long-horizon planning and broader closed-loop deployment remain future work. e15
The main comparison excludes extra video-only data, but the supplied text does not document exact train/test partitions, sample counts, baseline retraining procedures or uncertainty across runs. Small differences cannot be interpreted as statistically established gains. e9e10e11e12
Camera-token ablations measure waypoint errors rather than camera-pose, depth or metric reconstruction accuracy. They support a useful auxiliary target, but do not independently establish the claimed metric geometric understanding. e5e12e13
PSNR, SSIM and LPIPS are named, but numerical visual-quality results are absent from the supplied PDF. Figures 3 and 4 offer qualitative examples, without a quantified test of image–waypoint consistency. Figure 4 includes trajectories despite its video-only data lacking waypoint annotations; those curves are not validated ground truth. e9e16e18e19
7.2 Questions for discussion
- Does the camera-supervision benefit survive repeated seeds and matched training exposure when video-only data are added? [e12, e13]
- How much of Fast's accuracy comes from video supervision, geometry supervision and pretrained features individually? The supplied ablations do not fully separate these sources. [e8, e10, e13]
- Do lower offline waypoint errors translate into higher closed-loop goal-reaching success when controller, replanning rate and latency are held fixed? [e11, e14, e15]
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction requires the Wan transformer/VAE, the frozen DA3 geometry model, image-goal navigation sequences with local waypoint labels, and the listed video datasets for the mixed-data experiment. Full and Fast require separate training configurations. e5e8e9e10e12
Missing details include exact dataset splits and sample counts, frame spacing, goal sampling, waypoint scaling, geometry-feature selection and normalization, register-token count, video mixture proportions, baseline retraining, software versions, sampler schedule and camera-token initialization. These gaps prevent an exact replication from the supplied PDF alone. e3e5e6e7e8e9e10
The abstract promises a code release. That statement alone does not establish that implementation files or checkpoints are available. e1
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Repeat the geometry-by-video interaction with matched exposure
Reader-proposed check, not performed: reproduce Table 3's four RECON conditions with camera denoising on/off and added video on/off. Fix the split, navigation examples seen, initialization, sampler and optimizer-update budget. Add a repeated-navigation control to match the extra data exposure, and repeat across five seeds. Report ATE/RPE with uncertainty. The mechanism predicts that video improves errors more with camera denoising than without it; disappearance of that interaction under matched exposure would weaken the geometric-transfer interpretation. e7e9e10e13
Check 2: Test Fast's claimed independence from future-video tokens
Reader-proposed check, not performed: with one separately trained Fast checkpoint, hold history, goal, non-future states and sampling schedule fixed. Compare waypoint velocities and final trajectories with noisy future-video tokens present, perturbed, and removed. Under the stated mask, waypoint outputs should agree to numerical precision; a systematic change would reveal an unblocked dependency or a pruning mismatch. Then benchmark warmed-up latency with and without those tokens on the same RTX 4090, separating model computation from frame decoding. This tests the pruning mechanism without conflating Full/Fast training differences. e8e14e17
8.3 Reading coverage
Visual audit: All nine PDF pages were rendered and visually inspected. Page 1 verifies title, author order, affiliations, version and Figure 1; pages 2–5 cover motivation, representations, equations, the Fast attention restriction, datasets and training hardware; pages 6–7 contain all four quantitative tables, Figures 3–4 and limitations. Pages 8–9 contain References and end the supplied PDF. All six final crops were inspected, including headers, legends and complete table rows. Figure 2 was cross-checked against its caption, Equation (3) and Section 3.4: the missing inference camera-token branch remains disclosed. Figure 4's unlabeled trajectories were not treated as ground-truth evidence. No referenced Appendix was supplied.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors and arXiv version stamp
- Abstract
- 1 Introduction
- 2 Related Work
- 2.1 Image-Goal Conditioned Visual Navigation
- 2.2 Navigation World Models
- 2.3 Unified World-Action Models
- 3 Method
- 3.1 Overview
- 3.2 Geometry-Aware Token Representation
- 3.3 Unified Flow-Matching Objective
- 3.4 UniNav-Full and UniNav-Fast
- 4 Experiments
- 4.1 Experimental Setup
- 4.2 Main Results
- 4.3 Ablation Studies
- 4.4 Qualitative Results
- 5 Conclusion, including Limitations and Future Work
- References
Outside the original text pass
- The inspected title, identifier and four authors match the catalog. The PDF is arXiv:2608.03244v1, dated 4 August 2026. No title or author discrepancy was observed; other revisions were not supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-related visual gap was addressed by inspecting all nine PDF pages and all four figures and four tables. All four supplied text chunks were read completely, including References.
- Sections 4.3 and 4.4 refer to an Appendix with visual-prediction metrics, alternative prediction designs, additional visualizations and robot-collected-data evaluations. That Appendix is absent from this nine-page PDF, which ends with References; those results were not reviewed.
- Code and checkpoints were not inspected, and experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e1PDF p. 1, title block, arXiv version stamp and Abstract
The observed title is UniNav: A Unified World-Action Diffusion Model for Visual Navigation. Authors are Changqing Zhou, Yueru Luo, Zeyu Jiang and Changhao Chen. The title page lists The Hong Kong University of Science and Technology (Guangzhou) and The Chinese University of Hong Kong, Shenzhen. The stamp is arXiv:2608.03244v1, 4 August 2026; the abstract promises a code release.
Go to primary source ↓e2PDF pp. 1–2, Section 1 Introduction
The motivation contrasts direct waypoint policies with world models that generate and evaluate separate candidate rollouts, then proposes joint world-action generation and video-only supervision.
Go to primary source ↓e3PDF p. 3, Section 3.1 Overview
Inputs are history frames and a goal image. Outputs include K local waypoints with relative x/y translation and yaw change, plus optional future observations. Image and trajectory inference states begin from random initialization.
Go to primary source ↓e4PDF p. 3, Section 3.2, Visual tokens and Waypoint tokens; Eq. (1)
A frozen video VAE supplies visual latents. Noisy waypoints are embedded with an MLP, processed by the shared transformer and decoded into trajectory velocities.
Go to primary source ↓e5PDF pp. 3–4, Section 3.2, Camera tokens; Eq. (2)
A frozen geometry model processes history and future images. Normalized features for future views become camera denoising targets, which are noised and projected into transformer tokens.
Go to primary source ↓e6PDF p. 4, Section 3.2, Unified token sequence; Eq. (3)
Camera, future-video, waypoint, history, goal and register tokens share one denoising transformer. Visual positional encoding is retained; other listed prediction/register tokens receive identity positional encodings.
Go to primary source ↓e7PDF pp. 4–5, Section 3.3; Eqs. (4)–(8) and supervision-mask explanation
Targets interpolate clean samples with Gaussian noise and use noise-minus-clean velocities. Video/camera losses are squared errors; trajectory loss is SmoothL1. Availability masks disable trajectory supervision for video-only samples.
Go to primary source ↓e8PDF p. 5, Section 3.4 UniNav-Full and UniNav-Fast
Full jointly denoises video, waypoint and camera tokens. Fast is separately trained with asymmetric attention blocking non-future tokens from reading future-video tokens; inference removes future-video tokens and their decoding.
Go to primary source ↓e9PDF p. 5, Section 4.1, Datasets and Metrics
Evaluation datasets are RECON, SaCSoN, GO Stanford and SCAND. Additional video sources are ScanNet, DL3DV, CityWalker and LAVN. Defaults are three history frames, six future frames/waypoints and 256×192 images. Metrics include ATE, RPE, PSNR, SSIM and LPIPS.
Go to primary source ↓e10PDF p. 5, Section 4.1, Implementation details
Initialization uses Wan2.1-T2V-1.3B with frozen VAE. AdamW rates are 5×10^-6 and 5×10^-5. Training uses four RTX 4090 GPUs for two epochs, batch size one per GPU and four accumulation iterations. Weights are 1, 5 and 0.01; default inference uses two steps.
Go to primary source ↓e11PDF p. 5, Section 4.2; PDF p. 6, Table 1, ATE columns and RECON RPE column
Main results exclude extra video-only data. Best UniNav ATEs are 0.314, 0.464, 0.687 and 0.463, versus strongest baseline values 0.326, 0.616, 0.840 and 0.497. RECON RPE is 0.121 for Full, 0.122 for Fast and 0.115 for NWM. Section 4.2 reports relative ATE reductions of 3.7%, 24.7%, 18.2% and 6.8%.
Go to primary source ↓e12PDF p. 6, Table 2 and Effect of Camera-Token Geometry Supervision
ATE/RPE are 0.321/0.125 without camera tokens, 0.299/0.102 with clean conditioning and 0.314/0.121 with denoising. The text identifies DA3 and explicitly labels clean future-frame conditioning as an oracle.
Go to primary source ↓e13PDF p. 6, Table 3; PDF pp. 6–7, Effect of Mixed Navigation and Video-Only Training
Without camera denoising, adding video changes ATE/RPE from 0.321/0.125 to 0.330/0.125. With denoising, it changes 0.314/0.121 to 0.302/0.109.
Go to primary source ↓e14PDF p. 7, Table 4, all model/step rows; caption and Model Profile
Latency is measured on one RTX 4090. One-step Fast has ATE/RPE 0.318/0.123, latency 0.109 s and NFE 1; two-step Fast has 0.316/0.122, 0.212 s and NFE 2; two-step Full has 0.314/0.121, 0.389 s and NFE 2. ViNT is 0.659/0.215 at 0.015 s. NWM is listed at approximately 900 s and 12,000 NFE. Fast ATE is 0.300 at 10 steps, 0.313 at 20 and 0.331 at 50, showing nonmonotonic accuracy.
Go to primary source ↓e15PDF p. 7, Section 5, Limitations and Future Work
The authors describe short-horizon local prediction, primarily offline evaluation, and future work on long-horizon planners and broader closed-loop deployment.
Go to primary source ↓e16PDF pp. 5–6, Section 4.3 opening; PDF p. 7, Section 4.4
The paper directs visual-prediction metrics, alternative prediction designs and robot-collected-data evaluations to an Appendix. The supplied main text discusses qualitative figures without numerical visual-quality results.
Go to primary source ↓e17PDF p. 4, Figure 2, training/inference panels and caption; Eq. (3); PDF p. 5, Section 3.4
Figure 2 shows visual, geometry, trajectory, history, goal and registry tokens during training, with video, camera and trajectory losses. Its inference sketch displays image and trajectory refinement but no camera-token branch. Equation (3) includes camera tokens, and Section 3.4 says Full uses the complete sequence to jointly denoise camera, video and trajectory tokens. The text calls registry tokens register tokens; the Fast attention mask is specified in prose, not drawn here.
Go to primary source ↓e18PDF p. 6, Figure 3 and caption; PDF p. 7, Section 4.4
Four qualitative rows cover RECON and SCAND. Each shows three context images, six predicted images and a trajectory plot with blue GT and red Pred curves. No ground-truth future-image row, numerical plot axes or aggregate visual-quality statistic is included.
Go to primary source ↓e19PDF p. 7, Figure 4 and caption; Sections 4.4 and 5
Figure 4 shows context/prediction image rows and red trajectory curves for video-only datasets. Its caption states that training data lack waypoint annotations, and it supplies no ground-truth trajectory legend or trajectory error. Section 4.4 refers additional visualizations and robot-collected-data results to an Appendix; Section 5 states that evaluation is primarily offline.
Go to primary source ↓8.5 Primary sources
UniNav: A Unified World-Action Diffusion Model for Visual Navigation ↗
PDF · 5,060 extracted words
Source fingerprint
da1a0751d15c5cd81b1258d534b703007b6dc63771a4ae76e5bb23e89848a601