SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy
1. Paper overview
In one sentence: SpatialVAM makes robot positions look like multiview video, gaining strong low-demonstration control through shared RGB/heatmap prediction while retaining separate action decoding and substantial inference cost. e01e03e04e05e08e12e17e20
| At a glance | What to know |
|---|---|
| Research problem | Source description The paper targets the mismatch between 2D video representations and actions executed in 3D. Its proposed remedy is to encode observations and positional action targets through the same virtual-camera geometry, allowing video pretraining to support spatially grounded trajectory prediction. e02e03 |
| Core mechanism | |
| A key reported result | Real-world basic and unseen manipulation: 57.10 Average success rate (%). Franka Research 3; about ten demonstrations per base task; three basic and four unseen settings, ten trials each. BridgeVLA 41.42; UVA 5.70; pi0.5 1.40; DP3 0.00. SpatialVAM records 10/10, 4/10 and 7/10 on Put Lion, Push-T and Scoop Tortilla. It improves lighting/category shifts but loses to BridgeVLA on background (5/10 vs 8/10) and height (6/10 vs 7/10). These are execution outcomes, not video-quality scores. e10e11 |
| Reading caution |
Core contributions
Figure 2. One shared predictor generates scene appearance and positional targets; two decoding branches assemble the robot command. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow panel (a) from left to right. Point-cloud projections provide appearance, while projected end-effector positions provide colored heatmaps. Their VAE latents enter the same transformer; predicted heatmaps are decoded and back-projected into positions. A separate branch reads denoised latents for rotation and gripper state. Panel (b) inserts view attention between self-attention and text cross-attention. Section 3.2 specifies a temporary (B,T,V×H×W,C) layout, with B the batch, T time, V views, H/W spatial dimensions and C channels. Panel (c) expands the action decoder: temporal upsampling precedes global/local feature extraction, fusion, view aggregation and conditioning-frame attention. e03e04e05e06e15
What it supports. The coupling is architectural: future RGB and positional heatmaps share the predictive transformer, and translation is recovered from generated heatmaps. This supports joint video/action-position prediction. The 170M rotation/gripper branch nevertheless has its own training objective and uses noisy ground-truth latents during training; the complete policy is not a single undifferentiated predictor.
Where the evidence stops. Panel (b) repeats the original shape labels around reshaping; use Section 3.2's explicit view-attention layout. Section 3.3 also abbreviates back-projection as peak decoding, whereas Appendix B.3 specifies a 3D candidate search maximizing mean heatmap probability. The crop preserves those schematic simplifications.
2. Motivation
2.1 The problem and the proposed response
The paper targets the mismatch between 2D video representations and actions executed in 3D. Its proposed remedy is to encode observations and positional action targets through the same virtual-camera geometry, allowing video pretraining to support spatially grounded trajectory prediction. e02e03
2.2 What this reading follows
A video model understands images, while a robot must move through three-dimensional space. SpatialVAM connects those formats by rendering a colored point cloud from virtual cameras and representing end-effector positions as heatmaps in the same views. Its video backbone predicts how both the scene and those positional targets evolve. The reading below follows that representation into executable commands, then examines where the reported improvements come from. The strongest evidence concerns task success with few demonstrations and controlled architecture variants. Calibration, positional resolution, a separately trained rotation/gripper decoder and slow chunk generation remain important boundaries. This edition covers the supplied August 2026 arXiv v2. e01e03e04e05e08e12e17e20
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The shared DiT jointly generates future scene RGB and heatmaps that encode action positions, supporting WAM, multiview modeling and joint prediction. 'One Model' is supported at the shared predictive-backbone level, not as a claim of one trainable module: rotation/gripper decoding is separately trained. This is not an independent video planner followed by an inverse-dynamics model, and future prediction participates in inference rather than serving only as an auxiliary training loss. e04e05e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make a spatial target compatible with a video backbone
Start with the geometric correspondence. The point cloud and end-effector position are expressed in one workspace, then projected through the same virtual cameras. Consequently, a heatmap peak and the robot in an RGB projection refer to compatible image coordinates. The shared VAE can encode both because the heatmaps are colorized into three channels. This gives the pretrained transformer an image-like action representation while preserving a route back to three-dimensional coordinates. Appendix B.3 makes that route concrete: score candidate workspace points using their projected heatmap values across views and select the highest mean. As a reader interpretation, much of the useful 3D structure is supplied by this representation and decoder. The architecture need not learn an unconstrained mapping from unrelated image pixels directly to Cartesian commands, but it still depends on observed geometry and camera configuration. e03e04e07e15
5.2 Separate joint prediction from complete action generation
The video transformer jointly predicts future RGB and heatmap latents, so its positional action representation participates directly in future prediction. The remaining action components follow another path. During training, the rotation/gripper predictor receives ground-truth latents with small noise, rather than being described as jointly optimized through sampled video rollouts. During deployment, it receives denoised predicted latents, combines global context with local features around heatmap peaks, and attends to the conditioning frame. Its outputs are combined with geometrically decoded translation and sent to the controller. The reader's interpretation is that the catalog's shared-model label describes the central predictive representation, while the complete policy remains modular. This distinction also exposes a reproduction question: how much error arises from imperfect generated latents reaching a decoder trained on cleaner inputs? The paper does not isolate that distribution shift. e04e05e06
5.3 Read demonstration efficiency alongside control and deployment costs
The experiments ask whether useful manipulation can be learned from few demonstrations, and the Meta-World and robot tables support that claim within their stated tasks. Table 3 then shows why the result cannot be described as learning without pretrained knowledge: removing video initialization collapses performance. Figure 5 adds a different lesson: predicting scene RGB helps action success, yet spending more inference steps does not consistently improve it. That does not make video quality irrelevant, because the proposed human safety check depends on readable predictions. Finally, Appendix C reports costly multi-GPU training and a measured 4.6-second action-chunk latency. As a reader interpretation, demonstration count, execution success, video readability and feedback latency are separate axes of efficiency. A convincing deployment comparison would measure all four under one documented control schedule, instead of converting within-chunk action throughput into a replanning rate. e08e11e12e13e17e19e21
5.4 Training and inference
During training
Apply consistent SE(3) augmentation to point clouds and end-effector trajectories. Fine-tune the video transformer with LoRA and weighted RGB/heatmap objectives. Separately train the action decoder on ground-truth latents with small added noise, using cross-entropy for 72-bin Euler angles and binary gripper state. Predictions are relative to the conditioning frame. The pretrained VAE is reused; the paper identifies the transformer and action decoder as its trainable modules. e06e15
The objective is internally inconsistent: Sections 3.2–3.4 describe added-noise prediction with MSE, while Appendix A.2 describes flow-matching velocity prediction. The supplied text does not reconcile their targets or schedules. 'No additional pretraining' must not be read as training from scratch. e01e06e07e12
During inference
Encode the current observation and pose, predict clean future latents through iterative denoising, then decode heatmaps and rotation/gripper states in parallel. Combine these into commands executed by the robot controller. The default predicts 24 future frames with five denoising steps; the exact reobservation and chunk-execution schedule is not specified. e05e06e14
5.5 Implementation flow
- Render an aligned spatial representation
Crop the point cloud to a one-cubic-meter workspace and render three fixed orthographic virtual views; their directions need not be mutually orthogonal. Project end-effector positions into the same views and colorize truncated Gaussian heatmaps. Virtual rendering rearranges observed geometry but cannot recover unseen surfaces. e03e15
- Predict appearance and position together
A shared pretrained VAE encodes RGB and heatmaps. Concatenate their latents along the view dimension. Wan2.2's 5B transformer gains view-attention in each block: Section 3.2 reshapes tokens from (B,V,T×H×W,C) to (B,T,V×H×W,C), enabling interaction across views at each time. Text enters cross-attention. e04
- Recover executable action components
Decode heatmaps and recover translation geometrically. Although Section 3.3 calls this peak back-projection, Appendix B.3 specifies scoring discretized workspace candidates by their mean heatmap probability across three projections and selecting the maximum. The 170M rotation/gripper decoder temporally upsamples denoised latents, fuses global and peak-centered local features, aggregates views and attends to the conditioning frame before a four-layer transformer and separate heads. e05e15
6. Experiments & results
SpatialVAM turns a colored point cloud and robot position into aligned multiview RGB images and heatmaps, then adapts a pretrained video model to predict both future sequences. Geometric decoding recovers translation, while a separate latent decoder supplies rotation and gripper commands. The method performs strongly with few demonstrations, but depends on pretrained weights, calibrated depth sensing and expensive inference; visual plausibility alone does not establish safe execution.
6.1 Read the original evidence
Table 1. The strong Meta-World average includes a pronounced faucet-closing weakness. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task cell as successful rollouts out of 25, then use the rightmost column for the reported average percentage. Section 4.1 specifies five demonstrations per task and a 600-step evaluation horizon. The SpatialVAM row is particularly strong in door opening/closing, button pressing and handle manipulation, but the Fct-Cls column breaks the pattern. Compare that cell with DreamZero before interpreting the average as uniform dominance. Also inspect Fct-Open, where DreamZero records 25/25 against SpatialVAM's 24/25. The table therefore supports an aggregate advantage across these seven tasks, with meaningful variation across individual skills. e08e18
What it supports. SpatialVAM reports 89.1% average success, versus 67.4% for Track2Act and 61.1% for the evaluated DreamZero variant. The gap to Track2Act is 21.7 percentage points. Faucet-close remains only 8/25, compared with DreamZero's 20/25; representation alignment has not eliminated task-specific failures.
Where the evidence stops. Appendix F says baseline values follow prior works and separately describes a smaller Wan2.2-TI2V-5B backbone substitution for DreamZero. Interpret the displayed benchmark comparison within that mixed provenance; it does not establish a uniformly rerun, compute-matched ranking or seed-level statistical significance.
Table 2. Continuous-motion tasks improve, while background and height shifts remain stronger for BridgeVLA. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the three Basic Tasks columns, evaluated on a Franka Research 3 with about ten demonstration trajectories per base task. Each cell is a count out of ten trials. Then read the four Unseen Tasks: Put-B changes the table background, Put-H raises the lion on a 5.5 cm box, Push-L turns off ambient lighting, and Scoop-C substitutes plastic noodles for tortilla. The paper reports manually aligning photographed initial scenes across methods. Compare SpatialVAM and BridgeVLA within each column before reading their averages, because their relative advantage changes between motion-intensive tasks and appearance or height shifts. e10e11e18
What it supports. SpatialVAM reaches 57.10% overall versus BridgeVLA's 41.42%. Its 4/10 Push-T and 7/10 Scoop Tortilla counts exceed BridgeVLA's 0/10 and 4/10, respectively. Conversely, Put-B and Put-H favor BridgeVLA. These are physical manipulation outcomes under a limited demonstration budget, not a measure of generated-video realism.
Where the evidence stops. Ten trials per setting provide coarse success estimates without reported uncertainty across training seeds. The aggregate covers three basic tasks and four variations on them; it does not establish broad open-world competence or isolate video pretraining from all baseline architecture differences.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Seven-task Meta-World manipulation Five demonstrations per task; 25 random-initialization rollouts per task; 600-step success horizon. | 89.1 Average success rate (%) | Track2Act 67.4; modified DreamZero 61.1; DP 37.7. The reported gap to Track2Act is 21.7 percentage points. Faucet-close remains 8/25. Appendix F says baseline numbers follow prior works and separately specifies replacing DreamZero's backbone with Wan2.2-TI2V-5B; this is not a uniformly rerun, compute-matched comparison. e08e18 |
| Ten-task RoboCasa kitchen manipulation Ten demonstrations and 50 rollout trials per task. | 35.8 Average success rate (%) | Cosmos Policy 20.0; 3D Diffuser Actor 7.6. A 15.8-point gain over Cosmos Policy in this low-data setting, while most trials still fail. The source does not enumerate the ten tasks or provide per-task results, seeds or uncertainty. e09 |
| Real-world basic and unseen manipulation Franka Research 3; about ten demonstrations per base task; three basic and four unseen settings, ten trials each. | 57.10 Average success rate (%) | BridgeVLA 41.42; UVA 5.70; pi0.5 1.40; DP3 0.00. SpatialVAM records 10/10, 4/10 and 7/10 on Put Lion, Push-T and Scoop Tortilla. It improves lighting/category shifts but loses to BridgeVLA on background (5/10 vs 8/10) and height (6/10 vs 7/10). These are execution outcomes, not video-quality scores. e10e11 |
| Meta-World architecture ablation Table 3 variants and Figure 5's predicted-RGB-view sweep. | Default 89.1; full fine-tuning 87.4; channel concatenation 81.1; translation regression 76.6; no pretrained weights 4.6. Average success rate (%) | Predicting zero, one, two or three additional RGB views gives 61.1, 72.0, 81.7 and 89.1, respectively. Pretrained initialization and aligned heatmap decoding matter substantially. The channel variant also adds an adapter, and changing predicted views changes supervision and token workload; these are not pure tests of spatial understanding. e12e13 |
| Meta-World denoising-step robustness Inference-step sweep with the trained policy. | One step 85.7; five steps 89.1; forty steps 91.4; fifty steps 87.4. Average success rate (%) | The selected video examples show blurrier RGB at one step but relatively stable heatmap peaks. Execution success is relatively insensitive here, without evidence that a single-step video is equally useful for human safety screening. e13e21 |
| Real-world camera and calibration robustness Same seven settings, ten trials each; camera-coverage and extrinsic-perturbation tests. | Three cameras 57.1; one camera/original views 50.0; one camera/adapted views 60.0. Average success rate (%) | With 20 mm translation and 2-degree rotation perturbations: train/test mismatch 51.4; shared training/evaluation bias 54.3. Virtual-view selection mitigates incomplete coverage in this setup. Small trial counts do not establish that one camera is superior or that arbitrary calibration errors are harmless. e16 |
| Human video checking before deployment Four evaluators, 35 rollouts each; apparently unsafe rollouts were rerun. | With checking: 0/140 Reported collision events | Without checking: 6/140. This supports an assisted screening workflow, not a collision-free policy guarantee. Rejection counts, rerun costs and a detailed paired/randomized protocol are not reported. e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. More RGB prediction helps this policy more consistently than additional denoising steps. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two panels as different interventions. The left changes inference denoising steps N; its equally spaced tick positions represent unequal increments, so compare labeled settings rather than treating horizontal distance as proportional compute. One step gives 85.7%, five give 89.1%, forty give 91.4% and fifty give 87.4%. The right varies how many RGB views are predicted in addition to heatmap sequences: zero means heatmap-only prediction. Its bars rise through 61.1%, 72.0%, 81.7% and 89.1%. Both vertical axes measure manipulation success, not pixel accuracy or human ability to detect unsafe actions in predicted videos. e08e13e17e19
What it supports. The monotonic RGB-view trend supports retaining scene prediction alongside action heatmaps in this low-data setup. Meanwhile, one-step inference preserves much of the reported control performance. This suggests that the information needed for positional decoding can survive when additional denoising contributes little to task success.
Where the evidence stops. The RGB-view sweep changes supervision and token workload as well as view count. No error bars are shown. The nearby 5 Hz claim conflicts with measured 4.6-second chunk latency elsewhere; these plots do not measure feedback rate or resolve that discrepancy.
Table 3. The largest ablation loss comes from removing pretrained weights; heatmap decoding also matters. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use row 1 as the reference and decode the abbreviated column names using the source caption. View Concat means joining RGB and heatmap latents along the view dimension; Init W means pretrained video-model initialization; LoRA means adapting the backbone with low-rank updates. A dash under LoRA in row 2 denotes full fine-tuning. Row 3 changes to channel concatenation with an added convolutional adapter. Row 4 replaces heatmap-based translation decoding with TF-Regress, a transformer that regresses translation from latents. Row 5 removes pretrained initialization. The final column records average Meta-World success for each variant. e12
What it supports. Removing initialization drops success from 89.1% to 4.6%, making pretrained knowledge central to the claimed demonstration efficiency. Translation regression yields 76.6%, supporting the geometric heatmap route in this implementation. Full fine-tuning at 87.4% supplies no reported gain over the cheaper LoRA default.
Where the evidence stops. These variants test complete implementation choices. Channel concatenation also introduces an adapter, and no-pretraining failure may reflect optimization difficulty under this budget. The table provides neither uncertainty nor evidence that every alternative was independently tuned to its best achievable performance.
Figure 13, top example. One-step RGB becomes blurry even when positional heatmap peaks remain visually stable. Original paper, p. 28 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read time from left to right within each filmstrip. The upper block is RGB and the lower block is Heatmap; each has rows labeled steps=1, steps=5 and steps=50. Compare the same column vertically to assess the effect of denoising, then follow a row horizontally to inspect the predicted trajectory. In this top example, the moving robot becomes much less distinct toward the right of the one-step RGB row. The corresponding heatmaps remain sparse, with visible peaks. This crop retains the complete first example from Figure 13; it excludes the page's other two examples and its long caption. e05e13e19e21
What it supports. The visual difference supports the authors' distinction between scene appearance and the simpler positional signal used for translation. Together with Figure 5's one-step success result, it motivates measuring action accuracy and video quality separately. The image is qualitative corroboration, not a quantitative estimate of heatmap localization error.
Where the evidence stops. This is one selected predicted sequence, not a ground-truth comparison or a collision-screening evaluation. Similar-looking peaks do not verify rotation, gripper state or safe execution, and degraded RGB could still impair the human-checking workflow demonstrated elsewhere.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors require at least one fixed, approximately calibrated RGB-D camera. Appendix H attributes scooping failures to positional decoding at 256×256 resolution, approximately 4 mm per pixel; the benefit of increasing resolution is proposed, not measured. e19e20
Section 4.4 claims 5 Hz inference, but the conclusion and Appendix C report 4.6 seconds per 24-action chunk on an A100, averaged over 30 timings. Throughput of actions within a chunk cannot establish 5 Hz feedback or replanning; the wording remains unresolved. e13e17e19
Hyperparameter summaries disagree. Table 7 gives lambda=0.1/0.5/0.9: 92/89.1/89.1%, and sigma=1.5/2.5/3.5: 89.1/89.7/86.9%. Section 4.4 reports changes of 3.3%/2.5%, Appendix E 2.9%/2.2%; neither consistently states the full table ranges. No seed uncertainty supports significance claims. e22
7.2 Questions for discussion
- Would the RGB-view benefit persist with matched token count, compute and geometric information?
- Can one-step predictions preserve collision-screening accuracy as well as action success?
- How much scooping error comes from heatmap discretization versus rotation/gripper decoding?
8. Reproducibility audit
8.1 Requirements and known gaps
Table 4 supplies AdamW, learning rate 0.0001, 100 epochs, batch size 64, 256×256 images, sigma=1.5, LoRA rank 32 on q/k/v/o/ffn.0/ffn.2, 1,000 training timesteps and guidance 1. Weight decay is zero for the video transformer and 0.00001 for the action decoder. e14
Reported training uses 32 H20 GPUs for Meta-World, 21.8k steps and approximately 42/36 hours for the two modules; real-world training uses 32 H200 GPUs, 8.6k steps and 11/10 hours. Deployment exceeds 30 GB GPU memory and uses FastAPI communication. Data efficiency therefore does not imply low compute cost. e17
A faithful implementation still needs the numerical heatmap threshold, workspace-grid resolution, rendering/splatting settings, virtual-camera transforms, augmentation ranges, latent-noise magnitude, exact objective/sampler, control timing and versioned environment dependencies. These are absent or insufficiently specified. First resolve the objective discrepancy, then test decoding error separately from imitation performance. e03e06e07e14e15e17e20
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Locate the positional error before changing the policy
Reader-proposed check: take held-out Scoop Tortilla end-effector trajectories and run three decoding conditions with identical camera geometry: analytic ground-truth heatmaps, ground-truth heatmaps after a VAE encode/decode round trip, and policy-predicted heatmaps. Use the Appendix B.3 candidate-scoring decoder in all conditions; report millimeter translation error and vertical error near scoop contact. Repeat at 256×256 and a higher resolution while declaring both pixel scale and 3D candidate-grid spacing, and initially hold rotation/gripper commands fixed to isolate position. If resolution is the dominant limitation, oracle/round-trip errors and scoop-contact failures should decrease as the grid is refined. If only predicted heatmaps fail, increasing resolution alone would not support the authors' proposed remedy. No such experiment was run here. e03e05e14e15e20
Check 2: Test whether one-step videos still support human screening
Reader-proposed check: hold the trained checkpoint, initial scenes and controller schedule fixed, and compare one, five and fifty denoising steps using paired noise seeds where possible. Log end-to-end chunk latency, translation/rotation/gripper errors and robot success separately. In simulation, present shuffled, unlabeled predictions to evaluators before executing every proposed action chunk to obtain a collision outcome; record rejection decisions, unsafe-action misses, false rejections and all reruns. The falsifiable expectation is that one-step action success may remain close while blur increases unsafe-action misses. Equivalent screening accuracy would weaken that concern; a gap would separate control efficiency from visual explainability. Report chunk-generation latency and actual feedback frequency independently to resolve the paper's timing ambiguity. This is a proposed study, not a reproduced result. e05e13e14e17e19e21
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, all main-text method and experiment pages, and every appendix page. This includes all 14 figures, all seven tables, configuration and hardware details, baseline qualifications, failure analysis and qualitative video sequences. The six final crops were each inspected; the Meta-World crop was widened and inspected again to retain the success-direction arrow. Figure 2's branches and shape labels were compared with Sections 3.2–3.4 and Appendix B.3; the schematic/text differences are disclosed in its caution. Figure 13 is a faithful crop of its first example only. Reference pages 11–14 were read in the complete text but were not visually inspected; external references and separate supplements were not reviewed. All retained method, numerical, evaluation and reproduction claims are supported on the declared pages.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block, abstract and version stamp, PDF p. 1
- Sections 1–2.3: Introduction and Related Work, PDF pp. 1–3
- Sections 3.1–3.4: Projection, diffusion, action decoding, training and inference, PDF pp. 3–6
- Sections 4.1–4.6: Experiments, robustness, ablations and human video checking, PDF pp. 6–10
- Section 5: Conclusion and Future Work, PDF p. 10
- References, PDF pp. 11–14
- Appendix A.1–A.4: VAE, DiT, decoder and cache, PDF pp. 15–16
- Appendix B.1–B.3: Virtual views, rendering and back projection, PDF pp. 16–17
- Appendices C–E: Configuration, compute and robustness, PDF pp. 17–20
- Appendices F–H: Baselines and qualitative failures, PDF pp. 19–22
- Appendix I and Figures 8–14: Video predictions, PDF pp. 22–29
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity/version note: the inspected title page identifies arXiv:2604.03181v2 [cs.RO], 13 August 2026. Its title and all 16 authors agree with the current catalog fields. The supplied observedTitle removes the space after the colon; that exact supplied string is preserved in provenance and metadata.
- The catalog BibTeX instead names 'Multi-View Video Diffusion Policy: A 3D Spatio-Temporal-Aware Video Action Model'. No earlier PDF or primary revision history was supplied, so the relationship of that title to v2 and any scientific changes from v1 remain unverified. This report reviews only the supplied v2 artifact.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by visually inspecting the supplied PDF pages and final crops.
- Separate supplemental material availability has not been fully verified.
- All eight supplied text chunks were read individually. PDF pages 1–10 and 15–29 were visually inspected; reference pages 11–14 were read as text only.
- Code, linked websites, external baseline papers and separate supplements were not inspected. No experiments were reproduced. Source inconsistencies remain explicit below.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author/affiliation block, arXiv margin stamp and Abstract
The title page names SpatialVAM, lists all 16 catalog authors and seven numbered affiliations, and identifies arXiv:2604.03181v2, 13 August 2026. The abstract says no additional pretraining.
Go to primary source ↓e02PDF pp. 1–3, Introduction, Figure 1 and Sections 2.2–2.3
The motivation is joint spatial and temporal modeling through aligned multiview observation/action representations; novelty and broad superiority are author claims.
Go to primary source ↓e03PDF pp. 3–4, Section 3.1 and Equations (1)–(2)
A one-cubic-meter crop is rendered from three fixed orthographic virtual cameras. Projected positions define colorized Gaussian heatmaps, retained when their values are at least tau; camera directions need not be orthogonal.
Go to primary source ↓e04PDF pp. 4–5, Figure 2(a–b), caption and Section 3.2
Wan2.2's 5B transformer gains view-attention; RGB and heatmap VAE latents concatenate along views. The text specifies (B,T,V×H×W,C) during view attention, although Figure 2(b) repeats the original layout labels around reshaping.
Go to primary source ↓e05PDF pp. 4–6, Figure 2(c), Sections 3.3–3.4 and inference continuation
Positions come from heatmap back-projection; a 170M decoder uses upsampled latents, global/local features, view aggregation, conditioning-frame attention, a four-layer transformer and separate rotation/gripper heads. The controller executes combined chunks.
Go to primary source ↓e06PDF p. 5, Sections 3.2–3.4, Equations (3)–(4)
The main text describes noise-prediction MSE, LoRA, weighted RGB/heatmap losses, SE(3) augmentation and separate decoder training on noisy ground-truth latents. Euler angles use 72 bins and gripper state is binary.
Go to primary source ↓e07PDF pp. 15–16, Appendix A.1–A.4, especially A.2 and Equation (6)
Appendix A describes VAE spatial/temporal compression, decoding and feature caching. A.2 instead describes flow-matching velocity prediction, without reconciling this with main-text noise prediction.
Go to primary source ↓e08PDF p. 6, Table 1, all rows, and Section 4.1 setup
Seven Meta-World tasks use five demonstrations and 25 evaluation trials each with a 600-step horizon. SpatialVAM reports 89.10%, Track2Act 67.40%, DreamZero 61.10% and DP 37.70%; SpatialVAM faucet-close is 8/25.
Go to primary source ↓e09PDF p. 7, Figure 3 and Section 4.2
Ten RoboCasa tasks use ten demonstrations and 50 rollout trials per task. Figure 3 labels average success as 7.6% for 3D DA, 20% for Cosmos Policy and 35.8% for SpatialVAM; task identities and per-task scores are not supplied.
Go to primary source ↓e10PDF p. 7, Figure 4 and Section 4.3 setup
The robot is a Franka Research 3 with three calibrated ZED2i cameras. About ten expert demonstrations are collected per task; ten trials are evaluated per setting with manually aligned scenes. Unseen settings change cloth background, height by 5.5 cm, lighting, or tortilla to plastic noodles.
Go to primary source ↓e11PDF p. 8, Table 2, all rows/columns and Section 4.3 results
SpatialVAM's seven counts are 10,4,7,5,6,3,5 out of ten; BridgeVLA's are 9,0,4,8,7,0,1. Reported averages are 57.10% and 41.42%, with other baselines at 0.00%, 1.40% and 5.70%.
Go to primary source ↓e12PDF pp. 9–10, Section 4.5 and Table 3, rows 1–5 with caption definitions
Default/full fine-tuning/channel concatenation/translation regression/no pretrained weights yield 89.1/87.4/81.1/76.6/4.6%. Channel concatenation uses an added convolutional adapter. Removing pretrained weights fails even to fit the training set, according to the authors.
Go to primary source ↓e13PDF pp. 9–10, Figure 5, Section 4.4 modeling robustness and Section 4.5 continuation
Denoising steps 1,2,3,4,5,10,25,40,50 yield 85.7,87.4,88.0,88.6,89.1,88.0,88.6,91.4,87.4%. Zero through three predicted RGB views yield 61.1,72.0,81.7,89.1%. Section 4.4 claims five-step inference permits 5 Hz on A100.
Go to primary source ↓e14PDF p. 17, Table 4, both module columns
Configuration includes AdamW, learning rate 0.0001, 100 epochs, batch 64, 256×256 images, sigma 1.5, 24 future frames, equal RGB/heatmap weights, rank-32 LoRA, 1,000 training timesteps, five inference steps and guidance 1; module weight decays differ.
Go to primary source ↓e15PDF pp. 16–17, Appendix B.1–B.3
Virtual views expose available geometry and preserve consistent SE(3) augmentation but cannot restore unseen surfaces. Rendering uses depth ordering and screen-space splatting. Back-projection selects the discretized 3D candidate maximizing average heatmap probability across views; exact grid and camera configurations are not given.
Go to primary source ↓e16PDF pp. 18–19, Appendix D.1–D.2, Tables 5–6 and Figure 7
Ten trials per real-world task give camera-setting averages 57.1/50.0/60.0%. Extrinsics perturbed by 20 mm and 2 degrees give 51.4% with train/test mismatch and 54.3% with consistent bias. Figure 7 contrasts original and adapted virtual views.
Go to primary source ↓e17PDF p. 18, Appendix C continuation, compute and deployment paragraphs
Meta-World uses 32 H20 GPUs, 21.8k steps and approximately 42/36 hours; real-world training uses 32 H200 GPUs, 8.6k steps and 11/10 hours. A100 inference needs over 30 GB, uses FastAPI and averages 4.6 seconds per 24-action chunk over 30 timings.
Go to primary source ↓e18PDF pp. 19–21, Appendices F–G, especially F opening and DreamZero paragraph
Appendix F says baseline table values follow prior works and separately describes replacing DreamZero's Wan2.1-I2V-14B with Wan2.2-TI2V-5B. Appendix G describes different baseline inputs and prediction formats, including BridgeVLA key poses and motion planning.
Go to primary source ↓e19PDF p. 10, Figure 6, Section 4.6 and Section 5
Four evaluators perform 35 rollouts each and rerun unsafe-looking predictions; the chart reports 0/140 collisions with checking versus 6/140 without. The conclusion requires a fixed calibrated RGB-D camera and reports 4.6-second chunk latency, limiting high-frequency dexterity.
Go to primary source ↓e20PDF pp. 21–22, Appendix H, particularly SpatialVAM failure mode
Qualitative baseline failures are analyzed. SpatialVAM sometimes scrapes above the tortilla; the authors attribute this to finite 256×256 heatmap resolution, approximately 4 mm per pixel, and propose higher resolution.
Go to primary source ↓e21PDF p. 22, Appendix I; PDF pp. 23–29, Figures 8–14, especially Figure 13 top panel on p. 28
Selected predicted and ground-truth multiview sequences illustrate heatmap/RGB alignment. The 1/5/50-step visual comparisons show blurrier one-step RGB and relatively stable heatmap locations; they do not report quantitative dynamics or collision-screening accuracy.
Go to primary source ↓e22PDF p. 9, Section 4.4; p. 18, Appendix E; p. 20, Table 7
For lambda 0.1/0.5/0.9, Table 7 reports 92/89.1/89.1%; for sigma 1.5/2.5/3.5, it reports 89.1/89.7/86.9%. Main-text changes of 3.3%/2.5% differ from Appendix E's 2.9%/2.2% and do not consistently describe the full table ranges.
Go to primary source ↓8.5 Primary sources
SpatialVAM:Spatial-Aware Multi-View Video Diffusion as a Data-Efficient Robot Policy ↗
PDF · 12,029 extracted words
Source fingerprint
c2d392cc08df1e95c418d5730ed2a88638009ba473e8138352c6e06cbd2c1769