S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresight
1. Paper overview
In one sentence: S-VAM distills generated-video geometry and semantics into first-step foresight that improves manipulation while adding a modest latency cost over raw-feature control. e-shortcute-decouplerse-distille-actione-ablatione-latency
| At a glance | What to know |
|---|---|
| Research problem | Author claim Video-based robot policies face a latency–representation tradeoff: iterative video generation is expensive, while early denoising features can be entangled and temporally unstable. The authors seek structured future information suitable for precise monocular manipulation without generating complete videos at deployment. e-shortcut |
| Core mechanism | Source description Two spatiotemporal decouplers predict geometry-oriented DPAv3 features and patch-level DINOv2 semantics, anchored by representations of the current image. e-decouplers |
| A key reported result | CALVIN ABC→D long-horizon manipulation: 4.16; stages 1–5: 95.8, 90.7, 83.7, 77.0, 68.9. Average completed sequence length; stage-wise success (%). Train ABC; evaluate unseen D; primary third-view monocular RGB. VPP 3.58; HiF-VLA 4.08; Spatial Forcing 3.94. Best reported average, +0.58 over VPP; HiF-VLA remains higher at stage 5 (69.4%). No uncertainty is supplied. e-setupe-calvin |
| Reading caution | Reader analysis Figures 3–4 show selected attention trajectories, PCA features and probe-based depth maps. They illustrate successful versus failed rollouts; they do not quantify depth accuracy or establish that attention coherence causes success. e-qualitative |
Core contributions
- Source description
Two spatiotemporal decouplers predict geometry-oriented DPAv3 features and patch-level DINOv2 semantics, anchored by representations of the current image. e-decouplers
Figure 2. Current-image anchors and retained diffusion features connect the shortcut to action prediction. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the lower-left reference frame. It enters both the video backbone and the frozen VFM encoder; the instruction is encoded by CLIP. The middle concatenation joins one-step diffusion features with current-image reference features. Each decoupler then follows spatial attention, temporal attention and output projection, consistent with Eqs. (5)–(6) and the projection paragraph. Follow the bottom bypass arrow: raw diffusion features remain in the conditioning context alongside both foresight branches. At right, learnable queries attend to this context before self-attention and a feed-forward layer. The resulting tokens and instruction embedding condition the diffusion policy. e-backbonee-featurese-decouplerse-distille-action
What it supports. The shortcut changes what the action expert receives: it adds structured predicted representations without discarding the original feature volume. Generated future videos are needed to teach the decouplers, but the deployed path shown here retains current-image VFM anchoring and a distinct action-generation module.
Where the evidence stops. The single-step claim concerns video-feature extraction. Section 3.3 still specifies action diffusion without a sampler-step count. Eq. (2) also leaves image-versus-feature interpolation placement ambiguous; the faithful diagram cannot resolve that implementation detail.
2. Motivation
2.1 The problem and the proposed response
Video-based robot policies face a latency–representation tradeoff: iterative video generation is expensive, while early denoising features can be entangled and temporally unstable. The authors seek structured future information suitable for precise monocular manipulation without generating complete videos at deployment. e-shortcut
2.2 What this reading follows
A robot may benefit from imagining an interaction without waiting for a video generator to finish every frame. S-VAM trains two small decouplers to translate early diffusion features into the representation spaces of geometric and semantic vision models. Their teachers see the video backbone’s completed imagined future; the deployed students use only its first-step features and the current-image reference. A separate action expert turns this information into controls. The central question is whether this shortcut preserves useful future structure. Read the architecture alongside the target-replacement ablation, then distinguish the reported task gains from the more limited evidence about rapid visual feedback. e-shortcute-decouplerse-distille-actione-ablatione-latency
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Insufficient evidence to decide
WAM and Dual-system are supported: a video backbone supplies predicted future context to a separately trained action expert. IDM is plausible as a functional foresight-to-action description, but the source specifies a conditional action diffusion objective, not a conventional state-pair inverse-dynamics formulation. Thus Q4 depends on how broadly IDM is defined. Geometry is learned from monocular input; this does not establish explicit multiview control. e-actione-traininge-decouplerse-setup
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Teach the student the future that its own backbone imagined
The self-distillation choice addresses a subtle target problem. A first-step feature volume belongs to a particular noisy video-generation trajectory. The authors argue that a recorded demonstration can follow a different future, even when it begins from the same observation and instruction. Their teacher therefore completes the model’s own multi-step video and encodes it with frozen geometry and semantics models. Separate students learn squared-error mappings from early features to those completed-future representations. Table 3’s ground-truth-target replacement is the key empirical test: its average CALVIN length is 3.82 instead of 4.16. Reader interpretation: this supports compatibility of the selected targets, but does not prove that shared trajectory identity is the only cause; realism and target distribution also change. e-featurese-distille-backbonee-ablation
Figure 3. Read feature visualizations, attention trajectories and executed rollouts as separate evidence. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each sequence from left to right. The top VPP block pairs one-step features with a failed rollout for moving the blue block rightward. The lower S-VAM block adds spatial and semantic foresight above a successful rollout. The large images at left display attention-center trajectories, not measured end-effector paths. The caption explains that spatial foresight is rendered through a trained depth probe, while one-step and semantic features use PCA fitted across the entire sequence. Footnote 3 on page 10 defines attention tracking from the head-averaged final cross-attention layer’s maximum activation and its 3-pixel neighborhood. e-qualitativee-training
What it supports. This selected example is consistent with the authors’ interpretation that more coherent foresight helps the action expert focus on an instructed object. The useful visual distinction is between predicted representations and the simulator rollout beneath them: only the latter illustrates executed task behavior.
Where the evidence stops. PCA colors do not carry fixed object labels, and probe depth is not ground-truth geometry. One selected success/failure pair cannot establish causal dependence on attention coherence or the frequency of this failure mode.
5.2 Use the current image to stabilize foresight, then retain what distillation discards
The decouplers do not infer everything from noisy features alone. Each receives a repeated representation of the current observation, which supplies a spatial anchor across the predicted sequence. After compact projection, spatial and temporal transformer layers produce future features in the geometric or semantic teacher’s channel space. This makes the current-image VFM encoder distinct from the training-only extraction of teachers from generated future frames. The action expert then receives both predicted branches plus the original diffusion feature volume. In the authors’ account, the raw feature preserves residual global context; the Uni-Perceiver condenses the combined volume through learned queries. The ablations are consistent with this design: removing original features reduces average length to 3.93, while removing the Uni-Perceiver reduces it to 3.72. e-decouplerse-shortcute-actione-ablation
5.3 Separate representation quality, action prediction and feedback timing
The source gives three different kinds of evidence. Probed depth, PCA maps and attention trajectories illustrate representation behavior. Simulation tables measure task execution, with 4.16 average CALVIN length and 72.8% MetaWorld success. The physical experiment adds success rates and inference timing on a dual-arm robot. These levels should inform one another without becoming interchangeable: a smooth attention path is not a calibrated dynamics prediction, and a predicted action chunk is not immediate feedback from the world. The 307.6 ms forward pass produces eight actions, supporting the authors’ effective 25 Hz claim. Reader interpretation: useful command throughput can coexist with a slower observation-to-new-plan loop. A perturbation test during chunk execution would therefore assess a capability that the static timing panel cannot establish. e-qualitativee-calvine-metaworlde-reale-latency
5.4 Training and inference
During training
Stage 1 starts from embodied-scene-pretrained SVD weights: fine-tune 100k steps for MetaWorld or 40k for real tasks on four H100 GPUs; CALVIN reuses the VPP-fine-tuned backbone. Stage 2 freezes that backbone and trains decouplers for 50k steps on one H100. Stage 3 freezes both backbone and decouplers, training only the action expert for 60k CALVIN steps or 40k elsewhere on four H100s. e-training
Teacher VFMs are frozen. Generated future videos provide representation supervision, whereas demonstrations provide the action denoising targets. A separately trained DPT depth head is used for qualitative probing. e-backbonee-distille-actione-training
During inference
Encode the current reference and instruction, extract one-step video features, predict both foresight volumes, aggregate them, and denoise an action sequence. Multi-step video generation and future-frame teacher extraction are training-only; current-image VFM anchoring remains in the architecture. The paper does not specify the action sampler’s denoising-step count. e-shortcute-decouplerse-action
Deployment uses one RTX 3090 with 24 GB memory. Eight actions per forward pass support the authors’ stated effective 25 Hz control frequency. This is action-chunk throughput, not evidence of a fresh visual prediction every 40 ms. e-traininge-latency
5.5 Implementation flow
- Extract an inexpensive future feature volume
Stable Video Diffusion conditions on the observation and instruction. At its first denoising step, intermediate upsampling-layer features are resized to a common spatial grid and concatenated across channels into F. This is latent foresight, not an executed trajectory. e-backbonee-features
- Anchor and decouple
Each branch repeats the current-image VFM representation across the future sequence, concatenates it with F, and projects to a compact hidden space. K blocks apply spatial then temporal attention; an output projection restores the target VFM channel dimension. The paper calls its geometric teacher DPAv3; reference 25 identifies Depth Anything 3. e-decouplerse-vfm-name
- Distill a compatible imagined future
Frozen VFMs encode the backbone’s multi-step generated video. Each student branch independently minimizes squared feature error. The authors argue that sharing the diffusion trajectory aligns these targets with the first-step features better than ground-truth future frames do. e-distille-backbone
- Convert foresight into controls
The Uni-Perceiver concatenates geometry, semantics and F, then uses learnable queries, cross-attention, self-attention and a feed-forward network to condense the context. A DiT action policy receives those tokens and the instruction embedding through cross-attention and learns to predict action noise. e-action
6. Experiments & results
S-VAM turns first-step video-diffusion features into predicted geometric and semantic representations, then conditions a separate diffusion policy on that foresight. Generated videos supervise the shortcut during training. Reported manipulation gains come with higher latency than VPP and do not establish equally fast visual feedback (e-shortcut, e-action, e-latency).
6.1 Read the original evidence
Table 1. S-VAM leads the reported average but not every sequence stage. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the far-right average-length column for the headline comparison, then inspect the five stage-wise percentage columns to see where that summary comes from. S-VAM is the shaded bottom row. Compare it first with VPP, the direct baseline for the shortcut, and then with HiF-VLA, the nearest listed competitor by average length. Bold type denotes the best table entry and underlining the second-best. Section 4.1 specifies training on CALVIN environments ABC and testing on unseen D using the primary third-view RGB camera. These are simulated long-horizon manipulation results, distinct from the physical-robot evaluation. e-setupe-calvin
What it supports. The reported average rises from VPP’s 3.58 to 4.16, an absolute gain of 0.58 completed tasks. S-VAM also exceeds HiF-VLA’s 4.08 average, but its fifth-stage success is 68.9%, slightly below HiF-VLA’s 69.4%. Average superiority therefore should not be rewritten as dominance at every stage.
Where the evidence stops. No confidence intervals or seed variability accompany these entries. The paper states the camera/split protocol, but does not provide enough baseline training-budget detail here to treat the ranking as a controlled comparison of backbone efficiency.
Table 2. The overall gain comes with a clear difficulty-dependent tradeoff. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Unlike the CALVIN table, these entries are fractions between zero and one: 0.728 means 72.8% success. The headings give 28 easy tasks, 11 middle tasks and 11 hard tasks, with a separate reported average over all 50. Read S-VAM against VPP across the same row positions. The paper’s setup uses 50 demonstrations per task and only the primary monocular third-view image. The easy column matters because it checks whether improved aggregate performance is uniform. It is not: VPP retains the strongest easy-task entry, while S-VAM leads the middle and hard columns. e-setupe-metaworld
What it supports. S-VAM improves overall success from 68.2% to 72.8%, or 4.6 percentage points. Its largest difficulty-group improvement is on hard tasks, from 52.6% to 68.4%, or 15.8 points. Easy tasks instead decline from 81.8% to 79.3%, a 2.5-point reduction.
Where the evidence stops. The difficulty pattern supports an advantage on the tested harder tasks; it does not isolate geometric reasoning as the cause. Training demonstration count is supplied, but evaluation rollout counts and uncertainty are not stated in this table or setup.
Figure 5. Physical task gains accompany an added inference cost. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the platform image: the task observations come from the front camera, while leader and follower arms belong to the demonstration/deployment setup. Read the four task panels by their printed percentage labels; their horizontal axes start at different nonzero values in several panels, so bar lengths are not directly comparable across tasks. Section 4.5 reports one multi-task model, roughly 50 demonstrations per task and 25 evaluation trials per task. At right, the stacked bars separate video-backbone, decoupler and action-expert costs. Continue to page 15 for the eight-action chunk underlying the stated effective control frequency. e-reale-latencye-training
What it supports. S-VAM improves all four displayed success rates, including transparent-object placement from 16% to 32% and pouring from 20% to 44%. Its forward pass takes 307.6 ms versus VPP’s 265.7 ms. The unchanged 231.0 ms video backbone still contributes most of the latency.
Where the evidence stops. The reported 25 Hz is based on eight-action chunks, not demonstrated 25 Hz visual replanning. There are no uncertainty bars; 25 trials per task and success rates of 24–56% leave substantial physical failures and a limited basis for broad deployment claims.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CALVIN ABC→D long-horizon manipulation Train ABC; evaluate unseen D; primary third-view monocular RGB. | 4.16; stages 1–5: 95.8, 90.7, 83.7, 77.0, 68.9. Average completed sequence length; stage-wise success (%) | VPP 3.58; HiF-VLA 4.08; Spatial Forcing 3.94. Best reported average, +0.58 over VPP; HiF-VLA remains higher at stage 5 (69.4%). No uncertainty is supplied. e-setupe-calvin |
| MetaWorld 50-task manipulation 50 demonstrations per task; monocular third-view RGB; 28 easy, 11 middle and 11 hard tasks. | 72.8% overall; easy 79.3%, middle 60.7%, hard 68.4%. Success rate | VPP: 68.2% overall; 81.8%, 49.3%, 52.6% by difficulty. Overall improvement is 4.6 percentage points; hard-task improvement is 15.8 points. Easy-task performance falls by 2.5 points. e-setupe-metaworld |
| CALVIN component and target ablations CALVIN component removals and teacher-target alternatives. | Full 4.16; no geometry 4.01; no semantics 3.99; GT-future targets 3.82; no Uni-Perceiver 3.72; no raw F 3.93. Average completed sequence length | Alternative pairs: DINOv2 + VGGT 4.04; SigLIP + DPAv3 4.06. The GT-target row retains supervision. Results favor generated targets and complementary features, but do not independently prove the proposed trajectory-misalignment explanation. e-ablatione-targets |
| Real-world four-task manipulation Unified model; AgileX Cobot dual arms, seven degrees of freedom per arm, parallel grippers and front RGB; roughly 50 demonstrations/task; 25 evaluation trials/task. | Place-to-Pot 56; hard transparent-object variant 32; Pour-Water 44; Lift-Pot 24. Task success rate (%) | VPP respectively 40, 16, 20, 12. All four improve; absolute success remains limited. Transparent-object success doubles, while Pour-Water has the largest absolute gain (24 points). e-real |
| Real-world inference latency Single RTX 3090; reported forward-pass latency and eight-action chunks. | 307.6 ms: backbone 231.0, decouplers 40.1, action expert 36.5; effective 25 Hz. Milliseconds per forward pass; reported effective control frequency | VPP 265.7 ms: backbone 231.0, action expert 34.7; S-VAM adds 15.8% latency. The backbone dominates cost; action batching amortizes computation without establishing 25 Hz visual replanning. e-traininge-latency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. The self-distillation row replaces generated targets with ground-truth targets. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the shaded full-model row as the reference, then compare each variant in the average-length column. The names alone are insufficient for the third row: Section 4.3 defines ‘w/o Self-Distillation’ as keeping VFM supervision but extracting it from ground-truth future frames instead of generated video. It is not an unsupervised student. The first two rows test the geometric and semantic distillation components; the final two changes concern token aggregation and the raw-feature bypass visible in Figure 2. The fifth-stage column helps expose accumulated execution differences beyond the similar first-stage success rates. e-ablatione-decouplerse-actione-distill
What it supports. The full model reaches 4.16 average length. Ground-truth targets yield 3.82, removing the Uni-Perceiver yields 3.72, and excluding original diffusion features yields 3.93. Together these comparisons support the particular target construction and fusion design; structured foresight alone does not account for the full result.
Where the evidence stops. The authors attribute the target gap to trajectory misalignment, but no matched-seed versus mismatched-seed teacher control is reported. Removing modules can also alter capacity and optimization, and the precise replacement for the removed Uni-Perceiver is not specified.
7. Analysis & limitations
7.1 What the evidence leaves open
Figures 3–4 show selected attention trajectories, PCA features and probe-based depth maps. They illustrate successful versus failed rollouts; they do not quantify depth accuracy or establish that attention coherence causes success. e-qualitative
Tables omit confidence intervals and seed variability; simulation evaluation counts are not given. Four physical tasks with 25 trials each provide narrow deployment evidence. The source has no dedicated limitations section. e-setupe-calvine-metaworlde-ablatione-reale-conclusion
Table 4 supports the chosen teacher pair in this setup, not universal inferiority of static geometry or other video representations. Teacher selection changes representation properties together; reported rankings do not isolate each explanation. e-targets
7.2 Questions for discussion
- Does same-trajectory pairing explain the advantage over ground-truth targets when teacher quality and data volume are controlled? (e-distill, e-ablation)
- How much does eight-action chunking delay recovery from an unexpected object displacement? (e-real, e-latency)
8. Reproducibility audit
8.1 Requirements and known gaps
Required assets include the embodied SVD initialization, reused CALVIN checkpoint, frozen DPAv3/DINOv2 teachers, monocular demonstrations and synchronized first-step/generated-video pairs. Stage schedules and hardware are supplied, but optimizer, learning rates, batch sizes, exact layer selection, hidden widths, query count, temporal horizon and sampling schedules remain unspecified. e-featurese-decouplerse-distille-training
Exact image-versus-feature interpolation placement needs clarification: Eq. (2) places Interpolate inside the VFM encoder, while the accompanying prose describes aligning output feature maps. Preserve this ambiguity rather than silently selecting an implementation. e-backbone
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate shared-trajectory supervision from teacher appearance
Reader-proposed check, not performed: freeze one CALVIN video backbone and hold demonstrations, decoupler capacity, query count and training budget fixed. Train students with (a) generated targets continuing the same noise trajectory as each first-step feature, (b) generated targets from an independent noise seed under the same observation/instruction, and (c) ground-truth future-frame targets. Use identical action-expert training and evaluation sequences across several seeds. Report average completed length and held-out feature error against a common set of same-trajectory teacher targets. The shared-trajectory explanation predicts a repeatable advantage for (a) over (b); indistinguishable control performance would weaken that explanation even if both beat (c). e-featurese-distille-traininge-ablation
Check 2: Measure recovery while an eight-action chunk is executing
Reader-proposed check, not performed: compare S-VAM and VPP on the same placement tasks with identical observation processing and command timing. Randomly displace the object at a controlled phase of chunk execution. Compare the reported eight-action setting with a shorter executed prefix, explicitly measuring the computation cost of more frequent replanning. Log camera timestamps, inference completion, command issue time, first corrective motion and task success across repeated trials. If action throughput adequately captures responsiveness, perturbation recovery should remain rapid; if stale chunks dominate, corrections will wait for a new prediction and shorter prefixes may improve recovery despite lower throughput. Publish latency distributions as well as success rates. e-reale-latencye-training
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page, all body pages, Figures 1–5, Tables 1–4, method equations, hardware and training schedules on page 8, attention footnote on page 10, real-world protocol on page 14, chunk-frequency statement on page 15, and teacher identity in reference 25 on page 17. Figure 2’s feature bypass, reference anchoring, spatial-to-temporal order and query-to-context attention were checked against Sections 3.2–3.3 and Eqs. (5)–(9). All six final original crops were inspected at their native saved resolution; labels, table headers and chart legends remain visible. Eq. (2)’s interpolation ambiguity is disclosed rather than resolved from the figure. Remaining reference pages were read as text. No separate supplement or external cited paper was inspected; code was not inspected and experiments were not run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, authors, affiliations and abstract (p. 1)
- 1 Introduction (pp. 1–3)
- 2 Related Works, including 2.1–2.3 (pp. 3–4)
- 3 Method, including 3.1–3.3 and Eqs. (1)–(9) (pp. 4–8)
- 4 Experiment, including 4.1–4.5 (pp. 8–15)
- 5 Conclusion (p. 15)
- References (pp. 16–19)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/revision: the inspected title page is arXiv:2603.16195v2 [cs.CV], dated 18 March 2026. Its title and all 13 authors match the catalog identity; the PDF capitalizes Yingjie CAI. The catalog submittedDate is 17 March 2026. No earlier revision was supplied, so changes from v1 cannot be established (e-identity).
- Text extraction does not reconstruct figure images; this limitation was addressed by inspecting original PDF pages and all six final crops.
- Separate supplemental material availability has not been fully verified. No separate supplement or appendix was supplied; the PDF ends with references.
- All five supplied text chunks were read individually, including the complete references. Visual inspection covered body pages 1–15 and reference page 17; other reference pages were read as text.
- Code, project website and external references were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block and arXiv margin
Exact title, 13 author credits and two affiliations; arXiv:2603.16195v2 [cs.CV], 18 March 2026.
Go to primary source ↓e-shortcutPDF pp. 2–3, Figure 1 and Introduction
First-step features trade fidelity for speed; dashed generated-video/VFM teacher supervision is training-only.
Go to primary source ↓e-backbonePDF pp. 4–5, Section 3.1, Figure 2 and Eqs. (1)–(2)
Stable Video Diffusion backbone; frozen VFM teacher extraction; Eq. (2) and interpolation wording differ in operation placement.
Go to primary source ↓e-featuresPDF p. 6, Section 3.2, Eqs. (3)–(4)
Upsampling-layer features from the first denoising step are spatially resized and channel-concatenated into F.
Go to primary source ↓e-decouplersPDF pp. 5–7, Figure 2; Section 3.2, Eqs. (5)–(6) and output-projection paragraph
Current-image reference anchoring; DPAv3/DINOv2 branches; spatial then temporal blocks and output projection.
Go to primary source ↓e-distillPDF p. 7, Section 3.2, Self-Distillation Optimization Objectives and Eq. (7)
Independently optimize squared feature errors against generated-video VFM targets; authors motivate shared diffusion trajectories.
Go to primary source ↓e-actionPDF pp. 7–8, Section 3.3, Eqs. (8)–(9); Figure 2 on p. 5
Concatenate both foresights and raw F; learnable-query condensation; instruction-conditioned DiT predicts action noise.
Go to primary source ↓e-trainingPDF p. 8, Section 4.1, Implementation Details
Three training stages, frozen modules, exact step counts, H100 counts, RTX 3090 24 GB inference and separate DPT probe.
Go to primary source ↓e-setupPDF p. 8, Section 4.1, Simulated Benchmarks
CALVIN ABC→D; MetaWorld 50 tasks with 50 demonstrations each; primary third-view monocular RGB.
Go to primary source ↓e-qualitativePDF p. 9, Figure 3 and caption; p. 10, footnote 3; p. 11, Figure 4 and caption
Depth uses a trained probe; one-step/semantic maps use sequence-global PCA. Attention tracks maxima and a 3-pixel neighborhood in head-averaged final cross-attention.
Go to primary source ↓e-calvinPDF p. 10, Table 1, S-VAM, VPP, HiF-VLA and Spatial Forcing rows
Average lengths 4.16, 3.58, 4.08 and 3.94. S-VAM stage rates 95.8/90.7/83.7/77.0/68.9; HiF-VLA stage 5 is 69.4.
Go to primary source ↓e-metaworldPDF p. 11, Table 2, difficulty headers and S-VAM/VPP rows
Task counts 28/11/11. S-VAM 0.793/0.607/0.684, overall 0.728; VPP 0.818/0.493/0.526, overall 0.682.
Go to primary source ↓e-ablationPDF p. 12, Table 3 and Section 4.3
Full 4.16; removals yield 4.01, 3.99, 3.82, 3.72 and 3.93. The no-self-distillation variant uses GT-future VFM targets.
Go to primary source ↓e-targetsPDF p. 13, Table 4 and Section 4.4
DINOv2 + DPAv3 4.16; DINOv2 + VGGT 4.04; SigLIP + DPAv3 4.06; author explanations contrast dense/global and dynamic/static targets.
Go to primary source ↓e-realPDF p. 14, Figure 5 task panels and Section 4.5, Task setups/Results Analysis
Cobot setup, four tasks, roughly 50 demonstrations/task and 25 trials/task. S-VAM success 56/32/44/24%; VPP 40/16/20/12%.
Go to primary source ↓e-latencyPDF p. 14, Figure 5 latency panel and final paragraph; p. 15, opening sentence
307.6 ms split 231.0/40.1/36.5 versus VPP 265.7 ms split 231.0/34.7; 15.8% overhead; eight actions and stated effective 25 Hz.
Go to primary source ↓e-conclusionPDF p. 15, Section 5 Conclusion
The paper concludes with claimed fidelity/efficiency benefits; no dedicated limitations discussion precedes the references.
Go to primary source ↓e-vfm-namePDF p. 6, geometric-teacher paragraph; p. 17, Reference 25
The body calls the teacher DPAv3 and cites Reference 25, titled Depth Anything 3: Recovering the visual space from any views.
Go to primary source ↓8.5 Primary sources
S-VAM: Shortcut Video-Action Model by Self-Distilling Geometric and Semantic Foresight ↗
PDF · 6,841 extracted words
Source fingerprint
9c12534fac24ed78b28616e1159175f400f91290a6a9c26443970d50dff3e63a