This&That: Language-Gesture Controlled Video Generation for Robot Planning
1. Paper overview
In one sentence: Language and pointing disambiguate a generated video plan, while a separate controller learns to follow that plan; the demonstrated execution benefit is confined to simulation. e02e04e06e09e10e11e17
| At a glance | What to know |
|---|---|
| Research problem | Source description Language can identify an operation yet ambiguously specify its objects or destination. The paper pairs deictic commands such as “put this there” with image-plane points, then addresses the separate challenge of converting a visual plan into executable actions. e02 |
| Core mechanism | |
| A key reported result | Simulated pick-and-place execution: In-distribution: 95 / 93; OOD: 87 / 80. Pick / place success (%). Isaac Gym four-block tasks; regular and identical-object OOD scenes. DiVA training uses 900 demonstrations, with 100 held out; the OOD set size is not separately stated. Language-only video + DiVA: 93 / 82 and 60 / 26. Language–gesture ACT: 57 / 35 and 56 / 35. Gesture improves OOD place success by 54 percentage points over the language-only video pipeline. These are simulator outcomes under the paper’s scripted success rules. e11e12e16 |
| Reading caution | Source description The authors report changing object appearance, 2D-to-3D gesture ambiguity and a restriction to short modular tasks. No physical robot was tested; the predicted transfer benefit is an author expectation. e17 |
Core contributions
Figure 12. Gesture conditioning joins dense visual context before entering a separately trained branch. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the color legend: green marks stage-1 UNet finetuning, purple marks the stage-2 gesture branch, and blue denotes frozen components. At the lower left, noisy frames and first-frame latents feed the UNet. Their concatenated representation also joins VAE-encoded gesture images at the upper concatenation node, before zero convolution. Follow the black branch-to-decoder arrow for the added conditioning features. On the right, text and first-frame CLIP embeddings are concatenated and normalized; dashed red paths mark cross-attention and FiLM modulation of both networks. The branch provides spatial guidance while the normalized semantic embeddings help interpret what operation the points describe. e03e04
What it supports. The sparse point signal is accompanied by scene and noise context, so the branch need not infer the intended operation from isolated marks. The source trains this branch after adapting the video prior and freezing the UNet. This is staged video generation, not an action-output architecture.
Where the evidence stops. The schematic omits the latent mask specified in Appendix C.3; its equations mask initial-frame latents only on the gesture branch. Main-text first-two-frame placement differs from the appendix’s random ordered placement. Neither an exact inference placement nor a numeric mask probability is established.
2. Motivation
2.1 The problem and the proposed response
Language can identify an operation yet ambiguously specify its objects or destination. The paper pairs deictic commands such as “put this there” with image-plane points, then addresses the separate challenge of converting a visual plan into executable actions. e02
2.2 What this reading follows
Imagine telling a robot to move one of several identical blocks. A verbal description must distinguish the object and destination, while a pair of image points can locate them directly. This&That combines those points with a short instruction to generate a video of the intended task. A second model, DiVA, consults that video alongside live observations and predicts executable action chunks. The useful question is how these two stages cooperate: a plausible video alone does not establish control. This reading follows the sparse-conditioning design, the video-to-action interface, and the separate evidence for video quality, human intent alignment and simulated execution. e02e04e06e09e10e11e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded taxonomy is entirely unassigned, so no existing quadrant can be confirmed. Architecture evidence supports a separate video predictor and video-conditioned action model, with explicit inference-time future imagery. This is neither joint future/action prediction in one model nor the frame-pair inverse-dynamics alternative the authors abandoned. e02e06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Use language to interpret what a point means
A point is a location cue, not a complete task specification. In the paper’s folding example, the same image-plane gesture can be compatible with different three-dimensional interpretations; adding a folding command resolves the illustrated ambiguity. This explains the two conditioning routes. CLIP embeddings supply semantic context through cross-attention and FiLM, while gesture images enter a spatial branch together with first-frame and noise latents. Stage 1 learns robotics video generation; stage 2 learns sparse gesture control against a frozen UNet. Table I supports complementary information: regular-language alignment is 91.7% with gestures versus 51.4% without them, while gestures without text reach 68.1%. These are human judgments on generated clips. They show the value of multimodal communication in the tested cases, without establishing a universal resolution of 3D ambiguity. e03e04e10e17
Figure 4. A video plan serves as reference memory for a controller with live visual and pose inputs. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two left-hand streams separately. Each planned frame and the live observation becomes an 8×12×512 ResNet feature map, then 16 tokens of width 512 through TokenLearner. The top stream supplies 16N goal tokens. The bottom stream supplies 16 observation tokens, accompanied by an end-effector pose token. The encoder’s cross-attention references goal tokens while processing current-state information. The decoder starts from fixed positional embeddings and attends to the encoder output to predict an action chunk. The four and seven labels count encoder and decoder layers. Appendix D specifies k=10 actions and execution of the entire chunk before another policy query. e06e07e12
What it supports. The plan conveys a sequence of desired visual states, while the live stream tells the controller where execution currently stands. This separation supports feedback without requiring a fixed one-to-one correspondence between generated frames and physical action timesteps. Training uses demonstration goals; testing substitutes generated goals.
Where the evidence stops. The diagram does not imply action diffusion or joint training with the VDM. DiVA is trained with an L1 action loss. Feedback arrives between complete chunks, and the paper does not specify an online schedule for regenerating the video plan.
5.2 Follow a visual sequence without treating it as a clock
The video generator emits a fixed frame count, but demonstration action sequences vary in length. Appendix D reports that a frame-based inverse-dynamics attempt did not produce reasonable actions under this mismatch. DiVA instead conditions each action chunk on the current observation, pose and a sequence of goal images. ResNet and TokenLearner turn those images into a compact memory; the transformer can reference several goals while predicting the next chunk. During training, the goals come from the demonstration itself, and random selection within consecutive temporal groups changes their timing. At deployment, generated video frames occupy the same interface. My interpretation is that this design relaxes exact temporal correspondence without eliminating the need for accurate visual guidance. Figure 7 supports timing robustness, while the paper does not separately isolate the effect of each attention or compression component. e06e07e14e16
5.3 Keep video quality, intent and execution as separate tests
The three evaluations answer different questions. Bridge metrics compare generated video appearance and temporal content with reference videos; the human study asks whether generated actions match a stated intention. Simulator rollouts then test whether DiVA can execute video plans. The strongest execution contrast is in identical-object OOD scenes, where gesture-conditioned video planning reaches 80% place success versus 26% for the language-only video pipeline. That is a 54-percentage-point difference under the reported setup. It does not establish precise physical placement: the simulator allows wider planar tolerance for near tasks than stacking, and no physical robot experiment is reported. My reading is that the combined evidence supports the value of an explicit video-plan interface and disambiguating gestures for these tasks, while broader robustness and real-robot transfer remain open. e09e10e11e12e17
5.4 Training and inference
During training
Bridge V1/V2 provide 25,767 stage-1 and 14,735 gesture-training videos after filtering. Bridge uses 14-frame SVD; simulation uses 25-frame SVD-XT initialization. The main text’s “trained from scratch” wording must therefore not be read as random initialization for the custom VDMs. e08e11e15
DiVA uses 900 training and 100 held-out demonstrations, an L1 action loss, and ground-truth goal frames only. Sampling one frame from each of N consecutive temporal groups adds timing noise. Generated goals replace demonstration goals at inference. e06e07e16
During inference
Generate a task video, then repeatedly condition DiVA on that plan, the current observation and pose. Execute the entire k=10 action chunk before querying again. This provides feedback between chunks; no online video-replanning schedule is specified. e06e12
Deictic evaluation reuses weights trained on regular text. The appendix randomly places gesture-bearing frames in temporal order, whereas the main text describes the first two frames; the exact inference placement remains unresolved. e04e10
5.5 Implementation flow
- Adapt the video prior
Stage 1 fine-tunes SVD’s UNet on robotics videos using a noise-reconstruction objective. CLIP text and initial-image embeddings are concatenated and normalized; cross-attention and FiLM condition intermediate features. The VAE and CLIP are frozen in Figure 12. e03
- Make sparse gestures usable
Stage 2 freezes the UNet and trains an encoder-derived ControlNet branch. Gesture images, initial-frame latents and noisy video latents are concatenated before a zero-initialized convolution; branch features enter the UNet decoder through skip connections. Gesture marks are 10-by-10 pixels with Gaussian dilation. e04
- Label demonstrations
Gripper open/close events, a YOLOv8 detector trained on 450 manually labeled images, and TrackAnything provide interaction and landing coordinates. Failed tracking is filtered, as are sequences shorter than T or longer than 5T, where T is the target frame count; numeric tracking thresholds are unstated. e05
- Reference the plan
DiVA fine-tunes an ImageNet-pretrained ResNet-18 and compresses each image to 16 tokens using TokenLearner. A pose token joins live-image tokens; a four-layer encoder cross-attends to video tokens, and a seven-layer decoder maps fixed positional queries to actions. e06
6. Experiments & results
This&That turns an initial image, language and pointing coordinates into a generated video plan, then uses a separate DiVA behavior-cloning controller to follow it with live feedback. Gestures improve intent disambiguation in Bridge video generation and simulated block manipulation; real-robot execution remains untested.
6.1 Read the original evidence
Table II. Video metrics favor the robotics-adapted generator under the paper’s comparison protocol. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the arrows in the column headers: FID, FVD and LPIPS favor smaller values; PSNR and SSIM favor larger values. The bottom row is This&That. These measurements use 646 gesture-filtered Bridge V1 test videos, with outputs standardized to 256×384 and 14 frames. Appendix E.2 explains that AVDC natively produces seven 48×64 frames, repeated to reach fourteen, whereas the proposed model directly produces the evaluation format. StreamingT2V is truncated to its first fourteen frames. FID uses 9,000 sampled images. These protocol details matter when interpreting the large differences between the rows, since training exposure, conditioning modalities and original output formats also differ. e08e09e18
What it supports. This&That reports FVD 84.58 versus 657.49 for SVD and 1512.25 for AVDC, alongside favorable values in the other four columns. The table supports better generated-video quality in this evaluation. Human intent alignment and successful robot execution require the separate experiments reported elsewhere.
Where the evidence stops. AVDC was trained on the test videos. Its PSNR/SSIM/LPIPS here are 20.23/0.663/0.507; Table V instead gives 19.43/0.649/0.517 at the enlarged resolution, without explaining the discrepancy. Keep each table’s values attached to its own location.
Table III. Pointing improves execution most clearly in scenes containing identical objects. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Every slash separates in-distribution from out-of-distribution success percentages; it is not a numerator and denominator. The OOD scenes deliberately contain objects with identical shape and color. Read the first three rows as direct ACT policies and the final three as video plans converted to actions by DiVA. The most focused comparison is between the bottom two rows, which vary gesture conditioning in the custom video generator. The Pick column measures grasping and lifting, whereas Place also requires reaching the task destination. Appendix E.7 uses different placement tolerances for near and stack tasks, so success should be interpreted within those scripted rules. e11e12e16e17
What it supports. The full pipeline reaches 93% regular and 80% OOD place success. The language-only video pipeline reaches 82% and 26%, respectively: an 11-point regular improvement and a 54-point OOD improvement. Direct ACT with language and gesture reaches 35% place success in both settings.
Where the evidence stops. These are Isaac Gym rollouts, with no real-robot execution test. Near tasks permit three block diameters of planar error and stacking permits one, both with a one-radius vertical tolerance. Table III reports no confidence intervals or separate OOD sample count.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Simulated pick-and-place execution Isaac Gym four-block tasks; regular and identical-object OOD scenes. DiVA training uses 900 demonstrations, with 100 held out; the OOD set size is not separately stated. | In-distribution: 95 / 93; OOD: 87 / 80. Pick / place success (%) | Language-only video + DiVA: 93 / 82 and 60 / 26. Language–gesture ACT: 57 / 35 and 56 / 35. Gesture improves OOD place success by 54 percentage points over the language-only video pipeline. These are simulator outcomes under the paper’s scripted success rules. e11e12e16 |
| Bridge video prediction quality 646 gesture-filtered Bridge V1 test videos; evaluation standardized to 256×384 and 14 frames. | 17.28; 84.58; 21.71; 0.787; 0.112 (Table II). FID ↓; FVD ↓; PSNR ↑; SSIM ↑; LPIPS ↓ | AVDC: 163.93; 1512.25; 20.23; 0.663; 0.507. SVD FVD: 657.49. The reported scores favor This&That, but baselines differ in training, conditioning and native resolution. AVDC saw the test videos during training. These metrics do not measure executed robot success. e09 |
| Human-rated intent alignment Three raters; 24 cases covering pick/place, stacking, folding and opening/closing; independently generated videos. | 91.7 with regular text; 87.5 with deictic text. Reported average yes-response rate (%) | Language-only custom VDM: 51.4 / 25.0; vision plus gesture: 68.1 without text. Complementary modalities help in this small human study. No confidence intervals or inter-rater agreement are reported. e10 |
| Gesture-branch design ablation Appendix Bridge VDM ablation, Table IV. | 84.580 for the full method. FVD ↓ | Regular ControlNet: 124.710; SAM mask: 88.757; no CLIP LayerNorm: 92.527. The standard-ControlNet comparison jointly changes encoding and concatenation; it does not isolate either component’s causal contribution. e13 |
| DiVA temporal-noise ablation Figure 7(b), regular and OOD simulated rollouts. | With noise: 93 regular / 80 OOD. Overall success (%) | Without noise: 85 regular / 73 OOD. Temporal jitter improves this reported comparison. Figure 7(a) supports multiple goals, but its OOD curves are nonmonotonic; 25 frames are not uniformly optimal despite the caption. e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table IV. Encoding and contextualizing sparse marks outperform the tested alternatives. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the full method in the bottom row as the reference for every change. The first row replaces pretrained VAE gesture encoding with a simpler conditioning path and also removes the proposed concatenation with dense inputs. The next row uses SAM segmentation masks to supply a larger region around the selected object. Another removes LayerNorm from the concatenated CLIP embeddings. The remaining rows enlarge or shrink the gesture-conditioning area. All columns use the same higher/lower directions as the main quality table. Comparing rows therefore tests a set of concrete architectural and representation choices rather than merely changing the wording of the task. e13
What it supports. The full model has FVD 84.580, compared with 124.710 for regular ControlNet, 88.757 for SAM masks and 92.527 without LayerNorm. Larger and smaller marks also score worse in this table. The results favor the chosen sparse-mark representation plus contextual conditioning among these tested configurations.
Where the evidence stops. The regular-ControlNet row changes both encoding and concatenation, so it cannot assign the gain to either alone. The authors attribute SAM’s weakness to incorrect regions, but provide no controlled segmentation-error breakdown. Reported decimal precision is not an uncertainty estimate.
Figure 7. Multiple goals and timing jitter help DiVA, with nonmonotonic gains as frame count grows. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) varies N, the number of goal frames. Blue curves represent regular scenes, red curves OOD scenes; lighter shades show pick success and darker shades overall success. Notice the broken vertical axis before comparing slopes. One final goal image performs poorly, while multiple images supply intermediate task information. Panel (b) holds the goal-frame approach and compares temporal-noise training with its absence. Within each scene type, darker bars include noise. Its vertical axis starts at 50%, so compare the printed percentages rather than apparent bar-height ratios. The training perturbation samples one image from each consecutive temporal group of a demonstration. e07e14
What it supports. Temporal noise raises overall success from 85% to 93% on regular scenes and from 73% to 80% on OOD scenes. Pick success similarly rises from 88% to 95% and from 79% to 87%. These observations support robustness to temporal misalignment within the reported simulation setup.
Where the evidence stops. The caption calls 25 frames best overall, but panel (a) is not uniformly monotonic: OOD curves peak earlier, and regular pick also peaks before 25. The plots provide no error bars, so small differences and an optimal frame count remain uncertain.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report changing object appearance, 2D-to-3D gesture ambiguity and a restriction to short modular tasks. No physical robot was tested; the predicted transfer benefit is an author expectation. e17
Table V reports AVDC PSNR/SSIM/LPIPS of 19.43/0.649/0.517 at 256×384, differing from Table II’s 20.23/0.663/0.507 without an explained reconciliation. At 48×64, Table V still favors This&That: FVD 129.74 versus 620.46. e09e18
Near-task placement allows three block diameters of planar error versus one for stacking, both with a one-radius vertical tolerance. With 5-cm blocks these are permissive spatial criteria, so the aggregate success rate should not be read as precision placement. e12
Successful use of generated goals supports transfer within this setup; it does not establish negligible domain gap generally. Neither TokenLearner nor cross-attention receives a separate isolated ablation in the reported DiVA study. e07e14e16
7.2 Questions for discussion
- Would the gesture advantage persist under tighter placement tolerances and noisy user clicks?
- How much of the ControlNet ablation gain comes from VAE encoding versus dense-context concatenation?
8. Reproducibility audit
8.1 Requirements and known gaps
VDM training uses 8 L40S GPUs (48 GB each) for Bridge stage 1, 99K iterations, then 4 GPUs for 30K. Simulation uses 30K/15K iterations on 8/4 GPUs. Batch size is one per GPU; stage learning rates are 1e−5/5e−6 with AdamW and 8-bit optimizer states, without EMA. e15
DiVA uses one RTX 6000 Ada, 2,000 epochs, batch 8, learning rate 1e−5 and weight decay 1e−4; checkpoints every 500 epochs are selected by validation error. Preserve action coordinates, full-chunk execution and scripted termination when reproducing success rates. e16e12
A reproduction needs the filtered videos, gesture-label pipeline, SVD checkpoints and scripted simulator demonstrations. The PDF promises detector code and weights, without proving release. Mask probability, tracking rejection thresholds, exact split membership and the validation-selection protocol remain insufficiently specified. e04e05e08e16
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate gesture encoding from dense-context concatenation
Reader-proposed check, not performed: reproduce Table IV’s gesture-branch comparison as a factorial experiment, independently toggling pretrained VAE gesture encoding and concatenation of initial-frame/noise context. Keep the same frozen stage-1 UNet, filtered training examples, gesture marks, optimization budget and evaluation prompts. Control input adapters and report parameter-count differences. Across repeated training seeds, measure FVD and blinded intent alignment on identical-object cases. If only the combined configuration improves, the mechanism is an interaction; if one toggle accounts for the gain, the original bundled ablation overattributes the benefit. Record the mask probability and gesture-frame placement explicitly because the PDF does not settle them. e04e08e10e13e15
Check 2: Test whether timing noise protects execution with generated goals
Reader-proposed check, not performed: train paired DiVA policies with and without temporal noise on the same 900 demonstrations and fixed N=25, using the same validation selection. Evaluate both on ground-truth goals and fixed generated plans, then add controlled monotonic timing distortions to the goal sequences while preserving their content and order. Use identical regular/OOD scenes and rollout seeds. Report pick success, near success and stack success separately under the published rules, alongside stricter planar-error thresholds. The timing-robustness hypothesis predicts a smaller success drop for noise-trained policies as distortion increases. A benefit only with pristine goals would weaken that interpretation; a widening generated-goal deficit would challenge the generality of the claimed small domain gap. e06e07e12e14e16
8.3 Reading coverage
Visual audit: Visually inspected the title and v2 date, Figures 1–13 and Tables I–V on their original PDF pages, plus the method, equations, training, hardware and evaluation passages needed by this report and proposed checks. Six final crops were individually viewed: Figures 12, 4 and 7, and Tables II, III and IV. Figure 12’s branch arrows and frozen/trained legend were checked against Sections IV and Appendix C; Figure 4 was checked against Appendix D. Figure 7’s broken axis and caption qualification, Table II/V discrepancies and gesture-frame-placement difference are retained. All seven text chunks were read, including the reference-only page 8, which was not visually rendered. Separate linked videos, code and external supplements remain outside this pass.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; I. Introduction; II. Related Work; III. Overview
- IV. Language-Gesture Conditioned Video Diffusion Models, A–B
- V. Video-Conditioned Behavioral Cloning
- VI. Experiments, A–B; VII. Limitations; VIII. Conclusion
- References
- Appendix A. Document Overview; B. Additional Experiments and Ablation Studies, 1–5
- Appendix C. Video Diffusion Model Implementation Details, 1–4
- Appendix D. DiVA Model Implementation Details and inverse-dynamics alternative
- Appendix E. Experiment Details, 1–8; F. Limitation: Changes in Object Appearance Over Time
Outside the original text pass
- The inspected title page identifies arXiv:2407.05530v2 [cs.RO], 18 May 2025. Its title and seven authors match the catalog. The catalog submission date is 8 July 2024; only the supplied v2 artifact was reviewed, and changes relative to v1 were not verified.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All seven supplied text chunks were read individually. The retained PDF was visually inspected on pages 1–7 and 9–18, covering all figures and tables; reference-only page 8 was read as text.
- The text-extraction limitation was addressed by inspecting original PDF pages and six final crops. Linked code, websites, external videos and separate supplements were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin
Exact title and seven catalog authors; University of Michigan and University of Washington; arXiv:2407.05530v2, 18 May 2025.
Go to primary source ↓e02PDF pp. 1–2, I. Introduction and III. Overview
Language–gesture video generation and a separate video-based robot execution component address communication, planning and control.
Go to primary source ↓e03PDF p. 3, IV-A; p. 12, Appendix C.2; p. 15, Figure 12 and caption
SVD robotics finetuning, noise reconstruction, CLIP concatenation/normalization, cross-attention and FiLM; Figure 12 distinguishes trained and frozen modules.
Go to primary source ↓e04PDF pp. 3–4, IV-B; pp. 12–13, Appendix C.3 and unnumbered yGest/yUNet equations; p. 15, Figure 12
Frozen UNet and trained gesture branch; encoded inputs, zero convolution and asymmetric latent masking. Main text says first two gesture frames; appendix says random ordered positions. Mask probability lacks a numeric value.
Go to primary source ↓e05PDF pp. 13–15, Appendix C.4, Automatic Gesture Labeling on Real Data
YOLOv8 uses 450 annotated images; gripper events and TrackAnything supply coordinates. Tracking rejection threshold is qualitative; duration limits are T and 5T. Detector release is promised.
Go to primary source ↓e06PDF p. 4, V; p. 5, Figure 4; pp. 15–16, Appendix D
DiVA conditions action chunks on live observations, pose and video goals using ResNet-18, 16-token compression, four encoder and seven decoder layers. L1 training and k=10 full-chunk execution; unsuccessful inverse-dynamics alternative discussed.
Go to primary source ↓e07PDF p. 4, V, training paragraphs; p. 10, Appendix B.5
DiVA trains on ground-truth goals and uses generated goals at inference; one sampled frame per consecutive temporal group introduces timing noise.
Go to primary source ↓e08PDF p. 5, VI-A, Bridge Dataset; p. 16, Appendix E.1–2
25,767 initial finetuning videos and 14,735 gesture-training videos; front view and duration/annotation filtering; test set is 646 Bridge V1 videos after gesture filtering.
Go to primary source ↓e09PDF p. 5, Table II, all rows and VI-A; p. 16, Appendix E.2
Table II gives This&That 17.28/84.58/21.71/0.787/0.112 and AVDC 163.93/1512.25/20.23/0.663/0.507. Evaluation resizes outputs to 256×384 and 14 frames; AVDC repeats seven native frames and saw the test data.
Go to primary source ↓e10PDF p. 4, Table I, Average columns; p. 5, User Alignment Study; p. 16, Appendix E.3; p. 17, E.4
Three raters assess 24 cases: 8 pick/place, 5 stacking, 6 folding, 5 open/close. Full model averages 91.7/87.5; language-only 51.4/25.0; gesture-only 68.1. Deictic prompts reuse regular-text-trained weights.
Go to primary source ↓e11PDF p. 6, Table III and VI-B; p. 18, Appendix E.8
Pick/place columns contain regular/OOD rates. Full model: 95/87 and 93/80; language video: 93/60 and 82/26; gesture ACT: 57/56 and 35/35. Video baselines use DiVA. Main text also states SVD initialization despite “from scratch” wording.
Go to primary source ↓e12PDF p. 17, Appendix E.7, Simulation Experiment Details
7D delta-pose/gripper actions; 5-cm blocks; pick exceeds one diameter for more than five timesteps; near/stack planar limits are three/one diameters, vertical limit one radius. Rollouts last up to 250 steps, stopping five steps after first overall success.
Go to primary source ↓e13PDF p. 9, Appendix B.2(a–d); p. 10, Table IV, all rows
FVD: regular ControlNet 124.710, SAM 88.757, no LayerNorm 92.527, larger marks 96.794, smaller marks 106.953, full model 84.580. ControlNet ablation changes both encoding and concatenation.
Go to primary source ↓e14PDF p. 10, Figure 7(a–b), legends/axes and Appendix B.5
Temporal-noise overall success changes 85→93 regular and 73→80 OOD; pick changes 88→95 and 79→87. Goal-frame curves show poor single-goal success and nonmonotonic OOD performance, qualifying the caption’s 25-frame preference.
Go to primary source ↓e15PDF p. 16, Appendix E.1, VDM Training Details
L40S hardware, stage-specific iteration counts/GPU counts and learning rates, batch one per GPU, AdamW with 8-bit strategy and no EMA; 14-frame SVD for Bridge and 25-frame SVD-XT for simulation.
Go to primary source ↓e16PDF p. 17, Appendix E.6, DiVA Training Details
900/100 train/held-out instances, roughly 75–100 pairs per demonstration, RTX 6000 Ada, 2,000 epochs, batch 8, learning rate 1e−5, weight decay 1e−4, checkpoint selection and ground-truth-only goals.
Go to primary source ↓e17PDF p. 7, Figure 6 and VII. Limitations; p. 17, Figure 13; p. 18, Appendix F
Image-plane ambiguity and appearance drift are acknowledged; tasks are short and robot execution is simulation-only. Real-robot transfer and tolerance of appearance artifacts remain author expectations.
Go to primary source ↓e18PDF p. 16, Table V, AVDC enlarged row and both 48×64 rows
AVDC enlarged PSNR/SSIM/LPIPS are 19.43/0.649/0.517. At 48×64, full model FVD is 129.74 versus AVDC 620.46; the reported method leads all five metrics.
Go to primary source ↓8.5 Primary sources
This&That: Language-Gesture Controlled Video Generation for Robot Planning ↗
PDF · 12,075 extracted words
Source fingerprint
214f36b0d13471c27154dabeab403d5c5aa84a9852f28a58d238da7b4c9bf9ec