Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio
1. Paper overview
In one sentence: Schedule stronger video conditioning when action attention shifts toward imagined futures, while retaining current-observation grounding for precise manipulation. E02E03E05E06E12E18E19
| At a glance | What to know |
|---|---|
| Research problem | Source description A video model may imagine the correct object–destination pairing while its action head executes a memorized training trajectory. Conversely, an action head can complete a familiar task despite an incorrect video prediction. The problem is transferring compositional video priors into executed control, rather than judging generated videos alone. E02 |
| Core mechanism | |
| A key reported result | LIBERO compositional OOD success: 59.4 guided; 55.7 unguided Average task success rate (%). Three ten-task OOD suites; headline evaluation specifies 50 rollouts per task and three seeds. Joint LoRA, video noise 0.883, five latent frames. Reported baselines: pi_0.5 26.8; Cosmos-Policy 10.5; Fast-WAM 9.4; DiT4DiT 6.4. Guidance adds 3.7 percentage points, calculated from displayed averages. Average ID changes from 94.3 to 94.0; LIBERO-long falls from 84.3 to 82.6. Baseline ID values partly come from original papers. Mimic-Video OOD is unreported. E08E09 |
| Reading caution |
Core contributions
- Author claim
Temporal Ratio diagnoses future-versus-current attention; the authors associate its rises with planning and its falls with precise manipulation. E05
- Source description
TR-adaptive velocity guidance combines null-language and extended-plan counterfactual branches at inference, without an additional training objective. E06
Figure 2. Partially denoised video features connect imagination to a separate action predictor. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the solid blue path from Cosmos through Feature Projection to the yellow video prefix of Gemma. The instruction enters the video side, while the orange proprioceptive token and purple noisy action tokens enter the action side. The separate arrow marked by the video noise level explains why the projection carries information about feature reliability. The image strip depicts an observed first frame followed by uncertain future content. Appendix A clarifies that the observation latent is kept clean at each video solver call and that features are extracted from block 20. The two flow-matching blocks learn distinct video and action velocities. E02E03E07E18
What it supports. The interface makes future video information available without requiring complete pixel reconstruction before action sampling. The action head can nevertheless favor the current-frame features. Providing a predictive representation therefore does not, by itself, establish that the generated actions follow the imagined plan; that is the motivation for the attention diagnostic.
Where the evidence stops. The diagram does not specify the attention mask. Section 2.1 calls action attention causal, whereas Appendix A.1 describes bidirectional remaining action tokens after causal boundaries. The report retains the appendix detail and flags the inconsistency.
2. Motivation
2.1 The problem and the proposed response
A video model may imagine the correct object–destination pairing while its action head executes a memorized training trajectory. Conversely, an action head can complete a familiar task despite an incorrect video prediction. The problem is transferring compositional video priors into executed control, rather than judging generated videos alone. E02
2.2 What this reading follows
A plausible imagined video is only the first half of a robot policy. The action head must also use its future content to select the right object and destination. This paper studies that interface with a Cosmos video backbone and a separate Gemma action head. Temporal Ratio compares attention to predicted future frames with attention to the clean current frame. The authors use changes in this ratio to schedule language and longer-plan guidance during action denoising. Read the architecture first, then the attention traces and paired scheduling ablations. The results support this intervention within the tested model family, while contradictory implementation details and limited trial reporting constrain reproduction. E02E03E05E06E12E18E19
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Not applicable |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Conflicts with the recorded classification
The WAM/generalization topic is supported, but the paper explicitly studies a video backbone feeding a separate latent inverse-dynamics action head and introduces an inference-time control intervention. Thus “Not applicable” architecture and “Other mechanisms / Outside quadrants” underdescribe the implemented method. Joint loss optimization does not make these two transformers a single unified video-action model. This assessment preserves the recorded catalog snapshot. E02E03E04E06
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Trace the two prediction problems before interpreting attention
The model first predicts video dynamics in a latent space and then predicts an action chunk conditioned on that space. During training, both predictions use flow matching: interpolate clean targets with noise and learn the corresponding velocity. Joint training allows the action loss to update the adapted video backbone as well as the action head; two-stage training freezes the backbone before fitting actions. At inference, the clean current observation anchors the video sequence while future latents are partially denoised. The policy then uses cached intermediate features across its action Euler steps. Language reaches actions through those features, not a separate language input to Gemma. Figure 2 is therefore an information-flow diagram for two coupled predictors. The clean-anchor versus predicted-future distinction is the specific architectural asymmetry that makes the Temporal Ratio comparison meaningful. E03E04E07
Figure 5. Future attention varies during execution and across video-feature settings. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the numbered snapshots as an illustrative progression from initialization through object approach, grasp, receptacle approach and placement. The same numbers beneath the Joint-LoRA plot connect task phases to replanning steps. Each panel uses Temporal Ratio on the vertical axis and replanning step on the horizontal axis; note that vertical scales differ. Blue, orange and purple identify the noise levels shown in the shared legend. The original caption assigns upward triangles to successful episodes and downward triangles to failures. Compare trajectories within a panel before comparing absolute levels across training regimes, because TR sums future attention rather than measuring task success. E05E13
What it supports. The authors interpret rises around object or receptacle selection as greater use of imagined futures, and lower values during local manipulation as stronger current-frame grounding. This suggests a time-varying intervention: amplify compositional conditioning when future attention rises, while allowing precise manipulation to remain reactive. Low TR is therefore not inherently a failure.
Where the evidence stops. The trajectories are associations and sometimes cross; they do not prove a causal effect of attention. With five latent frames, four contribute to the numerator and one to the denominator. No uncertainty bands or episode counts accompany this figure.
5.2 Turn relative future attention into a guidance schedule
Temporal Ratio sums action attention over all future tokens and divides by attention over current-frame tokens. The intervention uses the ratio's change within an episode: Algorithm 3 reads layer 15 on the initial action-noise pass of each replan, then compares it with the first-replan baseline. At or below that baseline, guidance is zero; above it, both guidance weights grow until the scale cap is reached. The language direction subtracts the null-text branch from the conditional action velocity. The plan direction subtracts the conditional velocity from a velocity conditioned on an extended rollout. That rollout is truncated before entering the action head, preserving its five-frame input horizon. These signs matter: the algorithm amplifies instruction sensitivity and shifts toward the longer-plan branch. Its scalar schedule is held for the replan rather than recalculated from each subsequent Euler step. E05E06E11
5.3 Separate evidence for a useful schedule from evidence for a causal explanation
The tables support a narrower conclusion than saying future attention always causes successful planning. Flat guidance improves average OOD success but substantially reduces ID success; adaptive guidance preserves most ID performance while further improving OOD. This is evidence for the tested scheduling rule, yet its timing and effective strength change together. The attention traces also associate task phases with TR without directly intervening on future content. Reader interpretation: the mechanism is plausible, but a budget-matched shuffled schedule and controlled future-feature perturbations would make it more discriminating. The real-world results add executed-robot evidence, although their small unequal task budgets and incomplete trial accounting limit precision. Finally, the authors warn that guidance can strengthen an incorrect video future. A larger TR measures where attention goes; it does not certify that the imagined trajectory is physically feasible. E05E06E10E12E17E19
5.4 Training and inference
During training
Video and action heads learn flow-matching velocities toward clean demonstrations. Joint training uses action loss plus video loss with weight 1; two-stage training first adapts video, then freezes it and trains the action head on cached features. LoRA freezes the remaining backbone parameters. E04
Simulation joint training is specified as 50k steps with rank-32 LoRA, learning rate 0.0001 and batch 128 across 16 GPUs. Two-stage training uses 25k video steps plus 50k action steps. Video noise is uniform; action noise follows the specified shifted Beta distribution. Real-world data contain 5,600 episodes across 24 equally sampled task categories. E04E14
During inference
The canonical extraction uses video noise 0.883 and five latent frames. FlowUniPC follows a six-step schedule truncated at extraction; cached features condition ten Euler action steps. Simulation replans every ten environment steps. On YAM, predict 50 actions at 30 Hz, execute the first 30, then observe and replan. E07E09
Keep the first-replan TR baseline for the entire episode. Guidance is zero at or below that baseline and increases above it, capped at scale 2. Base language and plan weights are 0.1 and 0.3. Algorithm 3 applies conditional, null-text and plan action-head calls when their weights are active. E06
5.5 Implementation flow
- Imagine with an observation anchor
Cosmos-Predict2.5-2B partially denoises future latents while the current-frame latent remains clean. Features from DiT block 20 of 28 are projected into a separate 18-layer, approximately 300M-parameter Gemma action head. Language enters through the video backbone. E03E07
- Condition action denoising
Projected video features, a proprioceptive token and noisy action tokens form the action-head input. The video noise level is explicitly embedded. Appendix A.1 describes a prefix mask with bidirectional video attention, causal state/first-action boundaries and bidirectional remaining action tokens; the main text simplifies this to causal action attention. E03E18
- Measure and intervene
Read TR at Gemma layer 15 on the initial action-noise pass of each replan. A null-text video branch supplies language guidance; a 24-latent-frame rollout, truncated to the first five frames, supplies plan guidance. Both alter action-flow velocities, rather than choosing actions by scoring decoded videos. E06E13
6. Experiments & results
Temporal Ratio measures how much a video-conditioned action head attends to imagined futures relative to the current observation. The paper uses this signal to schedule language and long-horizon guidance during action sampling. LIBERO and bimanual experiments support improved compositional control, with remaining dependence on video accuracy and unresolved implementation inconsistencies.
6.1 Read the original evidence
Table 1. High familiar-task scores coexist with much weaker compositional control. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the four in-distribution suite columns separately from the three out-of-distribution columns. Their averages summarize different task sets, so they are not a paired before-and-after measurement of identical episodes. The OOD suites change familiar object, spatial or goal bindings; the headline protocol specifies fifty rollouts per task and three seeds. The bottom two rows isolate adding guidance to the selected latent VAM, whereas comparisons with the preceding rows also change model design and training. Keep the dashes in the Mimic-Video row as missing evaluations. They are not zero success rates and do not justify ranking that method on OOD tasks. E08E09
What it supports. The proposed model reports 55.7% average OOD success without guidance and 59.4% with guidance, a calculated gain of 3.7 percentage points. Average ID moves from 94.3% to 94.0%, and Long falls from 84.3% to 82.6%. The improvement is substantial relative to the listed OOD baselines but does not eliminate generalization failures.
Where the evidence stops. Baseline ID values partly come from original papers; OOD uses released checkpoints. Mimic-Video OOD is excluded because checkpoints are suite-specific. No confidence intervals are supplied. Printed OOD averages do not exactly match means of the displayed suite scores; the report retains the source values.
Table 2. Guidance improves the reported physical-task aggregate under a small compositional test. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use Appendix Table 13 to decode the task columns. Tasks 1–2 send fruits to a bin or paper bag; tasks 3–6 combine snacks or a bottle with those two receptacles; tasks 7–8 place snacks in a bin or on a shelf. The three scenes present competing targets and receptacles. Tasks 1–2 receive fifteen trials each, while the remaining tasks receive five each. Read the final column with those unequal budgets in mind: weighting by trials reproduces the proposed method's displayed aggregate, whereas an unweighted average of its eight task percentages does not. The comparison concerns robot execution, not generated-video ratings. E10E14E19
What it supports. The reported aggregate increases from 71.7% to 83.3%, a difference of 11.6 percentage points using rounded table values. For example, Task 5, bottle into paper bag, rises from 40.0% to 80.0%, while several already strong columns remain unchanged. These observations support improved object–receptacle selection in this particular bimanual testbed.
Where the evidence stops. The success criterion requires picking the named object first and reaching the named receptacle. Only sixty trials per method are specified. Cosmos-Policy's percentages do not fit the stated integer trial budgets, and the authors report difficulty training it; uncertainty is unreported.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO compositional OOD success Three ten-task OOD suites; headline evaluation specifies 50 rollouts per task and three seeds. Joint LoRA, video noise 0.883, five latent frames. | 59.4 guided; 55.7 unguided Average task success rate (%) | Reported baselines: pi_0.5 26.8; Cosmos-Policy 10.5; Fast-WAM 9.4; DiT4DiT 6.4. Guidance adds 3.7 percentage points, calculated from displayed averages. Average ID changes from 94.3 to 94.0; LIBERO-long falls from 84.3 to 82.6. Baseline ID values partly come from original papers. Mimic-Video OOD is unreported. E08E09 |
| Flat versus TR-adaptive guidance on LIBERO Combined language/plan guidance with the same base weights and extended horizon; schedules differ. | Adaptive 94.0 / 59.4; flat 84.3 / 57.3 Average ID / OOD success rate (%) | Unguided 94.3 / 55.7. Scheduling recovers 9.7 ID points over flat guidance while adding 2.1 OOD points. Object-OOD remains 40.0 in all three rows. Tables report no uncertainty intervals. E12 |
| Bimanual YAM compositional task success Eight tasks in three scenes; 15 trials each for tasks 1–2 and five each for tasks 3–8, totaling 60 per method. Success requires picking the named object first and reaching the named receptacle. | 83.3 guided; 71.7 unguided Reported aggregate success rate (%) | pi_0.5 55.0; pi_0 36.7; Cosmos-Policy 12.5. The displayed gain is 11.6 percentage points. These are physical trials under a specific success definition, not video-quality scores. Trial-weighted aggregation matches the proposed method rows; Cosmos-Policy trial accounting is unresolved. E10E14E19 |
| Video extraction noise ablation Table 4, Joint + LoRA, rollout horizon five; design-grid protocol differs from the headline comparison. | At noise 0.883: 94.3 / 55.7 Average ID / OOD success rate (%) | At noise 1.000: 78.0 / 41.0; at noise 0.024: 82.5 / 45.3. More denoising is not uniformly better. The appendix says the action head always receives five frames even when video rollout horizon changes. E08E11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 5. The schedule determines whether stronger conditioning damages familiar-task control. Original paper, p. 19 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two guided rows first. Both use language weight 0.1, plan weight 0.3 and an extended video horizon of twenty-four latent frames, as specified in the original caption. Flat guidance applies the base weights throughout action denoising. Adaptive guidance multiplies them by the clipped rise in TR relative to the episode's initial value. Now compare each row with the unguided reference. Spatial, Object, Goal and Long expose where the average changes originate. Read this table alongside Table 6: preserving familiar control and improving compositional tasks are distinct requirements, and the ID table alone cannot establish the latter. E06E12
What it supports. Average ID success is 84.3% with flat guidance and 94.0% with adaptive guidance, versus 94.3% unguided. Adaptive scheduling recovers 9.7 percentage points over the flat setting. This is the clearest controlled evidence that continuously amplifying video conditioning can be costly, even when an adaptive version preserves most aggregate ID performance.
Where the evidence stops. The schedules share base weights but not the realized timing or total strength of intervention. Thus the comparison does not isolate phase alignment from effective guidance magnitude. It also does not demonstrate identical success to the unguided model on every suite.
Table 6. Adaptive guidance's OOD gain comes from Spatial and Goal, with no measured Object gain. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the row order from the ID ablation and inspect each OOD column before the average. Spatial measures altered object or spatial bindings; Object changes semantic identity bindings; Goal introduces unseen object–destination combinations. The guided rows reuse the combined language and extended-plan signals described in Table 5, with different weight schedules. Spatial moves from 50.3 to 53.6 to 58.6, while Goal moves from 77.3 to 78.3 to 80.3. Object stays at 40.0. This breakdown reveals which compositional tests respond to the intervention and prevents the average from implying a uniform improvement across all types of recombination. E08E12
What it supports. Flat guidance reaches 57.3% average OOD success and adaptive guidance 59.4%, compared with 55.7% unguided. The adaptive-versus-flat difference is 2.1 percentage points. Together with the ID table, this supports a better observed tradeoff for adaptive scheduling, while the unchanged Object column identifies a remaining empirical limit.
Where the evidence stops. The table provides neither uncertainty intervals nor separate language-only and plan-only results. It cannot determine which branch produces the gain or whether the 2.1-point adaptive advantage is robust to new seeds, tasks or hyperparameter selection.
7. Analysis & limitations
7.1 What the evidence leaves open
The study covers one latent-VAM family. Guidance can amplify physically infeasible video futures; extra generation and guidance increase inference cost. No measured latency table is supplied. E17E16
Attention association does not establish causal dependence on useful future content. With four future frames and one current frame, uniform per-token attention already yields TR=4; absolute TR is not a universal confidence score. E05E13
Reproducibility conflicts remain: simulation chunks are 16 actions in Section 5.2 but 17 in the appendix; simulation actions are described as end-effector deltas and as joint deltas. Full-finetuning learning rate is 0.000001 in Section 2.2 versus 0.00001 in Appendix D. Video-data filtering and grid-rollout totals also disagree. E18
Real-world optimization describes a 10,000-step decay, yet compute text/Table 14 specify 1M training steps. Appendix E.6 calls action-head cost unchanged although Algorithm 3 adds branch evaluations. Table 2 lacks uncertainty and includes Cosmos-Policy percentages incompatible with the stated integer trial counts; its training also reportedly failed to work well. E15E16E19
Some appendix summaries overstate their tables: Two-Stage + Full at noise 0.883 has 29.7 OOD in Table 4, not the 33.0 attributed in C.1; block 20 is not the highest high-noise OOD entry in Table 7. These conflicts are preserved rather than used as additional findings. E20
The printed unguided and guided OOD averages in Tables 1 and 6 do not exactly equal the arithmetic means of their three displayed suite scores. The report preserves those printed aggregates; the source does not explain the discrepancy. E09E12
7.2 Questions for discussion
- Does TR predict useful future dependence after controlling for noise level and execution phase?
- Would adaptive scheduling still outperform a guidance schedule matched for total intervention strength?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include Cosmos-Predict2.5-2B and its tokenizer/text encoder, the Gemma expert, exact LIBERO OOD task bindings, and the YAM dataset. The physical setup uses two I2RT YAM arms and three RealSense D405 RGB cameras. Table 14 specifies 32 H200 80GB GPUs, effective batch 512 and bfloat16; its step-count conflict needs resolution. E03E08E14E15
The v1 conclusion promises code and trained models after double-blind review. The supplied paper does not establish a currently usable release. A faithful implementation also needs clarification of the conflicting masks, action conventions and training budgets, plus handling of a near-zero TR baseline. E17E18E15E06
Proposed checks: compare adaptive guidance with temporally shuffled, intervention-budget-matched guidance; separately perturb future features while preserving the clean anchor and measure action changes. Both would distinguish phase-sensitive use of future content from attention correlation alone. E05E06E12
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test phase alignment with a matched guidance budget
Reader-proposed, not executed: use the same joint-LoRA checkpoint, noise 0.883, five-frame policy input, twenty-four-frame plan branch and base weights 0.1/0.3. Compare unguided, flat, TR-adaptive and a phase-shuffled replay of adaptive weights on matched LIBERO initial states. Match each replay's total guidance weight and number of active replans to its adaptive reference, and log realized weights, task phases, grasp failures, ID/OOD success and branch-call latency. Repeat across seeds with uncertainty estimates. If shuffled schedules preserve the adaptive advantage, phase alignment is not needed to explain the result; if adaptive timing wins at a matched intervention budget, the phase-sensitive interpretation gains support. Resolve the paper's grid-budget discrepancy before fixing trial counts. E06E08E12E16E18
Check 2: Test whether high TR means actions depend on useful future content
Reader-proposed, not executed: freeze the model and collect matched replans from object selection, receptacle approach and grasp/placement. Hold the clean current-frame features, proprioception and action-noise sample fixed. Replace only future-frame features with temporally permuted or instruction-mismatched future features from the same checkpoint and extraction-noise setting; retain the original feature sequence as a control. Measure TR, changes in action velocity/chunks and downstream task success, stratified by phase and by independent inspection of whether the original future depicts the correct goal. The routing account predicts stronger future-content sensitivity in high-TR planning states. If actions remain unchanged there, or high TR preferentially follows wrong futures, the diagnostic's causal or reliability interpretation is weakened. E02E03E05E07E17
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block, Figures 1–9, Tables 1–14, Eqs. (1)–(3), Algorithms 1–3 and the supporting method, evaluation and implementation text on the declared pages. All six final crops were inspected; table crops retain every column and rule while surrounding captions are explained in the reading guides. Figure 2's flow directions and the guidance subtraction signs were checked against the text and Algorithm 3. Main-text/appendix mask, action, training-budget, evaluation-accounting and cost discrepancies are preserved in the base report. Reference and contents pages 9–13 were read in text only; separate supplemental videos and external code were outside the supplied material.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and Sections 1–7 (PDF pp. 1–8)
- References (PDF pp. 9–11) and contents (pp. 12–13)
- Appendix A.1–A.3: architecture, training, sampling and guidance; Algorithms 1–3 (pp. 14–16)
- Appendix B.1–B.3: simulation tasks and evaluation (pp. 17–18)
- Appendix C.1–C.4: design sweep, guidance and layer ablations (pp. 18–21)
- Appendix D.1–D.4: implementation and hyperparameters (pp. 20–23)
- Appendix E.1–E.6: hardware, data, real-world evaluation, training and cost (pp. 23–26)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting PDF pp. 1–8 and 14–26, including every figure and table. References and contents were read as text; their page images were not inspected.
- No separate supplementary videos, code, checkpoints or external project material were supplied or inspected; no experiments were reproduced.
- Identity notes: the title page identifies arXiv:2607.08127v1 [cs.CV], 9 July 2026. Its exact title and all six authors match the catalog. The title page additionally verifies Georgia Tech, omitted from the catalog affiliation string. No other revision or edition was supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
E01PDF p. 1, title/author block and arXiv margin
Exact title and six authors; Georgia Tech and Amazon FAR; Mishra internship note; arXiv:2607.08127v1 dated 9 July 2026.
Go to primary source ↓E02PDF pp. 1–2, Introduction; p. 4, Figure 4 and motivating example
The paper studies latent inverse-dynamics VAMs and illustrates both correct action despite wrong video and wrong action despite correct video.
Go to primary source ↓E03PDF pp. 2–3, Section 2.1 and Figure 2; p. 14, Appendix A.1
Cosmos features feed a distinct Gemma expert through feature/noise projection; current image, language, state and noisy actions have separate roles. A.1 specifies backbone, tokenizer, text encoder and action-head architecture.
Go to primary source ↓E04PDF pp. 2–3, Eqs. (1)–(2), Section 2.2; pp. 14–15, A.2 and Algorithm 1; pp. 20, 22, D.1–D.2 and Tables 8–9
Flow-matching targets, joint weight 1, frozen two-stage backbone, LoRA adaptation, uniform video noise, shifted Beta action noise, simulation steps, optimizer, rank and batch.
Go to primary source ↓E05PDF pp. 4–5, Section 4, Eq. (3), Figure 5 and Section 4.1
TR sums head-averaged action-to-future attention divided by action-to-current attention. Figure 5 uses upper/lower triangle markers for successful/failed episodes and shows phase-dependent trajectories.
Go to primary source ↓E06PDF p. 6, Section 4.2; p. 16, Algorithm 3 and following paragraph; pp. 22–23, D.4 and Table 11
Conditional-minus-null-text and extended-plan-minus-conditional velocity directions; TR read once per replan, episode baseline, clipped scaling, weights 0.1/0.3, cap 2, layer 15, extended horizon 24 truncated to policy horizon.
Go to primary source ↓E07PDF pp. 14–15, A.2 and Algorithm 2; p. 20, D.2; p. 24, E.2
Clean-anchor reinjection, truncated FlowUniPC and ten Euler steps; video-feature reuse; simulation replanning every ten steps; YAM 50-action chunks at 30 Hz with first 30 executed.
Go to primary source ↓E08PDF pp. 6–7, Section 5 and Table 1 caption; pp. 17–18, B.1–B.3 and Table 3
Four ID and three OOD ten-task suites; 50 demonstrations/task; headline budget 50 rollouts/task/three seeds; OOD changes language–object–goal bindings. Grid protocol states ten rollouts/task/three seeds but gives inconsistent total 700.
Go to primary source ↓E09PDF p. 7, Table 1, all method rows and ID/OOD average columns; Section 5.2 and footnote 1
Proposed unguided/guided averages are ID 94.3/94.0 and OOD 55.7/59.4. Guided Long is 82.6 versus 84.3. Baseline OOD values, original-paper ID provenance and missing Mimic-Video comparison are explicit. The printed OOD aggregates do not exactly reproduce arithmetic means of the three displayed suite scores.
Go to primary source ↓E10PDF p. 8, Table 2 and Section 5.2; pp. 24–25, E.4 and Table 13
YAM reported averages 71.7 unguided, 83.3 guided, 55.0 pi_0.5, 36.7 pi_0, 12.5 Cosmos-Policy. Tasks 1–2 have 15 trials each; tasks 3–8 five each; success criterion is specified in E.4.
Go to primary source ↓E11PDF pp. 18–19, C.1 and Table 4, Joint + LoRA / T=5 row; pp. 21–22, D.3
Joint-LoRA ID/OOD pairs are 78.0/41.0 at noise 1.000, 94.3/55.7 at 0.883, 82.5/45.3 at 0.024. Training and policy input use five latent frames; larger evaluation video rollouts are truncated.
Go to primary source ↓E12PDF p. 19, Table 5 and C.2; p. 20, Table 6
Unguided/flat/adaptive ID averages 94.3/84.3/94.0; OOD 55.7/57.3/59.4. Object-OOD is 40.0 throughout. Table 5 specifies shared base weights 0.1/0.3 and plan horizon 24.
Go to primary source ↓E13PDF pp. 19–21, C.4 and Figure 8
Layer 15 is selected for TR; shallow layers vary little with noise, deeper layers are more sensitive. The discussion explicitly compares one current frame with four future frames.
Go to primary source ↓E14PDF p. 23, Figure 9 and E.1–E.2; pp. 24–25, E.2–E.4 and Tables 12–13
Two YAM arms, three D405 cameras; 5,600 episodes, 24 tasks, equal task sampling; three evaluation scenes; exact eight instructions and per-task trial counts.
Go to primary source ↓E15PDF p. 25, E.5 Optimization and Compute budget; p. 26, Table 14
Optimization describes decay over 10,000 steps; compute statement/Table 14 specify 1M steps, 32 H200 80GB GPUs, effective batch 512, bfloat16.
Go to primary source ↓E16PDF p. 26, E.6; p. 16, Algorithm 3 lines 24–35
E.6 counts one unguided and three combined-guidance video branches and calls action-head cost unchanged; Algorithm 3 explicitly evaluates additional action velocities for active guidance branches.
Go to primary source ↓E17PDF p. 8, Conclusion and Limitations; p. 26, E.6
The authors restrict conclusions to this VAM class, warn of incorrect futures and inference overhead, and promise future code/model release. E.6 gives pass counts, not measured latency.
Go to primary source ↓E18PDF p. 3, Sections 2.1–2.2; p. 6, Datasets; p. 7, Section 5.2; pp. 14, 17, 20, 22, A.1/B.1/D.1–D.2 and Tables 8–9
Conflicts: causal action-mask shorthand versus detailed prefix mask; 16 versus 17 simulation actions; end-effector versus joint deltas; full-FT learning rates 10^-6 versus 10^-5; unfiltered video data versus filtered-demo description. Grid text says 700 despite 70 tasks × 10 rollouts × 3 seeds implying 2,100.
Go to primary source ↓E19PDF p. 8, Table 2, Cosmos-Policy row and footnote 2; pp. 24–25, E.4/Table 13
Cosmos-Policy entries include 8.3% on 15-trial tasks and 15.0% on five-trial tasks, which cannot be single integer-success fractions under those budgets. No alternate averaging protocol is specified; footnote reports poor training.
Go to primary source ↓E20PDF p. 18, C.1; p. 19, Table 4 and C.3; p. 20, Table 7
C.1 gives 33.0 OOD for Two-Stage+Full where Table 4 gives 29.7. Table 7 high-noise OOD is 7.0 at block 20 and 7.7 at block 28, qualifying C.3’s joint-optimum language.
Go to primary source ↓8.5 Primary sources
Understanding and Mitigating the Video-Action Generalization Gap via Temporal Ratio ↗
PDF · 12,649 extracted words
Source fingerprint
4d94ef759bb3d241d2ae5f6c86c8614a748d93195eb031953356bd323fd719ca