Hydra-0: Action Flow for Generalist World Modeling and Control
1. Paper overview
In one sentence: Action flow translates visible motion into a shared video-conditioning interface, improving prediction across embodiments while leaving physical control dependent on calibrated grounding and a supervised action readout. E02E03E04E08E10E15E16
| At a glance | What to know |
|---|---|
| Research problem | Author claim Native joint or end-effector commands require embodiment-dependent mappings to visible motion. The authors seek a shared interface for heterogeneous robot and human videos that retains a connection to executable commands. E02 |
| Core mechanism | |
| A key reported result | Multi-embodiment video prediction: Reported Wan2.2 A14B 4-step averages: gripper EPE 3.29 ± 0.10; object EPE 5.27 ± 0.35; FVD 155.9. Gripper EPE and object EPE in pixels; FVD; lower is better. 100 common held-out clips per dataset: XVLA-Soft-Fold, Deform360, DROID, MolmoAct2 and ABC-130k; object EPE excludes DROID. Native-action Cosmos 2.5: 34.28 ± 1.02; 13.23 ± 0.97; FVD 405.8, respectively. Reported gripper/object error reductions are 90.40%/60.16%. Averages are unweighted; the ± statistic is undefined. Backbone/configuration changes prevent attributing the entire gain to action flow. These are video-motion results. E02E10E11 |
| Reading caution | Author claim The authors report centimeter-scale grasp imprecision and ambiguous contact, hypothesizing insufficient depth awareness. Wrist-camera evidence is qualitative; closed-loop evaluation remains future work. E17 |
Core contributions
Figure 2. Training and deployment construct the same visual condition from different information. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper lane from left to right: UMI, bimanual and Franka videos supply observed trajectories, separated into embodiment and object tracks. The future video supplies the training target. The dashed downward arrow marks autoregressive conversion. In the lower lane, the initial observation accompanies an Isaac Lab command rollout; robot link motion is projected into the camera to provide embodiment action flow. The causal world model then predicts the scene consequence. The simulator supplies the commanded robot motion in this pathway, while the learned video model supplies the visual outcome. The two lanes explain how videos without robot metadata can contribute to a model later grounded in executable commands. E03E04E07E15
What it supports. The architecture makes a concrete distinction between recovering future tracks for training and constructing robot tracks before forward prediction. Their common image-plane format supports heterogeneous training data, while the deployment lane preserves a connection between candidate motor commands and the motion shown to the video model.
Where the evidence stops. This is the forward architecture, not the inverse action decoder. Deployment still needs robot geometry and camera calibration. The RoboLab experiment uses recorded achieved trajectories, so its replay result does not by itself validate the complete candidate-command pathway.
2. Motivation
2.1 The problem and the proposed response
Native joint or end-effector commands require embodiment-dependent mappings to visible motion. The authors seek a shared interface for heterogeneous robot and human videos that retains a connection to executable commands. E02
2.2 What this reading follows
Hydra-0 asks a video model to reason in the coordinates it already observes: moving points in an image. During training, those points can come from robot or human videos. During forward deployment, robot geometry and a calibrated camera turn simulated command execution into the same trajectory format. A complementary inverse mode supplies desired object motion and decodes robot actions from the resulting latent rollout. The central reading task is to distinguish evidence for visual prediction, replayed policy outcomes and physical execution. The paper supports these at different strengths, with quantitative prediction and replay experiments but only a qualitative inverse-control example. E02E03E04E08E10E15E16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Insufficient evidence to decide
WAM/IDM is functionally supported: desired object motion conditions robot motion and an embodiment-specific supervised readout. Dual-system is ambiguous because heads directly consume shared denoised DiT tokens, without a specified separate model consuming decoded video. This is architecture evidence beyond joint training, but Equation (7) also does not establish joint video/action diffusion. Preserve Q4 pending a defined system boundary. E08E16
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Translate a command into the motion the camera can see
A native command does not uniquely describe how a robot looks while moving: different kinematics and camera viewpoints change its visible consequences. Hydra-0 first obtains robot configurations by running the candidate command through the controller and physics simulation. Surface points fixed to robot links then move with those links and are projected through camera calibration. Visibility rejects points behind the camera, outside the image or inconsistent with the available depth buffer. The result is an image-plane trajectory condition, not a simulated prediction of every manipulated object. During training, dense video tracking can construct the same representation without robot descriptions. Sampling embodiment or object tracks changes what information is supplied through this common tensor. The paper distinguishes observed future training tracks from command-derived forward deployment tracks. E02E03E04
Figure 4. Trajectories move initial appearance features into a conditioning tensor. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the VAE encoder, which provides spatial features of the initial image. The curved arrow carries trajectory information into Feature Propagation. The propagated appearance and presence gate form the motion-aware visual condition; this joins the noisy video latent before the transformer predicts the denoising update. The appendix makes the propagation precise: temporal pooling uses mean visible positions, source features are sampled bilinearly, and each destination keeps the two largest raw Gaussian weights. Their weighted feature sum is not normalized; a separate clipped weight sum indicates where trajectories provide conditioning. Read the final decoder as the route to inspectable video, not a prerequisite for the inverse action heads. E05E06E08E10E13
What it supports. The interface injects spatially aligned appearance evidence rather than only an abstract action vector. Motion flow is a condition on video denoising, not a replacement for the noisy latent. The raw feature mass and presence gate distinguish locations with trajectory support from the surrounding visual context.
Where the evidence stops. The drawing illustrates a concatenation pathway. Cosmos instead adds separately projected motion tokens, and Wan I2V changes conditioning at its expert transition. No isolated ablation establishes that the top-two rule or gate is individually necessary.
5.2 Preserve the meaning of motion features across denoising and time
The trajectory tensor becomes useful to a video backbone by carrying initial appearance into future latent locations. Hydra-0 pools positions and visibility over each latent-frame window, samples a source feature from the initial image, and spreads that feature around the destination using Gaussian weights. Only the two strongest raw contributions survive at a cell. Normalizing their sum would change the implemented condition: the paper retains raw feature mass and separately clips that mass to produce a presence gate. Backbone-specific adapters inject this condition without replacing the noisy video state. Autoregressive conversion adds another alignment constraint. The full motion tensor is computed once and sliced at absolute chunk offsets; generated history is cached. This keeps a trajectory’s destination consistent as the model advances through successive chunks. E05E06E07
5.3 Audit the bridge from inverse prediction to an executed action
Reader interpretation: the inverse mode is best understood as a learned compatibility prediction followed by supervised embodiment-specific decoding. Desired object flow supplies task intent while gripper flow is withheld. The video representation must encode compatible robot motion, and action heads learn to recover executed commands from denoised tokens. Their training data can include failures because each trajectory still pairs observed consequences with actions. This does not remove the need for action-labeled robot data. It also does not make auxiliary state prediction an inference-time controller: states are not fed back. Figure 10 illustrates the resulting physical behavior, but its selected frames cannot establish a success distribution. Likewise, attaching heads to shared transformer features leaves the catalog’s Dual-system boundary debatable; no joint diffusion of action and video variables is established. E08E16
5.4 Training and inference
During training
Seven datasets supply 1,565,634 retained camera-stream windows and 2,201.7 hours, emphasizing deformables. Main windows use 81 frames at 480p/16 fps and a 128 × 128 tracking grid. DROID/EgoDex lack object masks. Static/frozen-gripper filters use a 50-pixel floor; filtering varies by source. E09
Mid-training uses flow matching, trainable patch embedding and rank-64 attention/feed-forward LoRA; other backbone weights stay frozen. Wan I2V-A14B training uses 40,000 steps over five days on 32 H100 GPUs. E06
DMD initializes the student and frozen real-score model from the causal checkpoint. A causal fake-score critic gets five updates per generator update. Seven-latent chunks use four student denoising steps; both score branches use zero guidance. E07
Inverse post-training updates rank-32 self-attention LoRA, motion projection and readout heads using normalized targets from paired real rollouts, including failures. E08
During inference
Forward deployment derives flow from candidate commands. RoboLab replay instead fixes the complete achieved trajectory and starts from the true first image, without querying policies on generated observations. E03E15
The demonstrated controller reads denoised tokens from the 50-step autoregressive teacher. Few-step throughput cannot be assigned to this controller; its feedback/replanning schedule is unspecified. E08E14E16
5.5 Implementation flow
- Construct visible motion trajectories
Isaac Lab executes candidate commands to obtain link transforms. Visible robot-surface points are propagated and projected through the calibrated camera, with visibility checks. Video-only training instead uses dense tracking and grounded embodiment/object masks; observed future tracks provide training conditions. E03E04
- Select the conditioning tracks
None/Embodiment/Object/All select tracks, not learned mode tokens. Canonical probabilities are 0.05/0.40/0.40/0.15, renormalized when pools are unavailable, with dataset/camera overrides. Embodiment/Object sample 1–128 tracks; All samples 256–1024. E04
- Transport appearance through latent space
Temporally pooled tracks transport bilinearly sampled first-frame features. At each latent-grid destination, the top two raw Gaussian weights produce an unnormalized feature sum; their clipped mass forms a presence gate. E05
- Inject motion into the video backbone
Cosmos adds a projected 17-channel motion input to video tokens. Wan I2V uses 16 feature/four mask channels, conditioning the high-noise expert before restoring pristine conditioning for the unmodified low-noise expert. TI2V concatenates a constant 49-channel side input. E06
- Roll out causal chunks
Clean-context teacher forcing and block-causal attention convert the model to autoregression. Generated history enters a KV cache; motion conditions retain absolute offsets instead of re-anchoring at chunk boundaries. E07
- Read actions from inverse-mode features
Desired object flow elicits robot motion. Action/state heads combine mean and single-query attention pooling, layer normalization and two-layer width-1024 GELU MLPs. State outputs provide auxiliary supervision without feedback. E08
6. Experiments & results
Hydra-0 conditions video prediction on visible point trajectories, transporting initial-image features along their paths. Forward mode projects robot motion into this interface; inverse mode takes desired object motion and reads executable actions from denoised transformer features. Prediction and recorded-trajectory replay have quantitative support; physical control is illustrated on one pipe-bending task.
Figure 7 supplies a mid-training/data-efficiency diagnostic, not isolated ablations of the sampling mixture, Gaussian gate or auxiliary state head; those component tests are absent. The inverse-control material contains one illustrated task without repeated-trial statistics or a quantitative control baseline. The edition preserves these limits instead of treating selected frames as a control benchmark. E04E05E08E10E13E16
6.1 Read the original evidence
Table 2. Separate the strongest configuration from the same-backbone representation comparison. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First compare Cosmos 2.5 with Ours (Cosmos 2.5 2B) within each dataset block: this changes native relative 6D action conditioning to action flow under the reported common data protocol. Then inspect the Wan variants to assess the strongest configuration. EPE measures tracked-point motion error in pixels; lower is better, as for FID and FVD. PSNR, SSIM and VLM point upward. Gray ATI/Wan-Move rows are released checkpoints evaluated without the multi-embodiment mid-training. Each dataset uses the same 100 validation clips for every method. Average is unweighted across available strata; DROID’s missing object EPE reflects unreliable object masks, not zero error. E10E11E12E18
What it supports. In the Average block, action-flow Cosmos reduces gripper/object EPE from 34.28/13.23 to 13.80/6.27 pixels. The strongest A14B 4-step row reaches 3.29/5.27 pixels and FVD 155.9, versus 405.8 for native-action Cosmos. The same-backbone result more directly supports the representation; the strongest row combines representation and model-configuration differences.
Where the evidence stops. The source does not define the displayed ± statistic, so it cannot be labeled as a confidence interval or standard error. VLM plausibility and motion following differ: the Cosmos VLM average slightly decreases. All these metrics assess generated video, not physical task completion.
Figure 8. Recorded-motion replay preserves much of the reference outcome ordering. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the horizontal axis as reference environment success and the vertical axis as success judged from generated rollouts. Each point aggregates ten trials. Colors and marker shapes identify pi0, pi0.5, GR00T N1.7, Cosmos-3 Edge and Cosmos-3 Nano; the last two are abbreviated C-3. The dashed line is a least-squares fit, not an identity line. Section 4.5 starts each generated clip from the true first observation and supplies the complete recorded achieved end-effector trajectory as flow. A human rater blinded to policy identity judges the generated outcome. The policy itself is not asked to act on the generated observations. E02E15
What it supports. The plot displays Pearson r = 0.96. The accompanying results report Spearman rho = 0.93 and success-rate MAE of 5.7 percentage points, with the same five-policy ranking after averaging tasks. This supports preserving outcomes of recorded behavior under the stated replay protocol.
Where the evidence stops. The introduction describes five policy aggregates, while Section 4.5 and this figure describe policy–task points. That aggregation discrepancy remains unresolved. High replay correlation does not show that policies would retain their ranking when reacting to generated observations in closed loop.
Figure 10. Desired object motion reaches physical execution through a trained latent action readout. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow a column downward to keep time aligned. Human demo supplies the held-out example; Extracted flow overlays the desired motion of the pipe. That object flow, with embodiment flow excluded, conditions the model’s predicted robot motion in Video generation. Policy execution is a separate physical sequence produced from the learned action-output head. Then follow each row left to right to compare the progress of the bend. The method text explains that the head reads denoised transformer features, so executable actions do not require decoding the generated video pixels. The generated row helps visualize the prediction, while the bottom row documents the illustrated execution. E08E14E16E17
What it supports. The figure shows the authors’ successful flexible-pipe-bending example from a human-derived object-motion condition. It demonstrates the intended path from desired object flow to latent robot motion to executable actions. The paper’s claim of no task-specific expert robot demonstrations coexists with supervised action-readout training on paired robot rollouts.
Where the evidence stops. Only one task is illustrated, with no repeated-trial count, success rate or quantitative baseline. Selected frames cannot establish reliability or precise contact accuracy. The reported controller uses the 50-step teacher; the few-step video-generation speed belongs to a different configuration.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Multi-embodiment video prediction 100 common held-out clips per dataset: XVLA-Soft-Fold, Deform360, DROID, MolmoAct2 and ABC-130k; object EPE excludes DROID. | Reported Wan2.2 A14B 4-step averages: gripper EPE 3.29 ± 0.10; object EPE 5.27 ± 0.35; FVD 155.9. Gripper EPE and object EPE in pixels; FVD; lower is better | Native-action Cosmos 2.5: 34.28 ± 1.02; 13.23 ± 0.97; FVD 405.8, respectively. Reported gripper/object error reductions are 90.40%/60.16%. Averages are unweighted; the ± statistic is undefined. Backbone/configuration changes prevent attributing the entire gain to action flow. These are video-motion results. E02E10E11 |
| Conditioning-representation comparison Same Cosmos backbone, dataset mixture and validation clips; native relative 6D actions versus action flow. | Reported action-flow Cosmos: gripper EPE 13.80 ± 0.85; object EPE 6.27 ± 0.67; FVD 277.4; VLM 3.83 ± 0.05. Average gripper/object EPE; FVD; VLM score | Native-action Cosmos: 34.28 ± 1.02; 13.23 ± 0.97; 405.8; VLM 3.88 ± 0.05. This mechanism comparison improves motion/distribution metrics while VLM plausibility is mixed; neither is executed task success. E10E12E18 |
| Transfer and target-data efficiency Six IWS tasks excluded from mid-training; shared seeded, nested adaptation subsets. | MT beats PT on all three metrics across all tasks at 0%; maximum per-task changes from 20% to 100% data are 3.4%/6.7%/6.8%. LPIPS, object-flow EPE and FVD | MT starts after multi-embodiment training; PT starts from pretrained Wan2.2. The inspected Figure 7 supports transfer and early flattening. FVD uses 40 clips per point and has high variance; this does not establish convergence. E13 |
| Video-generation throughput One 80-GB H100, bfloat16, batch one, 81 frames at 480 × 832; median of five clips after three warm-ups. Guidance is disabled and VAE decoding excluded. | Reported few-step student: 1.31 s/clip, 61.98 FPS, 16.0× speedup. Seconds per clip; FPS; speedup | Bidirectional teacher: 20.92 s/clip, 3.87 FPS. Autoregressive teacher: 12.48 s/clip, 6.49 FPS, 1.68× speedup. Generation-only, short three-chunk timing leaves the attention cap inactive; the bidirectional baseline uses quantized attention. This is neither end-to-end control timing nor Table 2’s LightX2V A14B configuration. E10E14 |
| Open-loop policy evaluation RoboLab: five policies × six tasks × ten recorded rollouts = 300 episodes; a human blinded to policy identity judges generated outcomes. | Reported r = 0.96, ρ = 0.93 and MAE = 5.7 percentage points; task-averaged ranking matches all five policies. Pearson correlation; Spearman correlation; success-rate MAE | Generated-video success judgments versus reference environment success. Section 4.5/Figure 8 use policy–task aggregates; the introduction describes five policy aggregates. Preserve this ambiguity. Recorded-trajectory replay does not establish closed-loop evaluation. E02E15 |
| Object-flow-conditioned physical control One flexible-pipe-bending task using object flow transferred from a held-out human demonstration; no embodiment flow is supplied. | Figure 10 shows an execution the authors identify as successful; no trial count, success rate or uncertainty is supplied. Qualitative executed behavior | No quantitative control baseline is reported. The inspected image supports an illustrative control example, not broad reliability or generalization. E08E16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. Multi-embodiment mid-training improves the starting point for held-out IWS tasks. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read columns as tasks and rows as metrics. The tasks are Bimanual box, Bimanual rope, Bimanual sweep, Push-T, Single chain-in-box and Single grasp. Green circles are Ours (MT), initialized after multi-embodiment mid-training; blue squares are Ours (PT), initialized from the original Wan2.2 checkpoint. Both use the same nested target-data subsets. At the leftmost 0% point, neither receives target-task adaptation, so the comparison probes transfer from mid-training. All three metrics are lower-is-better. Their vertical axes use logarithmic ticks; compare values within a panel rather than treating equal pixel distances as equal absolute changes. ATI and Wan-Move are zero-shot points, not adaptation curves. E13
What it supports. The green MT curves begin below PT across all six tasks and three metrics. Much of the later improvement occurs with limited target data. The accompanying prose reports maximum per-task changes of 3.4% in LPIPS, 6.7% in flow EPE and 6.8% in FVD between 20% and 100% adaptation data.
Where the evidence stops. This is an initialization/data-volume diagnostic, not a single-component ablation or executed-control study. FVD uses 40 clips per point and is described as high variance. Error-bar semantics are unspecified; small late changes do not establish convergence.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors report centimeter-scale grasp imprecision and ambiguous contact, hypothesizing insufficient depth awareness. Wrist-camera evidence is qualitative; closed-loop evaluation remains future work. E17
Forward grounding requires geometry/calibration, and visibility rendering excludes unknown object occlusions. Calibration sensitivity is not quantified. E03E10
ATI/Wan-Move lack Hydra-0 mid-training, confounding those comparisons. No isolated sampling-mixture, Gaussian-gate or state-head ablation is reported. E04E05E08E10E13
How reliably can human object flow transfer across camera geometry, initial states and contact requirements when image-plane motion may underdetermine feasible actions? E08E16E17
7.2 Questions for discussion
- Does replay agreement survive feedback from generated observations? [E15]
- Can the fast student preserve inverse-action accuracy? [E08, E14]
- Would depth/contact inputs improve grasp precision? [E17]
8. Reproducibility audit
8.1 Requirements and known gaps
Rebuilding requires the seven datasets, window/split identities, captions, masks and tracks. Motion floors are given, but full source-specific overrides and episode/scene separation for every held-out split are absent. E04E09E10
Match controller/calibration settings, the 1.2-cm depth tolerance in a 3 × 3 neighborhood, and top-two unnormalized Gaussian aggregation. Learning-rate schedules and complete stage-specific optimization settings are missing. E03E05E06E07
Control additionally requires paired action-labeled rollouts, normalization, action-slot alignment and human-flow transfer. Training-set size, full action parameterization and execution scheduling are missing. E08E16
Reader-proposed checks: repeat matched Cosmos conditioning comparisons under calibration perturbations; compare correct, shuffled and absent object flow in repeated pipe-bending trials. E03E12E16E17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Repeat the representation comparison and perturb camera grounding
Reader-proposed, not run: train native-action Cosmos and action-flow Cosmos on identical episode-disjoint clips, command trajectories, update counts and evaluation seeds. First test whether the Table 2 EPE advantage repeats. Then keep commands, model weights and reference videos fixed while introducing known camera-extrinsic/intrinsic perturbations only into projected deployment flow. Measure robot/object EPE and plot error against induced projection displacement. Include unperturbed calibrated flow and directly tracked achieved-motion flow as controls, reporting the latter separately as a replay condition. Repeated seeds should reveal whether degradation tracks projection error. Failure to recover the initial matched advantage, or severe degradation under small perturbations, would weaken the claimed deployment benefit. E03E09E10E12
Check 2: Test whether correct object flow determines physical pipe bending
Reader-proposed, not run: use the reported teacher/readout configuration and the same paired-rollout training set, then randomize matched physical starting configurations across correct human-derived object flow, another demonstration’s shuffled flow, and absent flow. Hold camera setup, action horizon, execution schedule and model sampling seeds fixed or balanced. Predefine bending success and grasp/contact-error measurements, record repeated trials, and report uncertainty rather than selected successes. Correct flow should yield systematically better target bends than both controls if the inverse pathway uses task intent as claimed. Similar outcomes would suggest weak conditioning or dominance of learned behavior; persistent contact error despite good visual motion would expose a prediction-to-execution gap. E04E08E14E16E17
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version on page 1; Figures 1–11; Tables 1–3; method and training on pages 2–7; corpus and compute on pages 6–9; evaluation protocols and limitations on pages 10–14; and all appendix implementation details on pages 21–23. Every final crop was separately viewed. Table 2 was checked against the original rows; Figure 7 was read for trends without digitizing exact curve coordinates. The six crops retain original labels, axes and legends while excluding long captions; necessary caption qualifications are supplied in the reading guides. References on pages 15–20 were read as text; cited works, external videos, code and separate supplements were not inspected. No experiments were run.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title page and abstract, PDF p. 1
- 1. Introduction, PDF pp. 2–3
- 2. Method, including Sections 2.1–2.4, PDF pp. 3–6
- 3. Datasets, PDF pp. 6–7
- 4. Experiments, including Sections 4.1–4.6, PDF pp. 7–13
- 5. Related Work, including Sections 5.1–5.2, PDF pp. 13–14
- 6. Limitations and 7. Conclusion, PDF p. 14
- References, PDF pp. 14–21; referenced publications were not separately inspected
- 8. Appendix, including Sections 8.1–8.9, PDF pp. 21–23
Outside the original text pass
- Identity: the inspected title, all eleven authors and identifier match the catalog. The title-page stamp is arXiv:2608.18077v1 [cs.RO], 18 August 2026. This report covers that supplied edition only; no title, author or version discrepancy was observed. [E01]
- All eight supplied text chunks were read in full, including references and Appendix Sections 8.1–8.9. Referenced publications were not separately inspected.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The extraction limitation above was addressed by visually inspecting all eleven figures, all three tables, and supporting method, evaluation and appendix pages. Figure 7 was read without inventing exact coordinates or interpreting undefined error bars.
- Separate supplemental material availability has not been fully verified.
- No separate supplements, project resources, code or checkpoints were inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
E01PDF p. 1, title page, author block and arXiv version stamp
The observed title is Hydra-0: Action Flow for Generalist World Modeling and Control. The eleven authors, in order, are Hongyu Li, Bowen Wen, Xinghao Zhu, Yixuan Wang, Yilun Du, Yunzhu Li, George Konidaris, Stan Birchfield, Soha Pouya, Chenran Li and Yan Chang. Affiliations are NVIDIA, Brown University, Columbia University and Harvard University. The stamp identifies arXiv:2608.18077v1, 18 August 2026; title, identifier and authors agree with the catalog.
Go to primary source ↓E02PDF p. 2, Section 1, motivation, overview and Contributions items 1–4
The authors motivate image-plane motion by embodiment-dependent command-to-appearance mappings and propose shared forward/inverse use. The introduction describes r = 0.96 across five policy aggregates, creating an aggregation ambiguity with Section 4.5.
Go to primary source ↓E03PDF pp. 3–4, Section 2.1, Equation (1); PDF p. 21, Section 8.2; PDF p. 3, Figure 2, offline and online lanes
Robot-surface points are propagated through link transforms and calibrated projection. Deployment transforms come from Isaac Lab controller-and-physics rollouts. Visibility requires positive depth, image bounds and depth-buffer agreement within 1.2 cm over a 3 × 3 neighborhood. Unknown object occlusions are not explicitly removed; large projection errors can degrade correspondence.
Go to primary source ↓E04PDF p. 4, Section 2.1, Video-only construction and Training-time flow sampling; PDF p. 7, Training; PDF pp. 21–22, Section 8.3
Tracking and grounded masks supply video-derived conditions. Four modes select trajectories without a learned mode token. Canonical probabilities are 0.05/0.40/0.40/0.15 for None/Embodiment/Object/All; track counts are 1–128 or 256–1024. Unavailable pools cause renormalization, with dataset/camera overrides.
Go to primary source ↓E05PDF pp. 4–5, Section 2.2, Equations (2)–(4); PDF p. 22, Section 8.4, Equations (8)–(10); PDF p. 5, Figure 4, feature propagation and DiT input
The model predicts future video through an encoder, conditioned latent dynamics and decoder. Pooled tracks transport bilinearly sampled initial features using Gaussian locality 220. Only the top two raw weights contribute; their unnormalized weighted sum forms motion features and their clipped sum forms the gate.
Go to primary source ↓E06PDF p. 5, Section 2.2, Equation (5) and freezing policy; PDF p. 7, Section 4.1; PDF p. 9, Training continuation; PDF pp. 22–23, Sections 8.5–8.6
Training uses flow matching, trainable patch embedding and rank-64 attention/feed-forward LoRA. Cosmos uses a 17-channel side input; Wan I2V uses 16 feature and four mask channels with conditioning confined to the adapted high-noise expert; TI2V uses 49 side-input channels. A14B training uses 40,000 steps, five days and 32 H100 GPUs.
Go to primary source ↓E07PDF p. 5, Section 2.3, Equation (6); PDF p. 23, Sections 8.7–8.8
Causal conversion uses clean-context teacher forcing, block-causal attention and cached generated history. Motion tensors retain absolute offsets. DMD uses seven-latent chunks, four student denoising steps, causal real/fake score models, five critic updates per generator update and zero guidance in both score branches.
Go to primary source ↓E08PDF p. 6, Section 2.4, Inverse mode and Equation (7)
Desired object flow conditions robot motion. Action/state heads pool clean DiT tokens and use normalization plus two-layer width-1024 GELU MLPs. Post-training updates rank-32 self-attention LoRA, motion projection and heads. Normalized masked Huber action/state losses and action-difference L1 loss use weights 0.1 and 0.1. State output is auxiliary; deployment reads denoised tokens without pixels.
Go to primary source ↓E09PDF pp. 6–7, Section 3 and Table 1, Total row
Seven datasets yield 1,565,634 retained windows and 2,201.7 hours counted on annotated camera streams. Main windows use 81 frames at 480p/16 fps and a 128 × 128 AllTracker grid. DROID/EgoDex object masks are unavailable. Static and frozen-gripper filters use a 50-pixel floor; a separate DROID wrist stream uses 17-frame windows.
Go to primary source ↓E10PDF p. 9, Table 2 caption and Section 4.2; PDF p. 10, Section 4.2 protocol and Results
Methods share 100 validation clips per dataset across five held-out sets. DROID object EPE is omitted. Native-action Cosmos is fine-tuned; ATI and Wan-Move use released checkpoints without multi-embodiment mid-training. The A14B four-step evaluation includes a LightX2V checkpoint. The prose identifies the Cosmos representation comparison as controlled.
Go to primary source ↓E11PDF p. 9, Table 2, Average block, Cosmos 2.5 and Ours (Wan2.2 A14B 4-step) rows
Native-action Cosmos reports object EPE 13.23 ± 0.97, gripper EPE 34.28 ± 1.02 and FVD 405.8. A14B four-step reports object EPE 5.27 ± 0.35, gripper EPE 3.29 ± 0.10 and FVD 155.9. Averages are unweighted across available dataset strata.
Go to primary source ↓E12PDF p. 9, Table 2, Average block, Cosmos 2.5 and Ours (Cosmos 2.5 2B) rows; PDF p. 10, Section 4.2 Results
Action-flow Cosmos reports object EPE 6.27 ± 0.67, gripper EPE 13.80 ± 0.85, FVD 277.4 and VLM 3.83 ± 0.05. Native-action Cosmos reports 13.23 ± 0.97, 34.28 ± 1.02, 405.8 and 3.88 ± 0.05. The authors distinguish mixed VLM results from improved motion accuracy.
Go to primary source ↓E13PDF pp. 10–11, Section 4.3, shared task/subset protocol and Results; PDF p. 11, Figure 7, all six task columns and three metric rows
IWS is excluded from mid-training; six tasks use common nested adaptation subsets. The prose reports lower zero-shot LPIPS/EPE/FVD for MT than PT on all tasks. From 20% to 100% data, maximum per-task changes are 3.4%/6.7%/6.8%. FVD uses 40 clips per point and is described as high variance. Figure 7 labels MT green circles, PT blue squares, IWS orange triangles, Cosmos purple diamonds, ATI red triangles and Wan-Move cyan crosses; all metrics are lower-is-better. Tasks are Bimanual box, Bimanual rope, Bimanual sweep, Push-T, Single chain-in-box and Single grasp.
Go to primary source ↓E14PDF pp. 11–12, Section 4.4, Protocol and Results; PDF p. 12, Table 3
On one 80-GB H100, generation-only timing for 81 frames at 480 × 832 gives 20.92/12.48/1.31 seconds and 3.87/6.49/61.98 FPS for bidirectional teacher, autoregressive teacher and student. Guidance and VAE decoding are excluded; the three-chunk horizon does not exercise the attention cap. The control model uses the 50-step autoregressive teacher.
Go to primary source ↓E15PDF p. 6, Section 2.4, Forward mode; PDF p. 12, Section 4.5, Protocol and Results; PDF p. 12, Figure 8, axes, legend and caption
Replay fixes complete recorded gripper motion and starts from the true first observation without querying policies on generated frames. RoboLab covers five policies, six tasks and ten rollouts per pair, totaling 300 episodes. Section 4.5 describes policy–task comparisons and reports r = 0.96, ρ = 0.93, MAE 5.7 percentage points and matching task-averaged policy ranking. Each plotted point aggregates ten trials; the dashed line is a least-squares fit. Policies are pi0, pi0.5, GR00T N1.7, Cosmos-3 Edge and Cosmos-3 Nano.
Go to primary source ↓E16PDF p. 13, Section 4.6 and Figure 10, all four rows and caption
The qualitative control study uses one flexible-pipe-bending task. Desired object flow comes from a held-out human demonstration; embodiment flow is excluded. A trained action head converts latent rollout features to commands. The caption reports successful execution, but the text supplies no repeated-trial success rate or control baseline. Six matched time steps separate Human demo, Extracted flow, Video generation and Policy execution.
Go to primary source ↓E17PDF p. 14, Section 6, Limitations
The authors identify centimeter-scale grasp imprecision, hypothesized insufficient depth awareness, and ambiguous grasp/contact states. Depth, tactile and force sensing are proposed directions. Wrist-camera evidence is qualitative, and closed-loop policy evaluation remains future work.
Go to primary source ↓E18PDF p. 23, Section 8.9, VLM Judge Details
Gemma 4 31B uses deterministic decoding and 16 uniformly spaced frames from each 81-frame generated clip. It rates physical plausibility, temporal consistency, object permanence and motion realism on a 1–5 scale; ratings are averaged within and across clips.
Go to primary source ↓8.5 Primary sources
Hydra-0: Action Flow for Generalist World Modeling and Control ↗
PDF · 13,010 extracted words
Source fingerprint
6f1e0ad362a26dc4b0fe98f1ac5c2e616b25c87f1ff3f41144ad709f51a188ae