LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models
1. Paper overview
In one sentence: LD4WAM learns motion-grounded transition targets from human and robot videos, then predicts those targets between future-video generation and action, while retaining a dependence on the quality of imagined scenes. e-ldme-bridgee-implementatione-reale-generalizatione-ablation
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that predicting human-video pixels leaves useful dynamics weakly connected to robot actions, while directly retargeting human motion entangles transferable behavior with embodiment-specific kinematics. They seek an intermediate representation that captures semantic change and is grounded in observed motion without requiring robot-action labels for every video. e-problem |
| Core mechanism | Author claim A latent transition representation trained through semantic reconstruction and motion alignment is proposed as an embodiment-agnostic bridge. Its invariance is a claimed property, evaluated indirectly through retrieval, regression and downstream control. e-probleme-ldme-probee-ablation |
| A key reported result | Physical robot manipulation: 70.5 Mean task success rate (%). Five gripper tasks and two dexterous-hand tasks; 50 demonstrations per task and 30 rollouts per setting. Lingbot-VA: 64.3; π0.5: 63.3; Fast-WAM: 47.1. This is reported physical execution, not video quality. The model is best or tied best on six of seven tasks, but π0.5 is stronger on Tidy Desk (73.3% versus 70.0%). Spray Water remains at 46.7%, and the table supplies no uncertainty intervals. e-evale-real |
| Reading caution | Reader analysis Representation invariance is stronger than the evidence establishes. Tested object transfer and motion readout are encouraging, but background failures persist and the proposed video-degradation explanation remains an author interpretation. e-generalizatione-probe-results |
Core contributions
- Author claim
A latent transition representation trained through semantic reconstruction and motion alignment is proposed as an embodiment-agnostic bridge. Its invariance is a claimed property, evaluated indirectly through retrieval, regression and downstream control. e-probleme-ldme-probee-ablation
- Source description
WDAM introduces a clean query stream between separately parameterized video and action experts, retaining future-video generation while predicting compact dynamics for action conditioning. e-bridge
- Source description
The curated pretraining corpus combines existing human and robot datasets into 274.66 million frames and 5,086.3 hours, with human video accounting for 76.4% of frames. e-data
Figure 3. Learn a transition target first, then predict it inside the video-to-action stack. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left block from bottom to top. Temporally skipped human and robot frames enter frozen DINOv3, then a spatiotemporal transformer and SoftVQ. Two decoder branches supervise the resulting transition representation: semantic reconstruction and end-effector motion alignment. On the right, green video, blue latent-dynamics and purple action experts retain separate parameters while exchanging information through shared attention. The blue inputs are clean learnable queries, not another noisy diffusion sequence. Finally, read the three stages along the bottom: broad video pretraining, robot alignment and task-specific adaptation. The snowflake under the first action block records a stage-specific freeze, not the deployment schedule. e-ldme-bridgee-implementatione-stages
What it supports. The dynamics representation provides an additional supervised path from video features to action. Appendix C makes the deployment distinction explicit: the frozen LDM supplies training targets, while the latent expert predicts them during control. Video is denoised before actions are solved; predicting dynamics does not introduce another denoising loop.
Where the evidence stops. The attention mask is described in the text rather than drawn as a complete matrix here. A separate video stream does not imply unchanged pretrained parameters: the video DiT is optimized during training.
2. Motivation
2.1 The problem and the proposed response
The authors argue that predicting human-video pixels leaves useful dynamics weakly connected to robot actions, while directly retargeting human motion entangles transferable behavior with embodiment-specific kinematics. They seek an intermediate representation that captures semantic change and is grounded in observed motion without requiring robot-action labels for every video. e-problem
2.2 What this reading follows
LD4WAM separates two questions that are easy to conflate: can a representation describe motion across embodiments, and can a controller use it to act? First, an LDM learns semantic transitions with end-effector supervision where labels exist. Then a video–dynamics–action model predicts that representation rather than running the LDM at deployment. The paper offers three complementary checks: cross-domain neighbors, frozen-latent motion regression and executed manipulation. Read them together. The strongest physical mean is useful evidence, but lower performance under background shifts and the cumulative nature of the curriculum prevent a simple claim of universal motion invariance. e-ldme-bridgee-implementatione-reale-generalizatione-ablation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Insufficient evidence to decide
WAM and IDM are supported: Appendix C explicitly describes inverse-dynamics training, and deployment solves actions after predicting video. The Dual-system boundary is less clear: experts have separate weights and staged sampling, yet interact throughout one integrated MoT attention stack. LDM is a training-target generator absent at deployment, so counting LDM and WDAM alone does not establish two deployed systems. Without an operational catalog definition, Q4 cannot be conclusively endorsed or rejected. Latent prediction and action alignment are supported; semantic reconstruction does not by itself establish JEPA. e-ldme-bridgee-implementation
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Give an unlabelled visual transition a motion-grounded target
The training signal is deliberately asymmetric across datasets. Every suitable video transition can supervise future semantic reconstruction, but only transitions with verified motion labels contribute end-effector alignment. The current semantic feature and the transition latent jointly predict the next feature, so the latent need not store all static appearance itself. A frozen DINOv3 encoder defines the feature space; a trainable transformer and SoftVQ learn the transition code. The appendix specifies sixteen 32-dimensional slots, giving 512 dimensions per transition, with an eight-entry codebook. A stricter roughly 1,500-hour subset trains this teacher before the larger WDAM curriculum. Neither semantic reconstruction nor the auxiliary motion decoder alone constitutes a robot policy; they define targets for the later model. e-ldme-ldm-traininge-data
Figure 5. Selected neighbors share motion across visibly different embodiments and scenes. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each colored row begins with a two-frame query, followed by three retrieved two-frame examples. The first two queries come from Agibot-World and distinguish forward from backward robot movement. The last query comes from EgoDex and shows upward human-hand motion. Dataset labels under the panels distinguish human video from robot demonstrations; compare the change between frames rather than static resemblance between objects. Retrieval uses cosine similarity in the frozen LDM latent space. This tests the structure of representations extracted from observed videos. It does not run the WDAM controller or demonstrate that a generated future will produce the same movement. e-retrieval-visuale-probee-ablation
What it supports. The examples illustrate the intended semantic-motion abstraction: related changes can be neighbors despite different arms, objects and backgrounds. They motivate checking the quantitative motion probe and the control ablation, which test whether this visual impression extends to measurable readout and task execution.
Where the evidence stops. These are selected examples. The paper supplies no retrieval accuracy, total query population or failure distribution for this figure, so the montage cannot establish general embodiment invariance.
5.2 Predict the teacher’s dynamics without invoking it during control
WDAM uses distinct parameter sets for video, dynamics and action, connected through shared attention at each layer. Video attends to video; clean dynamics queries read video and themselves; action reads video and dynamics. The queries regress the frozen teacher’s transition targets without a diffusion timestep. During inference, video denoising happens first and the action expert then solves actions conditioned on the fixed predicted video and dynamics. LDM itself is absent. This distinction matters when counting systems: the teacher is a training dependency, whereas the interacting experts form the deployed path. Appendix C also refines the main-text statement about offline targets: Stage I computes them online from frozen LDM, and subsequent stages use precomputed targets. e-bridgee-implementatione-stages
5.3 Test whether a useful latent survives the move from observations to predictions
My reading is that the regression and policy studies test two different interfaces. The probe asks whether dynamics extracted from observed videos expose motion to a small decoder; the controller must instead use dynamics inferred from generated futures. A strong probe therefore leaves a transfer gap. The control ablation narrows that gap: aligned semantic targets outperform the motion-unaligned and pixel-reconstruction alternatives in the same intermediate architecture. It does not remove the dependence on future-video quality. Training uses teacher-forced video, deployment uses model predictions, and background shifts remain problematic. A discriminating follow-up would hold the downstream policy fixed while comparing observed versus generated future conditioning under matched shifts, rather than treating retrieval examples as a causal explanation of robot success. e-probee-probe-resultse-ablatione-generalizatione-implementation
5.4 Training and inference
During training
LDM is trained on a more strictly filtered subset of roughly 1,500 hours. Motion supervision is introduced after semantic warmup and uses verified labels from EgoVerse, EgoDex, Xperience and Agibot-World. DINOv3 remains frozen. e-ldm-training
WDAM Stage I trains video and dynamics experts on human/robot video. Stage II trains all experts on general robot data and initializes or reinitializes the action expert and proprioception encoder. Stage III adapts to task-specific robot demonstrations. The respective loss weights are (1,1,0), (1,0.5,1) and (1,0.1,1), over one, one and five epochs. e-stages
Video denoising uses teacher forcing during inverse-dynamics training. Appendix C specifies frozen-LDM targets computed online in Stage I and precomputed thereafter, refining the main text's blanket offline-target description. T5, the video VAE and LDM remain frozen; the video DiT is trained. e-implementatione-stages
During inference
Deployment first denoises video, then solves actions from that video and predicted dynamics. LDM itself is not invoked, and dynamics prediction adds no separate denoising loop. e-bridgee-implementation
Asymmetric attention restricts information flow into the video stream; it does not establish that pretrained physical priors remain unchanged, because the video expert's parameters are optimized during training. e-bridgee-stages
5.5 Implementation flow
- Standardize observations and motion
Segment videos using annotated subtask boundaries, retain clips lasting 3–60 seconds, and downsample to 15 FPS. Human poses are expressed relative to the initial head camera; wrist and gripper frames are constructed with corresponding physical axes. Filtering removes excessive camera motion, missing hands and insufficient manipulation. e-datae-filters
- Encode observed transitions
A frozen DINOv3 encoder maps temporally subsampled frames into semantic features. A spatiotemporal transformer and SoftVQ produce a 512-dimensional representation per transition, assembled from 16 code slots of dimension 32 using an eight-entry codebook. e-ldm
- Ground the representation
A semantic decoder reconstructs future features from current features and the transition latent. A motion decoder predicts end-effector deltas where labels exist. Thus unlabeled video contributes semantic supervision, while labeled transitions provide motion grounding. These decoders train the representation; they do not execute robot actions. e-ldme-implementation
- Predict futures and distill dynamics
WDAM combines video, latent-dynamics and action experts with separate weights and shared per-layer attention. Within that attention, video tokens receive no downstream tokens; clean learnable queries attend to video and themselves; action tokens receive video and dynamics information. Text also conditions the experts. The query stream has neither diffusion noise nor a timestep. e-bridge
- Translate predicted futures into actions
After video denoising, the action expert generates actions conditioned on the fixed predicted video and one-shot dynamics. Its transformer body is shared across embodiments, while input/output projections accommodate different action dimensions. e-bridgee-implementation
6. Experiments & results
LD4WAM connects generated robot futures to action through a learned representation of motion. A separately trained Latent Dynamics Model supplies semantic transition targets grounded by end-effector supervision; a three-expert transformer then predicts video, distills those targets through learnable queries, and generates robot actions. The strongest mechanism evidence combines improved motion readout with executed-control ablations. Object transfer improves, but background robustness remains below the evaluated VLA baseline.
6.1 Read the original evidence
Table 2. Physical execution improves on average, with task-specific failures still visible. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first five task columns belong to the parallel-gripper platform; the last two use dexterous hands. Each cell is a success percentage, based on 30 rollouts per setting after collecting 50 demonstrations per task. Read across the LD4WAM row before focusing on Average: Sorting ties Lingbot-VA at 80.0%, while Tidy Desk remains below π0.5. Shift Test Tube and Spray Water reach only 50.0% and 46.7%. The far-right column summarizes the seven tasks. It should be compared with the same column for other policies, rather than with the nine-condition average in the separate generalization ablation. e-reale-evale-stages
What it supports. The reported mean rises to 70.5%, compared with 64.3% for Lingbot-VA, 63.3% for π0.5 and 47.1% for Fast-WAM. The 6.2 percentage-point gain over Lingbot-VA concerns executed physical tasks. The strongest aggregate includes one tie and one task on which another method is better.
Where the evidence stops. Thirty rollouts per setting provide limited precision, and no confidence intervals or multi-seed training variation are shown. Baseline pretraining exposure and computation are not demonstrated to be matched.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin manipulation 50 tasks; reported training uses 50 clean and 500 randomized demonstrations per task and a unified multitask policy; 100 evaluation rollouts per task. | 93.96 clean; 92.78 randomized; 93.4 reported average. Success rate (%) | Lingbot-VA averages 92.2%; ACE-Ego-0 averages 90.9%. Reported simulated execution is competitive, with modest aggregate gains over the strongest baselines. The authors characterize this benchmark as a competitiveness check. e-evale-sim |
| Physical robot manipulation Five gripper tasks and two dexterous-hand tasks; 50 demonstrations per task and 30 rollouts per setting. | 70.5 Mean task success rate (%) | Lingbot-VA: 64.3; π0.5: 63.3; Fast-WAM: 47.1. This is reported physical execution, not video quality. The model is best or tied best on six of seven tasks, but π0.5 is stronger on Tidy Desk (73.3% versus 70.0%). Spray Water remains at 46.7%, and the table supplies no uncertainty intervals. e-evale-real |
| Unseen-object generalization Object substitutions on Tidy Desk, Fold Shirt and Handover Mug. | 88.6, as reported by the authors. Retention of in-distribution performance (%) | Referenced to LD4WAM's own base performance; the authors report the highest retention among evaluated methods. This is a relative retention measure, not an 88.6% absolute success rate. It supports transfer across tested objects without proving general embodiment invariance. e-generalization |
| Background generalization The same three tasks under lighting, tabletop-texture and distractor changes. | 44.4 Mean success rate (%) | π0.5: 54.5. Background robustness remains a weakness. The authors attribute it to degraded generated futures, but do not experimentally isolate that causal explanation. e-generalization |
| End-effector motion readout Frozen LDM; RoboTwin is outside its training corpus. Eight-frame clips from 2,500 clean and 2,500 randomized episodes are split 8:2 for probe training/validation; the reported comparison is at 5 Hz. | 1.815 Scale-normalized translation-and-rotation MSE; lower is better | Without motion alignment: 8.497; without multi-stride sampling: 2.205; pixel reconstruction: 2.260; hard VQ: 3.236. Motion alignment most strongly improves this readout. Translation and rotation are normalized by 0.01 m and 0.08 rad, so these values are neither raw physical-unit errors nor task success. e-probee-probe-results |
| Architecture and curriculum ablation Mean over three real-world tasks, each evaluated in base, object and background conditions. | 40.0 baseline → 48.1 with dynamics expert → 58.9 with pretraining → 63.7 with alignment training. Mean success rate (%) | At the dynamics-expert stage, removing motion alignment gives 43.3; substituting pixel reconstruction gives 44.5. Executed-control results support the representation's usefulness. This cumulative curriculum does not isolate human-video contribution under matched data and compute. Its nine-setting average differs from the seven-task 70.5% result. e-ablatione-real |
| Hanging Mug in RoboTwin One of the same 50 simulated tasks; clean and randomized evaluation columns in Table 8. | 37 clean; 49 randomized Success rate (%) | Fast-WAM: 58 clean; 62 randomized. The table contains a clear task-level weakness beneath the leading aggregate. These are observed simulated executions, not physical-robot tests. e-sim-taske-eval |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Follow the cumulative rows and replacement branches separately. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each task contributes three columns: Base, Obj and Bg. The rightmost Average therefore combines nine task-condition settings, not the seven tasks in Table 2. Start at the dual-expert baseline, then follow the plus rows downward. The bent-arrow rows replace the LDM supervision used immediately above; they are alternatives, not ingredients added to the subsequent model. Adding latent dynamics raises the mean from 40.0% to 48.1%; removing motion alignment at that point gives 43.3%, while pixel reconstruction gives 44.5%. The next plus rows restore the intended representation and add broad pretraining, then robot alignment. e-ablatione-generalizatione-reale-stages
What it supports. The cumulative sequence reaches 58.9% after pretraining and 63.7% after alignment. The source also reports 88.6% retention under object changes, which is relative retention rather than absolute success. Background success remains 44.4%, below π0.5 at 54.5%, exposing a failure mode hidden by an overall improvement.
Where the evidence stops. The curriculum changes both data and optimization, so its gains do not isolate human video under matched compute. The claim that degraded future images cause background failures is an author explanation, not a separately controlled causal test.
Table 7. Motion grounding helps a frozen-latent probe, especially over larger temporal gaps. Original paper, p. 12 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Columns vary the frame sampling rate from 50 Hz to 2.5 Hz. These are separate readout experiments, not the robot controller frequency. At each rate the LDM remains frozen and a shallow head predicts both arms’ translation and rotation changes. Lower scale-normalized MSE is better. Read one column vertically to compare supervision choices without changing the rate. Then inspect the clip-count rows: the lower-rate experiments contain far fewer clips because longer temporal spacing produces fewer non-overlapping samples. Translation and rotation are divided by fixed scales of 0.01 m and 0.08 rad before scoring, so the printed values are not errors expressed directly in metres or radians. e-probee-probe-resultse-ablation
What it supports. At 5 Hz, the full latent gives 1.815 MSE, versus 8.497 without motion alignment. At 2.5 Hz the corresponding values are 3.956 and 20.686. This supports motion information being more accessible in the aligned representation under the specified probe, while the separate control ablation tests usefulness for actual actions.
Where the evidence stops. The described split is over clips; episode-disjoint validation is not explicitly established. Rate comparisons also change motion scale and available sample count. Neither these errors nor the retrieval examples can substitute for robot success rates.
7. Analysis & limitations
7.1 What the evidence leaves open
Representation invariance is stronger than the evidence establishes. Tested object transfer and motion readout are encouraging, but background failures persist and the proposed video-degradation explanation remains an author interpretation. e-generalizatione-probe-results
Real-world evaluation uses 30 rollouts per setting without reported confidence intervals or variation across training seeds. The supplied comparison does not establish matched baseline pretraining data or compute. e-evale-reale-stages
The probe splits clips rather than explicitly documenting episode-disjoint validation. Non-overlapping clips can still share an episode, so independence of validation trajectories is unresolved. e-probe
Training uses teacher-forced video while deployment consumes predicted futures. The supplied evaluation does not quantify this distribution shift through video-error measurements, nor provide deployment latency or replanning-frequency results. e-implementatione-eval
7.2 Questions for discussion
- Does motion alignment retain its control advantage when pretraining data, optimization budget and episode-level evaluation splits are matched? [e-ablation, e-probe]
- Would conditioning actions on semantic future predictions directly reduce background sensitivity, or does decoded-video generation supply essential information? [e-bridge, e-generalization]
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction requires the curated datasets, DINOv3 ViT-L/16, Wan2.2 TI2V-5B, coordinate conversion and embodiment-specific projections. WDAM training is reported on 64 NVIDIA H20 GPUs with bf16 and ZeRO-2; Table 6 supplies stage schedules and optimizer settings. e-datae-ldme-implementatione-stages
Numeric video-only filtering thresholds are supplied, but action-annotated filtering thresholds and stricter LDM-subset cutoffs remain unspecified. Other gaps include LDM loss coefficients, motion-warmup duration, temporal-stride sampling details and deployment denoising settings. e-filterse-ldme-ldm-traininge-implementation
A focused reproduction would first compare full and motion-unaligned LDMs with an episode-disjoint probe, then repeat the corresponding control ablation under matched training budgets. This separates representation readout from executed usefulness. e-probee-ablation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate motion readout from episode overlap
Reader-proposed: split RoboTwin episodes before creating clips, fit identical probes on full and motion-unaligned LDMs, and match the number of clips within each rate. Report translation and rotation separately as well as the normalized joint score. If the alignment advantage persists on unseen episodes at matched sample counts, it is stronger evidence for transferable motion information; a large collapse would expose dependence on the original clip-level split. e-probee-probe-results
Check 2: Locate the background-shift bottleneck
Reader-proposed: fix the trained action expert and evaluate held-out shifted scenes with either predicted future conditioning or an oracle future observation supplied offline for diagnosis. Compare aligned dynamics, unaligned dynamics and masked dynamics with the same horizon and sampling budget. Measure action error before any physical execution, then use a controlled simulator for task success. Recovery only with oracle futures would implicate prediction quality; persistent failure would point toward the dynamics-to-action interface. The oracle route is diagnostic and cannot be described as a deployable policy. e-implementatione-generalizatione-ablation
8.3 Reading coverage
Visual audit: Read the complete 14-page text and Appendices A–G. Visually inspected the title block, Figure 3 architecture/curriculum, Figure 4 shift settings, Figure 5 retrieval examples, Figure 6 physical rollouts, Tables 1–4 and Table 7, and selected Table 8 columns. Final crops of Figure 3, Figure 5 and Tables 2, 4 and 7 were re-opened after rendering at 220 DPI (350 DPI for the narrow Table 7). Figure 1, Figure 2, Figure 7 and Figure 8, external videos and code were outside this visual pass. No experiments were run.
PDF pages inspected for this edition: 1, 4, 5, 6, 7, 12, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- Introduction
- Related Work
- Method: Data Process and Composition
- Method: Latent Dynamics Model
- Method: World Dynamics Action Model
- Method: Training Procedure
- Experiment: Setup
- Experiment: Result Analysis
- Experiment: Ablation Analysis
- Conclusion
- References, as supplied bibliography only
- Appendix A: Data Processing Details
- Appendix B: Dataset Details
- Appendix C: Training Details
- Appendix D: Real-World Setup and Task Description
- Appendix E: Latent Dynamics Regression
- Appendix F: RoboTwin Detailed Results
- Appendix G: More Real-world Deployment Visualizations
Outside the original text pass
- The complete 14-page PDF text, including Appendices A–G and the bibliography, was read. Images were inspected on PDF pages 1, 4, 5, 6, 7, 12 and 14; Figure 1, Figure 2, Figure 7 and Figure 8 were not visually audited.
- Table 8 was visually checked for the aggregate row and the Hanging Mug comparison; its remaining per-task results were not independently transcribed or recomputed.
- Tables 5 and 6 were read from extracted text; their original table layouts were not visually audited in this edition.
- Separate supplements and the project implementation were not inspected. No experiments were reproduced.
- The source title, identifier and author list agree with the catalog. The inspected title page identifies arXiv:2608.22403v1, dated 23 August 2026.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title block, author list and arXiv version stamp
The supplied title and authors match the catalog; the stamp identifies 2608.22403v1 dated 23 August 2026. A project-page URL is listed.
Go to primary source ↓e-problemPDF pp. 1–2, Introduction and contribution bullets
The authors motivate semantic, motion-grounded transition representations as a bridge between human-video prediction and executable robot actions.
Go to primary source ↓e-dataPDF p. 3, Data Process and Composition; p. 10, Table 5, Total and Frame ratio rows
Clips are restricted to 3–60 seconds and 15 FPS. Table 5 reports 274.66M frames, 5086.3 hours and a 76.4% human/23.6% robot frame ratio.
Go to primary source ↓e-filtersPDF p. 10, Appendix A, Unified coordinate frames and Filtering criteria
The appendix defines initial-camera and corresponding wrist/gripper frames. It specifies numeric hand-detection and optical-flow filters for video-only data, while describing action-annotated rejection criteria qualitatively.
Go to primary source ↓e-ldmPDF p. 3, Latent Dynamics Model, Eqs. (1)–(2); p. 4, encoder-only target extraction; p. 10, Appendix C, Latent Dynamics Model
Frozen DINOv3 features feed a transformer and SoftVQ. Semantic and labeled-motion decoders supervise transition latents. Eight codebook entries and sixteen 32-dimensional slots yield a 512-dimensional transition representation.
Go to primary source ↓e-ldm-trainingPDF p. 10, Appendix C, Latent Dynamics Model training paragraphs
A stricter roughly 1500-hour subset trains LDM. Motion alignment follows warmup and uses EgoVerse, EgoDex, Xperience and Agibot-World labels.
Go to primary source ↓e-bridgePDF p. 4, Figure 3 caption and World Dynamics Action Model; p. 5, opening continuation
Separate expert weights interact through shared attention with video-to-dynamics-to-action visibility. Clean queries predict dynamics; inference denoises video first and subsequently generates actions without extra dynamics-denoising steps.
Go to primary source ↓e-implementationPDF pp. 10–11, Appendix C, World Dynamics Action Model
WDAM uses Wan2.2 TI2V-5B, shared action-transformer weights with embodiment-specific projections, online LDM targets in Stage I and offline targets thereafter. LDM is absent at deployment. Training explicitly uses an inverse-dynamics protocol with teacher-forced video.
Go to primary source ↓e-stagesPDF p. 5, Eq. (3) and Training Procedure; pp. 11–12, Table 6 and Appendix C curriculum/optimization paragraphs
The three stages use loss weights (1,1,0), (1,0.5,1), (1,0.1,1) and schedules of one, one and five epochs. Stage II initializes or reinitializes action/proprioception modules. T5, VAE and LDM are frozen; training uses 64 H20 GPUs.
Go to primary source ↓e-evalPDF p. 5, Experiment—Setup, Simulation and Real World; p. 12, Appendix D
RoboTwin covers 50 tasks with 100 rollouts per task. Physical evaluation covers five gripper and two dexterous tasks, with 50 demonstrations per task and 30 rollouts per setting.
Go to primary source ↓e-simPDF p. 5, Table 1, Ours, Lingbot-VA and ACE-Ego-0 rows; p. 6, Simulation
Ours reports 93.96 clean, 92.78 randomized and 93.4 average success. Lingbot-VA and ACE-Ego-0 average 92.2 and 90.9. The authors characterize the closely clustered benchmark as a competitiveness check.
Go to primary source ↓e-realPDF p. 6, Table 2, Average column and Ours/Spray Water cell; Real World paragraph
Average success is 70.5 for LD4WAM, 64.3 for Lingbot-VA, 63.3 for π0.5 and 47.1 for Fast-WAM. LD4WAM achieves 46.7 on Spray Water.
Go to primary source ↓e-generalizationPDF p. 5, Figure 4 caption; p. 6, Generalization; p. 7, Table 4
The tested shifts replace objects or alter lighting, texture and clutter. Authors report 88.6% object-performance retention and background success of 44.4 versus π0.5's 54.5, attributing background weakness to generated-video degradation.
Go to primary source ↓e-ablationPDF p. 7, Table 4, architecture rows/Average column and Full Architecture Ablation
The cumulative averages are 40.0, 48.1, 58.9 and 63.7. Replacing the aligned dynamics targets with motion-unaligned or pixel-reconstruction alternatives gives 43.3 and 44.5.
Go to primary source ↓e-probePDF p. 5, Latent Dynamics Regression; p. 12, Appendix E, Data, Regression target and Head and optimization; p. 13, Rate bands continuation
The frozen-latent probe uses 2500 clean and 2500 randomized RoboTwin episodes, non-overlapping eight-frame clips and an 8:2 clip split. RoboTwin is outside LDM training. Targets normalize translation by 0.01 m and rotation by 0.08 rad.
Go to primary source ↓e-probe-resultsPDF p. 12, Table 7, 5 Hz column
Scale-normalized MSE is 1.815 for LDM, 8.497 without motion alignment, 2.205 without multi-stride sampling, 2.260 with pixel reconstruction and 3.236 with hard VQ.
Go to primary source ↓e-retrieval-visualPDF p. 6, Figure 5, three query rows and labeled retrieval panels; pp. 5–6, Latent Dynamics Retrieval
The three rows show forward and backward robot-motion queries and an upward human-hand query. Neighbors from labeled human and robot datasets are selected by cosine similarity in the frozen LDM space. These are selected qualitative examples; no retrieval accuracy or query-population statistic is supplied.
Go to primary source ↓e-sim-taskPDF p. 14, Table 8, Hanging Mug row, Ours and Fast-WAM clean/randomized columns
Hanging Mug success is 37% clean and 49% randomized for LD4WAM, versus 58% and 62% for Fast-WAM. This is a task-specific counterexample to uniform superiority despite the stronger aggregate result.
Go to primary source ↓8.5 Primary sources
LD4WAM: Learning Latent Dynamics from Human Videos for World Action Models ↗
PDF · 8,058 extracted words
Source fingerprint
50e48c4457d54f8d6a160622b752c72e4895613c60351f4a713798021931ffcd