AMPLIFY: Actionless Motion Priors for Robot Learning from Videos
1. Paper overview
In one sentence: A discrete motion reference lets video learning and robot action learning scale separately, but accurate image-plane motion does not by itself establish controllable dynamics. problemarchitecturetask-transferreal-protocol
| At a glance | What to know |
|---|---|
| Research problem | Source description Action-free videos show how tasks unfold but omit robot commands. AMPLIFY asks whether visual motion can become a reusable reference for a robot trained with limited action labels. Goal-associated videos and robot interaction data have different roles; an expert demonstration can supply both. problemdecomposition |
| Core mechanism | Source description The central representation compresses tracked velocities into discrete FSQ tokens, with local displacement classification supplying motion-focused supervision. tokenizer |
| A key reported result | LIBERO transfer without target-task action labels: Long/Object/Spatial/Goal: 0.52/0.80/0.69/0.41; mean 60.5%. Executed-task success rate. Forward model: all-suite videos. Inverse model: LIBERO-90 actions only. Target evaluation uses ten rollouts per task, 20 Hz and a 500-step limit. QueST: 0.07/0.00/0.01/0.01; Diffusion Policy: all 0.00. Strong transfer from target videos and off-task actions. Target tasks are unseen in action training, not unseen in all modalities. task-transferlibero-protocol |
| Reading caution | Source description Two-dimensional tracks can correspond to multiple robot actions. The authors restrict the setting to deterministic dynamics; online exploration and larger foundation backbones remain proposals. limitations |
Core contributions
- Source description
The central representation compresses tracked velocities into discrete FSQ tokens, with local displacement classification supplying motion-focused supervision. tokenizer
- Source description
Separately trained forward and inverse dynamics permit independent use of videos and robot actions. The same motion interface also conditions AVDC video prediction. decompositionvideo
Figure 2. Motion codes form the interface between video prediction and robot commands. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start in panel (a): CoTracker converts a video window into tracks κ_t; differencing produces velocities u_t. The encoder, quantizer h and decoder turn these into distributions over local displacements. The illustrated argmax selects a displacement before tracks are reconstructed. Panel (b) predicts the discrete codes instead: image and language tokens precede SOS, and dashed links feed generated codes into subsequent positions. Panel (c) receives image, proprioception and motion tokens through cross-attention and outputs an action chunk. Sections 2.3–2.5 clarify the training freezes and attention mask; the diagram alone does not show them. The deployed policy follows panels (b) and (c), bypassing reconstructed tracks. architecturetokenizerforwardinversedecomposition
What it supports. The interface represents a future motion reference, not a robot action label. That permits videos to train the forward model and robot interactions to train the inverse model separately. The absence of a direct goal input to the action head is deliberate: task selection enters through the predicted motion.
Where the evidence stops. The forward model is goal-conditioned rather than action-conditioned. It does not evaluate candidate motor commands. Figure 2 is architectural evidence; a plausible motion reference still requires a competent inverse model and does not establish physical success.
2. Motivation
2.1 The problem and the proposed response
Action-free videos show how tasks unfold but omit robot commands. AMPLIFY asks whether visual motion can become a reusable reference for a robot trained with limited action labels. Goal-associated videos and robot interaction data have different roles; an expert demonstration can supply both. problemdecomposition
2.2 What this reading follows
Imagine watching a person open a box without knowing any robot commands. AMPLIFY first learns to describe such motion through compressed point trajectories. A separate model predicts that description from the current image and task, while an inverse model learns how a robot can follow it. The separation matters because videos and action-labeled demonstrations can train different components. This reading follows the interface from local displacement classes to executed actions, then tests its value against prediction, transfer and ablation evidence. The strongest task-transfer result still includes target-task videos, and the real-world evidence comes from three tasks with a small evaluation budget. problemarchitecturetask-transferreal-protocol
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The supplied taxonomy is wholly unassigned, so no existing quadrant can be confirmed. Architecture evidence supports separate forward and inverse models linked by latent future-motion tokens. This is inverse-dynamics control with inference-time motion prediction, not one network jointly predicting world states and actions, nor merely an auxiliary training loss. architectureforwardinverse
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn motion into a vocabulary of local alternatives
The tokenizer begins with an intentionally ordinary spatial interface: a fresh uniform grid in the current frame. CoTracker supplies future positions during preprocessing, and differencing expresses them as velocities. Reinitializing the grid means that prediction always starts from known points instead of relying on points selected using future motion. The encoder compresses this sequence and FSQ discretizes it. The decoder then answers a local classification question for every point and step: which displacement in the window occurred? Equation (1) trains those categorical predictions against observed-track labels. This makes motion, rather than rendered appearance, the reconstruction target. The authors argue that local classification better accommodates multiple possible movements than coordinate regression. Table 11 supports the reconstruction advantage, while its combined output-and-loss change leaves that proposed explanation only partly isolated. preprocessingtokenizertokenizer-ablation
5.2 Give video learning and action learning different jobs
Once the tokenizer is trained, the forward model learns which motion codes should follow an image and instruction. Image and language conditioning are available throughout its prefix; motion codes are generated causally. The inverse model solves a different problem: given current robot state and a motion reference, which actions should realize it? It receives no instruction directly and is trained with robot interactions. Both upstream components remain frozen during inverse training, and the authors fine-tune the action decoder using predicted references. This distinction explains the task-transfer protocol. Target-task videos teach the forward model what to predict, while LIBERO-90 action data teaches the inverse model how the robot can move. A successful target rollout therefore demonstrates transfer of an execution mapping, with target-task observation supervision still present. forwardinversedecompositiontask-transfer
5.3 Judge the interface by what the robot completes
A reader's interpretation of Tables 12 and 13 is that the easiest future to predict may be too short to specify a useful action. The paper reports progressively worse pixel accuracy at longer horizons but better task success. That pattern motivates evaluating reference utility separately from reconstruction quality. It does not prove that additional motion context alone caused the gain, because action horizon and temporal aggregation can also matter. The same separation is useful elsewhere: CoTracker agreement measures a learned visual target, the UR5 tables measure physical completion, and AVDC scores measure generated imagery. None is interchangeable with the others. Even the auxiliary video result needs caution: Table 6 increases LPIPS while Appendix D.3 defines it as a feature distance. The broad claim of improvement on every video metric therefore remains unresolved. horizon-ablationinversemetricshuman-transfervideo
5.4 Training and inference
During training
Train the tokenizer by reconstruction, freeze it for forward-model cross-entropy, then freeze both upstream modules for inverse learning. The inverse decoder can use observed-motion codes, but the authors fine-tune it on predicted codes to accommodate prediction errors. tokenizerforwardinverse
LIBERO training uses approximately 50k/25k/75k accumulated steps for tokenizer/forward/inverse modules. Reported training uses one RTX 6000 or L40S, AdamW at 1e−4 and effective batch size 256; module sizes are 31M/70M/57M parameters. training
During inference
At each timestep, predict a fresh motion sequence from the current observation and task, decode an action chunk with current proprioception, and temporally ensemble overlapping chunks using the same discount as training. This provides observation feedback without online trajectory search or full video synthesis in the robot-control path. inversearchitecture
5.5 Implementation flow
- Build motion targets
Initialize 400 grid points in each frame, track each window with CoTracker, normalize coordinates and difference them into velocities. The default horizon is 16 frames, described as 0.8 seconds. Reinitialization avoids reliance on future-dependent point selection but increases offline tracking cost. preprocessing
- Compress local displacement
A causal transformer encoder and FSQ form a 16-token motion sequence. An unmasked decoder reconstructs 225 displacement classes in a 15×15 local window per point and timestep. Multiple camera views share one code sequence when available. tokenizer
- Predict the reference
A frozen ResNet-18 supplies 49 image tokens per view; a T5 summary supplies language. An autoregressive transformer predicts motion codes with unmasked conditioning and causally masked motion tokens. The target is future motion conditioned on a goal, without a candidate robot action input. forward
- Translate motion into commands
Learned queries cross-attend to image, proprioception and motion tokens. The inverse model receives no goal directly. Its default Gaussian head predicts action chunks, bypassing track reconstruction; real-world comparisons substitute a diffusion head. inversehuman-transfer
6. Experiments & results
AMPLIFY learns a compact vocabulary of visual motion from point tracks, predicts that motion from an image and task instruction, and translates it into robot actions through a separate inverse model. Its strongest evidence concerns scarce target-task action labels, including transfer where target-task videos remain available. Better track prediction and physical task completion are evaluated separately.
6.1 Read the original evidence
Table 2. Prediction improves against the available comparisons, under different conditioning protocols. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read one dataset block at a time. LIBERO reports normalized-coordinate MSE, reported ΔAUC and exact pixel accuracy; only MSE points downward. AMPLIFY's motion codes are decoded back into tracks for this comparison. BridgeData and Something-Something v2 report only ΔAUC here, so the dashes must remain missing measurements. The target trajectories come from CoTracker rather than physical position sensors. Section 3.1 says Track2Act receives a goal image, while Seer generates a video and then applies tracking. Appendix D.3 further states that Track2Act values are taken directly from that work. These distinctions matter when interpreting the rows as evidence about representation quality. predictionmetrics
What it supports. On LIBERO, MSE drops from ATM's 0.022 to 0.006 and pixel accuracy rises from 0.250 to 0.629. AMPLIFY also reports higher ΔAUC in the available BridgeData and human-video comparisons. These results support accurate prediction of tracker-derived motion; they are not measurements of executed robot success.
Where the evidence stops. Appendix D.3 calls ΔAUC normalized to [0,1], but its printed sum lacks a threshold-normalization factor. The table values are retained as reported. Conditioning differences and unspecified exact evaluation trajectory splits also limit strict like-for-like interpretation.
Table 5. Target-task videos can supply task information when target action labels are unavailable. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column as a target suite excluded from inverse-model action training. The forward model still sees observations from every LIBERO suite, including these targets. Inverse dynamics and the listed behavior-cloning baselines receive action data only from LIBERO-90. Thus the bottom row asks whether a motion reference learned from target videos can be executed using action knowledge from other tasks. Compare it both with QueST and with AMPLIFY's without-tracks row. Appendix D.1 supplies the rollout protocol: ten random seeds per target task, actions at 20 Hz and a maximum of 500 steps. The values are success fractions, not prediction accuracies. task-transferlibero-protocoldecomposition
What it supports. AMPLIFY reaches 0.52 on Long, 0.80 on Object, 0.69 on Spatial and 0.41 on Goal, for the reported 60.5% average. The large separation from near-zero baselines supports a reusable inverse mapping when useful target-task motion can already be learned from videos.
Where the evidence stops. Zero-shot here means zero target-task action labels, not absence of target-task information. Behavior-cloning baselines lack a mechanism for using the extra videos. The comparison establishes this combined capability, not an isolated architecture advantage under identical accessible modalities.
Table 17. Physical transfer results depend on the task, action-data budget and success definition. Original paper, p. 28 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the grouped headers before comparing values. Opening the box alone counts as partial success; full success also requires placing the eggplant in the bowl. Cup stacking similarly distinguishes completing one stacking relation from completing the entire task. The five/ten/all columns count robot demonstrations. All means 24 for Place Cube, 13 for Stack Cups and 15 for the box task; the forward model also uses 48, 59 and 30 human demonstrations respectively. Both policies use diffusion heads, and evaluation uses ten rollouts with a 90-second limit. The displayed average includes partial and full metrics, unlike Table 4's full-success-only average. human-transferreal-protocol
What it supports. With all robot demonstrations, AMPLIFY completes Stack Cups in all ten reported trials versus five for Diffusion Policy. Benefits are not universal: five-demo box-opening partial success is 0.1 versus 0.5. Separating these outcomes gives a more informative view than the table's overall 0.59 versus 0.49 average.
Where the evidence stops. The fixed-camera UR5 experiments are small, and 1.0 success is not evidence of reliable deployment beyond those trials. A robot-video-only AMPLIFY control would be needed to isolate the added human videos from the motion-conditioned pipeline.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO transfer without target-task action labels Forward model: all-suite videos. Inverse model: LIBERO-90 actions only. Target evaluation uses ten rollouts per task, 20 Hz and a 500-step limit. | Long/Object/Spatial/Goal: 0.52/0.80/0.69/0.41; mean 60.5%. Executed-task success rate | QueST: 0.07/0.00/0.01/0.01; Diffusion Policy: all 0.00. Strong transfer from target videos and off-task actions. Target tasks are unseen in action training, not unseen in all modalities. task-transferlibero-protocol |
| Future point-track prediction LIBERO, BridgeData v2 and Something-Something v2; uniform initial grids and CoTracker reference tracks. Exact held-out trajectory membership is not specified. | LIBERO: MSE 0.006, pixel accuracy 0.629, ΔAUC 0.913. BridgeData/SSv2 ΔAUC: 0.968/0.725. Normalized-coordinate MSE, pixel accuracy and reported ΔAUC | ATM LIBERO: 0.022/0.250/0.767. Track2Act BridgeData/SSv2: 0.770/0.700; Seer BridgeData: 0.914. Lower prediction error supports the representation; comparison inputs differ and tracker agreement does not measure physical correctness. predictionmetrics |
| LIBERO-Long with two action demonstrations per task All demonstration videos train forward dynamics; two action-labeled demos per task train inverse dynamics. Standard LIBERO rollout protocol. | 0.55 Success rate | ATM 0.16; inverse-only AMPLIFY 0.00. Motion conditioning helps this severe low-action-data setting. Table 16 does not establish uniform superiority across every suite and data budget. few-shotlibero-protocol |
| Full-data LIBERO behavior cloning Forward and inverse models trained using the demonstration dataset; standard LIBERO evaluation. | Long 0.75; Object 0.93; Spatial 0.73. Success rate | BAKU Long 0.86; inverse-only AMPLIFY Object 0.64 and Spatial 0.83. Benefits depend on task and data availability; full-data control is not uniformly improved. in-distributionlibero-protocol |
| UR5 learning with human videos Three tasks; human+robot videos for forward dynamics, 5/10/all robot demos for inverse dynamics. Matched diffusion heads; ten 90-second rollouts per task/setting. | Table 4 average 0.58; full-data Stack Cups 1.0. Full task success | Diffusion Policy average 0.42; full-data Stack Cups 0.5. Physical execution improves on average. Table 17's 0.59/0.49 averages also include partial success and are a different aggregate. human-transferreal-protocol |
| LIBERO-Long representation and horizon ablations Appendix E's component ablations, with standard task evaluation for the inverse model. | Local classification ΔAUC 0.919. At horizons 4/8/16: pixel accuracy 0.757/0.678/0.613; policy success 0.36/0.64/0.75. Tokenizer ΔAUC; forward pixel accuracy; policy success | MSE tokenizer ΔAUC 0.883. Longer references can aid control while becoming harder to predict. Output/loss changes and separate horizon sweeps do not isolate one causal mechanism. tokenizer-ablationhorizon-ablationlibero-protocol |
| BridgeData conditional video prediction AVDC with observed-motion codes during training and predicted codes during generation. | 16.40, 0.19, 0.59 PSNR, LPIPS, SSIM as printed | AVDC alone: 15.93, 0.16, 0.56. PSNR and SSIM increase. Table 6's upward LPIPS arrow conflicts with its distance definition; the claimed improvement on all metrics remains unresolved. video |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 11. Temporal context and local displacement classification help reconstruct motion. Original paper, p. 26 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat each horizontal block as a separate ablation on LIBERO-Long. The attention block tests whether tokenization can exchange information across timesteps: causal and full attention are close, while per-timestep processing is worse. The next block changes the decoder from coordinate regression with MSE to local-window classification. Read that as a joint output-and-objective change, not merely a different scalar loss applied to identical predictions. Lower blocks inspect image conditioning, horizon and compression capacity. Appendix E explains that the chosen sequence contains 16 codes and uses an implicit 2048-entry codebook. The measurements here concern track reconstruction, not the action head's task success. tokenizertokenizer-ablationmetrics
What it supports. Causal attention achieves 0.919 ΔAUC versus 0.877 for per-timestep tokenization; full attention gives 0.918. Local-window classification also reaches 0.919 versus 0.883 for coordinate MSE. These findings support temporally informed motion codes and motivate the local-displacement representation used by the downstream predictor.
Where the evidence stops. The table gives no uncertainty or complete per-row configuration manifest, and default-looking values differ between blocks. Treat the comparisons within their own blocks. The authors' multimodality explanation is a hypothesis, not a separate measurement of captured motion modes.
Tables 12–13. Prediction accuracy and control utility favor different horizons. Original paper, p. 26 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First read the horizon block in Table 12, then the corresponding block in Table 13. Four-, eight- and sixteen-step forward prediction becomes progressively less pixel-accurate, whereas the inverse-policy success rates rise. The tables evaluate different components, so do not interpret the rows as a paired trial-level correlation. Other blocks compare vision encoders, pooling, depth, tuning and action heads; their metric column changes, and their values cannot all be ranked on one scale. Appendix E interprets longer horizons as providing more context for action inference. The default training table lists eight forward layers, although Table 12 highlights four, a detail to resolve in reproduction. horizon-ablationtraininginverse
What it supports. Moving from four to sixteen steps changes forward pixel accuracy from 0.757 to 0.613 but policy success from 0.36 to 0.75. Meanwhile the Gaussian, diffusion and flow heads score 0.74, 0.74 and 0.73. The evidence favors examining reference content before assuming a more elaborate action head will help.
Where the evidence stops. Horizon sweeps can change reference context, action-chunk length and aggregation effects together. They do not isolate which causes the gain. The action-head differences also lack uncertainty estimates, so the table does not establish statistical equivalence.
7. Analysis & limitations
7.1 What the evidence leaves open
Two-dimensional tracks can correspond to multiple robot actions. The authors restrict the setting to deterministic dynamics; online exploration and larger foundation backbones remain proposals. limitations
Track targets inherit CoTracker errors. Reported ΔAUC normalization is internally inconsistent: the formula sums ten thresholds but divides only by horizon despite a stated [0,1] range. Numbers are preserved without silently repairing the metric. metrics
Tables provide no uncertainty across training runs. Real-world trials are small and use fixed cameras. Human-video benefits are not isolated from introducing the motion-conditioned pipeline itself by a matched robot-video-only ablation. real-protocolhuman-transferhorizon-ablation
Video scaling is non-monotonic. Table 14's 50-video/two-action result is 0.55, whereas Table 16's Object/two-demo result is 0.73; their configuration difference is unexplained, so they should not be merged. scaling
7.2 Questions for discussion
- Does useful motion conditioning require target-task videos, or can the prior transfer when those videos are also withheld?
- How much of the horizon benefit comes from motion context versus longer action chunks and temporal ensembling?
8. Reproducibility audit
8.1 Requirements and known gaps
Preserve time normalization: BridgeData uses four frames interpolated to sixteen; human recordings use eight frames. Reproduce task-specific robot/human counts and distinguish partial from full success. The primary text supplies hardware and broad hyperparameters, but not exact trajectory splits, software versions or a complete tokenizer configuration. preprocessingreal-protocoltrainingtokenizer
Resolve implementation ambiguities before comparing results: Eq. (2) omits explicit FSQ quantization in its target notation; Table 10 lists eight forward layers while Table 12 highlights four. Record these choices and metric normalization explicitly, then repeat paired data splits and training seeds. forwardtraininghorizon-ablationmetrics
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Measure dependence on target-task videos and informative tokens
Reader-proposed check, not performed: keep LIBERO-90 action data, inverse architecture, evaluation seeds and training budgets fixed. Compare forward models trained with all-suite videos versus LIBERO-90 videos only, using equal video counts where feasible. Include a matched image/proprioception action baseline and a separately trained control whose motion conditioning is shuffled independently of the target motion. Report per-suite success and prediction accuracy with repeated training seeds. A substantial loss when target videos are removed would bound the observed transfer by target observation supervision; little difference between informative and shuffled conditioning would weaken the proposed motion-interface explanation. Keep shuffling consistent between control training and evaluation to avoid testing only an artificial input-distribution shift. task-transferdecompositioninverselibero-protocol
Check 2: Separate reference horizon from action-chunk horizon
Reader-proposed check, not performed: on LIBERO-Long, cross motion horizons of 4, 8 and 16 steps with independently chosen action-chunk horizons. Train each compatible interface using the same demonstrations, optimizer budget and Gaussian head. Evaluate with fixed control rate and a declared temporal-ensembling rule; repeat once without ensembling as a control. Record common-prefix track accuracy, full-horizon accuracy and task success over repeated training seeds. If longer motion context improves success at a fixed action horizon, the reference-context hypothesis gains support. If gains appear only when action chunks or ensembling change, the existing horizon result primarily reflects control scheduling rather than better motion guidance. horizon-ablationinversetraininglibero-protocol
8.3 Reading coverage
Visual audit: Personally viewed the title/byline, all method and result pages, every appendix scientific page, all Figures 1–11 and Tables 1–17, and each of the six final original crops. Figure 2's tracking, displacement, autoregressive and action branches were checked against Sections 2.2–2.5 and Algorithm 1; attention masking and freezing are explained from the text, not inferred from unmarked graphic elements. The final architecture crop was viewed again after removing excess whitespace. Supporting pages include all numerical, evaluation, hardware, preprocessing and proposed-check evidence. References on pp. 10–16 were read in the complete text chunks but not rendered. The qualitative video/track panels were inspected as static PDF figures, not as continuous videos. No external supplements, project site or code were inspected. Source inconsistencies retained in this bundle include LPIPS direction, ΔAUC normalization, forward-depth reporting, and the differing Object scaling/few-shot values.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, abstract and Sections 1–5 (pp. 1–9)
- Acknowledgments and References (pp. 9–16)
- Appendix A: Notation; B: Three-stage decomposition and Algorithm 1 (pp. 17–18)
- Appendix C.1–C.4: Extended related work (pp. 18–19)
- Appendix D.1–D.5: Tasks, setup, metrics, preprocessing and training (pp. 19–24)
- Appendix E: All ablation discussions and tables (pp. 24–27)
- Appendix F.1–F.3: Scaling, detailed results and qualitative panels (pp. 27–32)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Poppler layout text is faithfully retained without manual repairs. Mathematical symbols, table ordering, figure labels and ligatures require inspection against the original PDF; successful extraction is not scientific reading.
- Separate supplemental material availability has not been fully verified.
- All ten supplied text chunks were read individually. All scientific figures and tables were also visually inspected; references on pp. 10–16 were read as text only.
- This report concerns the supplied arXiv v1, dated 17 June 2025. Title and authors match the catalog; no later revision or venue edition was supplied or compared. AMPLIFY appears in small capitals, which extraction splits as A MPLIFY.
- External project material and code were not inspected; no experiments were reproduced. The v1 acknowledgments contain template text rather than verified funding credits.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, byline, affiliation footnotes and arXiv margin stamp
The title and six authors match the supplied catalog: Jeremy A. Collins, Loránd Cheng, Kunal Aneja, Albert Wilcox, Benjamin Joffe and Animesh Garg. Collins and Cheng share equal contribution. Affiliations are Georgia Tech and Georgia Tech Research Institute. The stamp identifies arXiv:2506.14198v1, 17 Jun 2025.
Go to primary source ↓problemPDF pp. 1–3, Abstract, Section 1 and Section 2 problem setup
The work addresses expensive action-labeled demonstrations by separating motion prediction from action inference. Its inputs are goal-associated videos and robot observations, proprioception and actions.
Go to primary source ↓architecturePDF p. 3, Figure 2 and caption; pp. 4–5, Sections 2.2–2.5
Figure 2 separates FSQ motion tokenization, autoregressive latent prediction and a cross-attention action head. The tokenizer reconstructs local displacements; the policy consumes latent tokens without reconstructing tracks.
Go to primary source ↓preprocessingPDF p. 4, Section 2.1; p. 23, Appendix D.4
CoTracker tracks a newly initialized 20×20 grid for each window. Tracks are normalized to [−1,1] and differenced. The default is 16 frames/0.8 seconds; BridgeData's four frames at 5 Hz are interpolated to 16 by cubic spline, and human recordings use eight frames. Reinitialization costs T times more preprocessing than tracking one grid through a video.
Go to primary source ↓tokenizerPDF p. 4, Section 2.2 and Eq. (1); p. 25, Motion Tokenization; p. 26, Table 11
A causal transformer encoder and FSQ produce discrete motion codes; an unmasked decoder cross-attends from learned point encodings. Cross-entropy supervises local displacement classes. The reported choices include W=15, a 2048-entry implicit codebook and 16 latent tokens; multiview tracks are tokenized together.
Go to primary source ↓forwardPDF pp. 4–5, Section 2.3 and Eq. (2); p. 25, Forward Dynamics
ResNet-18 supplies 49 spatial tokens per image and T5 supplies a task summary. Conditioning tokens are unmasked, motion tokens causal. The tokenizer remains frozen during code prediction; ResNet is also frozen. Equation (2) writes Eθ(u_t) inside stop-gradient although the prose specifies discrete code targets.
Go to primary source ↓inversePDF p. 5, Sections 2.4–2.5 and Eq. (3)
A goal-independent transformer decodes image, proprioception and motion tokens to action chunks. The default Gaussian head uses temporally discounted NLL. In practice the action decoder is fine-tuned on predicted tokens with the tokenizer and forward model frozen. Temporal ensembling combines successive chunks at inference.
Go to primary source ↓decompositionPDF pp. 17–18, Appendix B, Algorithm 1 and Table 8
Goal-directed action-free videos train forward dynamics; robot interaction data trains inverse dynamics. Expert demonstrations can enter both sets. Undirected interaction is an architectural possibility, not a reported online exploration experiment.
Go to primary source ↓predictionPDF pp. 5–6, Section 3.1 and Table 2; p. 27, Table 15
Aggregate LIBERO MSE/ΔAUC/pixel accuracy are 0.006/0.913/0.629 for AMPLIFY and 0.022/0.767/0.250 for ATM. BridgeData ΔAUC is 0.968 versus Track2Act 0.770 and Seer 0.914; Something-Something v2 is 0.725 versus Track2Act 0.700. Track2Act uses goal images; Seer predicts videos before tracking.
Go to primary source ↓metricsPDF pp. 22–23, Appendix D.3, Track Prediction Metrics
CoTracker outputs serve as ground truth and all methods use uniformly spaced initial queries. Track2Act numbers are taken from its paper. MSE uses normalized coordinates; pixel accuracy is exact pixel agreement. ΔAUC is described as normalized threshold accuracy over distances 1–10, but the printed double sum divides only by horizon, leaving threshold normalization unresolved.
Go to primary source ↓libero-protocolPDF p. 19, Appendix D.1
LIBERO contains 130 tasks with 50 demonstrations each. Evaluation uses 128×128 images, normalized axis-angle actions at 20 Hz, at most 500 steps, and ten rollout seeds per task except LIBERO-90, which uses one rollout per task.
Go to primary source ↓in-distributionPDF p. 6, Table 3; p. 7, In-Distribution Performance
Full AMPLIFY obtains 0.75/0.88/0.93/0.73/0.92 on Long/90/Object/Spatial/Goal. BAKU reports 0.86/0.90 on Long/90; AMPLIFY inverse-only gives 0.76/0.83/0.64/0.83/0.92. Video pretraining is not uniformly superior with full action data.
Go to primary source ↓few-shotPDF p. 7, Figure 4 and Few-Shot Learning; p. 28, Table 16
Forward dynamics uses all videos while inverse dynamics uses 2, 5 or 10 action demonstrations per task. On Long with two demos, AMPLIFY/ATM/inverse-only give 0.55/0.16/0.00. Table 16 has no ATM LIBERO-90 values; AMPLIFY trails ATM on Goal with ten demos, 0.75 versus 0.77.
Go to primary source ↓task-transferPDF p. 6, Table 1, Generalization row; p. 8, Table 5 and Generalization paragraph
Forward dynamics uses observations from all LIBERO suites, but inverse dynamics and BC baselines use action data only from LIBERO-90. Target Long/Object/Spatial/Goal success is 0.52/0.80/0.69/0.41, averaging 60.5%. QueST gives 0.07/0.00/0.01/0.01 and Diffusion Policy all zeros.
Go to primary source ↓human-transferPDF p. 7, Cross-Embodiment Transfer and Table 4; p. 28, Table 17
Human plus robot videos train forward dynamics, and robot actions train inverse dynamics. Both AMPLIFY and the baseline use a diffusion action head. Table 4's full-success average is 0.58 versus 0.42; Table 17 adds partial metrics and reports 0.59 versus 0.49. Full-data Stack Cups success is 1.0 versus 0.5; five-demo box-opening partial success is 0.1 versus 0.5.
Go to primary source ↓real-protocolPDF pp. 19–22, Appendix D.2, Figures 6–7 and Table 9
The UR5 uses three static RGB cameras, no wrist camera, 60 Hz images downsampled to 20 Hz and resized to 224×224; outputs are absolute normalized end-effector positions. Place Cube/Stack Cups/Open Box & Place Eggplant have 24/13/15 robot and 48/59/30 human demos. Each task evaluation uses ten rollouts, with a 90-second limit and separate partial/full criteria.
Go to primary source ↓videoPDF p. 8, Section 3.3 and Table 6; p. 23, Appendix D.3; p. 24, Video Prediction
BridgeData AVDC+AMPLIFY reports PSNR/LPIPS/SSIM 16.40/0.19/0.59 against 15.93/0.16/0.56. Table 6 marks LPIPS upward, but D.3 defines a squared feature distance. Motion and text tokens are concatenated before AVDC's Perceiver resampler; training uses tokenizer codes and inference refreshes predicted codes every T steps.
Go to primary source ↓trainingPDF p. 24, Appendix D.5 and Table 10
Tokenizer/forward/inverse modules have 31M/70M/57M parameters and LIBERO uses approximately 50k/25k/75k gradient steps. Each accumulated step spans four backward passes. Training uses one RTX 6000 or L40S, AdamW at 1e−4, effective batch size 256, hidden width 768 and action discount 0.99. Table 10 gives eight forward layers.
Go to primary source ↓tokenizer-ablationPDF p. 24, Appendix E setup; p. 25, Motion Tokenization; p. 26, Table 11
Ablations use LIBERO-Long. Causal/per-timestep/full attention reconstruction ΔAUC is 0.919/0.877/0.918. Local-window classification gives 0.919 versus MSE 0.883. The change modifies output representation and loss together; no uncertainty is reported.
Go to primary source ↓horizon-ablationPDF pp. 25–27, Appendix E; p. 26, Tables 12–13
At horizons 4/8/16, forward pixel accuracy is 0.757/0.678/0.613 and inverse-policy success is 0.36/0.64/0.75. Gaussian/diffusion/flow heads yield 0.74/0.74/0.73. These are separate factor sweeps, without error bars. Table 12 bolds four forward layers; Table 10 lists eight as the default.
Go to primary source ↓scalingPDF p. 27, Appendix F.1 and Table 14; p. 28, Table 16
With two action trajectories in the LIBERO-Object scaling experiment, success at 0/2/5/10/50 videos is 0.00/0.12/0.34/0.23/0.55. The authors explicitly caution about non-monotonicity and insufficient investigation. Table 16 separately reports 0.73 for Object with two action demonstrations; the difference from Table 14 is not explained.
Go to primary source ↓limitationsPDF p. 9, Section 5
Authors identify ambiguity from 2D tracks and restriction to deterministic dynamics. Training inverse dynamics from online exploration and using a larger VLM/video backbone are future work.
Go to primary source ↓qualitativePDF p. 5, Figure 3; p. 28, Appendix F.3; pp. 29–32, Figures 8–11
Qualitative panels show robot/human/BridgeData track predictions and AVDC videos. Yellow denotes current points and red future points; stationary points are suppressed. Video panels have no paired baseline or ground-truth comparison labels.
Go to primary source ↓8.5 Primary sources
AMPLIFY: Actionless Motion Priors for Robot Learning from Videos ↗
PDF · 14,609 extracted words
Source fingerprint
d6c5e8c54cbaf2fb25cc8ce6b5e4a564ff9ee883f5e9eab4e11895eb8870fc60