DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control
1. Paper overview
In one sentence: DynaMo uses compact latent transitions and future-embedding prediction to pretrain useful robot vision features, while action selection remains the job of a separately trained policy. e-motivatione-neighborse-flowe-eval-headse-sim-resultse-initializatione-ablation
| At a glance | What to know |
|---|---|
| Research problem | Source description Small robot demonstration datasets must support perception and control simultaneously. Frame-based self-supervision can preserve the visually dominant arm while overlooking small manipulated objects; pretrained visual features may also mismatch the robot's viewpoint or embodiment. DynaMo asks whether ordered in-domain observations supply a better visual learning signal before action-supervised policy training. e-motivatione-neighbors |
| Core mechanism | Source description A dynamics-based objective jointly learns an image encoder and two auxiliary dynamics models without action labels, image augmentations or contrastive sampling during visual pretraining. e-flowe-objective |
| A key reported result | Block Pushing: 0.65 Mean blocks in targets; maximum 2. 1,000 scripted demonstrations; frozen encoder plus VQ-BeT; 100 simulated rollouts. RPT 0.52; ImageNet 0.12. Best reported table entry; a gain of 0.13 blocks over RPT. This is not a success percentage. e-sim-protocole-eval-headse-sim-results |
| Reading caution | Reader analysis The authors leave larger-scale unlabeled training and manipulation outside laboratory settings untested. In-domain representation gains do not establish cross-embodiment or broad deployment generalization. e-limitationse-sim-protocole-real-protocol |
Core contributions
- Source description
A dynamics-based objective jointly learns an image encoder and two auxiliary dynamics models without action labels, image augmentations or contrastive sampling during visual pretraining. e-flowe-objective
- Source description
Frozen-feature evaluations cover four simulation benchmarks, eight physical tasks, several policy classes, initialization variants and component ablations. These test downstream control utility rather than image-generation quality. e-sim-protocole-eval-headse-real-resultse-policye-ablation
Figure 3. DynaMo learns the encoder by explaining observed transitions in latent space. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left half downward: observations become blue embedding tokens, then the pink inverse model produces transition latents. On the right, gray bands mark concatenation of an observation embedding with its inferred transition latent. The green model predicts the next embedding, compared with the starred target below. Both dynamics blocks are labeled causally masked. Section 3.1 specifies forward inputs only through the penultimate observation; the faded final observation is not an additional forward input. Figure 1 and Equation (1) clarify that target embeddings use stop-gradient or a momentum encoder. The deployed policy is a later stage, shown separately in Figure 1. e-flowe-objectivee-eval-heads
What it supports. The learning signal asks visual features to support transition prediction. The inverse latent supplies information about which observed transition occurred, while its small dimension restricts copying the entire next embedding. This explains the training architecture; downstream experiments measure the usefulness of the encoder after pretraining.
Where the evidence stops. The diagram does not enumerate token-level attention masks. Inferring a latent from an observed transition during training should not be read as predicting an executable action from a live image. The paper evaluates a separate control policy.
2. Motivation
2.1 The problem and the proposed response
Small robot demonstration datasets must support perception and control simultaneously. Frame-based self-supervision can preserve the visually dominant arm while overlooking small manipulated objects; pretrained visual features may also mismatch the robot's viewpoint or embodiment. DynaMo asks whether ordered in-domain observations supply a better visual learning signal before action-supervised policy training. e-motivatione-neighbors
2.2 What this reading follows
A visual encoder can recognize the robot arm yet miss the small block that determines whether a push succeeds. DynaMo uses demonstration order to address this mismatch: infer a compact description of the observed transition, then predict the next image embedding from it. This illustrated reading follows that training signal into frozen-feature policy evaluations, physical trials and component ablations. The evidence supports useful in-domain representations across several control settings, with important exceptions. Some headline comparisons are ties, ImageNet initialization helps only some tasks, and removing a component can affect environments very differently. Dynamics prediction here supplies visual supervision; the evaluated policy does not plan through the learned dynamics. e-motivatione-neighborse-flowe-eval-headse-sim-resultse-initializatione-ablation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is Related resources. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The recorded taxonomy is entirely unassigned, so there is no positive catalog judgment to support or contradict. Architecturally, this is auxiliary latent inverse/forward dynamics pretraining followed by a separate action policy. Jointly optimizing those modules does not establish a single model that jointly generates future observations and executable actions, nor inference-time world-model planning. e-flowe-eval-headse-action-labels
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why the next observation needs a transition latent
Consider two demonstrations that reach similar images but then move the object differently. Section 3.1 argues that predicting the next embedding from the current embedding alone can be multimodal. DynaMo therefore infers a transition latent from observed consecutive embeddings before making the forward prediction. This does not mean the pretraining model chooses which action to execute. It describes the transition already present in the demonstration sequence. Reader interpretation: the latent gives the predictor a way to account for variation between observed outcomes, while prediction pressure encourages the encoder to retain features needed to explain those outcomes. Figure 2's object-sensitive neighbors are consistent with this interpretation, but neither that example nor the architecture proves that the latent recovers uniquely identifiable physical actions. e-flowe-neighborse-motivation
5.2 Two different shortcuts require two different safeguards
The source identifies two ways the prediction objective can become uninformative. A high-dimensional transition latent could carry the entire next-frame embedding, reducing the need to model a meaningful transition. DynaMo restricts that channel by choosing a much smaller latent dimension. Separately, the encoder could produce constant embeddings, making prediction easy without representing the scene. Targets therefore use stopped gradients or a momentum encoder, with covariance regularization added to the loss. These mechanisms occupy different parts of the architecture and should not be conflated. Table 6 shows especially poor downstream performance without target stabilization, while removing the bottleneck is most damaging on Push-T. Covariance removal also lowers all four reported scores, even though the text characterizes its effect as relatively slight. e-flowe-objectivee-ablation
5.3 What passes from self-supervision into robot execution
After pretraining, the transferred object is the visual encoder. The principal evaluation freezes its embeddings and learns an imitation policy using action-labeled demonstrations. This separation is essential to interpreting the phrase action-free: it describes the pretraining objective, not the complete route to control. VQ-BeT supplies the main simulated action policy, xArm uses goal-conditioned BAKU with a VQ-BeT head, and Allegro uses nearest neighbors. The physical systems also operate at different rates: four hertz for Allegro and ten hertz for xArm. Table 4 checks several policy heads on Push-T and finds a Diffusion Policy tie with ImageNet. Thus representation quality and policy capability interact; the experiment does not establish that DynaMo's latent dynamics can itself execute actions or support online planning. e-motivatione-flowe-eval-headse-real-protocole-policy
5.4 Training and inference
During training
Train all three modules with the dynamics and covariance losses. Stop-gradient targets or an exponential-moving-average encoder counter collapse. Random-initialized training uses AdamW, 40 epochs, batch size 64 and learning rate 10^-4; ImageNet fine-tuning uses 10^-5. Kitchen uses two-frame context; the other tabulated environments use five. e-objectivee-hyperparameters
Freeze the resulting encoder for downstream policy evaluation. Allegro uses ImageNet initialization even though DynaMo can pretrain from scratch elsewhere. The action-label variants in Table 7 provide no improvement on the reported tasks; this does not remove the action labels used by downstream imitation. e-eval-headse-action-labels
During inference
Encode live observations and pass the features to the downstream policy: VQ-BeT for the main simulated comparisons, goal-conditioned BAKU with a VQ-BeT action head for xArm, and kNN for Allegro. Executed actions change the environment before subsequent observations. The reported control pipeline does not roll out the learned forward model or optimize plans through it. e-motivatione-flowe-eval-heads
5.5 Implementation flow
- Encode observations
ResNet18 maps each observation to an embedding. Temporal context enters the dynamics models, which are causally masked transformer encoders. Multi-view pretraining averages per-view losses; downstream policies concatenate the views' embeddings. e-flowe-objectivee-eval-heads
- Infer a compact transition
Inverse dynamics infers a transition latent from consecutive embedded observations and their context. The paper assumes this latent is well determined given consecutive frames. Its dimension is much smaller than the observation embedding, restricting trivial storage of the next embedding. e-flow
- Predict the next embedding
Forward dynamics receives current embeddings concatenated with inferred transition latents and predicts one-step future embeddings. The latent conditions an otherwise potentially multimodal transition. This is training with observed sequences; the latent is not a commanded actuator action. e-flowe-motivation
6. Experiments & results
DynaMo uses action-free dynamics pretraining to make visual features useful for imitation learning. An encoder, latent inverse model and forward model jointly learn to predict next-frame embeddings; a separate policy then learns from frozen features and labeled demonstrations. Results favor DynaMo on several manipulation tasks, but include ties and initialization regressions. The contribution is a representation-learning objective, with no demonstrated use of the dynamics models for online planning.
6.1 Read the original evidence
Table 1. Frozen-feature comparisons show two clear table leads, one tie and one lower score. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column with its own metric. Kitchen counts completed tasks out of four; Block Pushing counts blocks in targets out of two; Push-T measures final target coverage; LIBERO measures success rate. These values cannot be averaged as if they shared units. The upper group contains pretrained or random representations, while the lower group contains in-domain self-supervised methods. Section 4.2 uses a frozen encoder and VQ-BeT policy head for these comparisons, with multi-view embeddings concatenated. Underlining marks the best pretrained entry and bold type the best self-supervised entry; neither typography nor the caption establishes statistical significance. e-sim-protocole-eval-headse-sim-resultse-compute
What it supports. DynaMo reaches 0.65 blocks versus RPT's 0.52 and 0.66 Push-T coverage versus MoCo-v3's 0.57. Its Kitchen score, 3.64, is below BYOL's 3.75. On LIBERO, DynaMo and ImageNet both report 0.93, qualifying the prose claim of outperforming all other representations there.
Where the evidence stops. No confidence intervals or seed dispersion accompany these cells. Backbone and pretraining differences also remain: MAE-style methods use ViT-B, while DynaMo uses ResNet18. The comparison supports the reported pipelines, without isolating the objective alone.
Table 2. Physical trials show substantial gains on some tasks alongside task-specific failures and ties. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the numerator as successful trials and the denominator as attempted trials. The upper three rows use an Allegro hand on a Franka arm, a kNN policy and ImageNet-initialized encoders; each task has ten tests. The lower five rows use the xArm kitchen and a goal-conditioned BAKU policy, with five tests per task. Appendix A supplies six, seven and six Allegro demonstrations and sixty-five total xArm demonstrations. Keep these different policy and data regimes separate. For each row, compare the full set of baselines rather than selecting only the weakest entry; the ketchup and water rows are useful checks on any blanket performance claim. e-eval-headse-real-protocole-real-resultse-rollouts
What it supports. DynaMo records 7/10 sponge picks, 5/10 teabag picks and 9/10 microwave openings. It achieves 5/5 on three xArm tasks, but ketchup insertion is 4/5 against two baselines at 5/5, and water retrieval ties MoCo-v3 at 3/5. These are reported physical execution outcomes.
Where the evidence stops. Five or ten trials provide limited precision, and no uncertainty estimates are supplied. Section 4.3's aggregate percentage wording should not replace these taskwise counts. Appendix rollout stills show selected successes and failures, not all trials or continuous videos.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Block Pushing 1,000 scripted demonstrations; frozen encoder plus VQ-BeT; 100 simulated rollouts. | 0.65 Mean blocks in targets; maximum 2 | RPT 0.52; ImageNet 0.12. Best reported table entry; a gain of 0.13 blocks over RPT. This is not a success percentage. e-sim-protocole-eval-headse-sim-results |
| Push-T 206 human demonstrations; frozen encoder plus VQ-BeT; 100 simulated rollouts. | 0.66 Final target coverage; maximum 1 | MoCo-v3 0.57; RPT 0.56. Higher reported coverage, without supplied uncertainty or seed dispersion. e-sim-protocole-eval-headse-sim-results |
| Franka Kitchen 566 demonstrations; frozen encoder plus VQ-BeT; 100 simulated rollouts. | 3.64 Mean completed tasks; maximum 4 | BYOL 3.75; ImageNet 3.01. DynaMo is below the strongest listed SSL baseline. e-sim-protocole-eval-headse-sim-results |
| LIBERO Goal 500 demonstrations, 50 per goal; ten evaluation rollouts per goal across ten goals. | 0.93 Average success rate | ImageNet 0.93; VC-1 0.91. Ties ImageNet at the reported precision; a strict win is unsupported. e-sim-protocole-eval-headse-sim-results |
| Allegro manipulation ImageNet-initialized SSL encoders and kNN; 6/7/6 demonstrations for sponge/tea/microwave; ten trials per task. | Sponge 7/10; tea 5/10; microwave 9/10. Successes / trials | Best Table 2 baseline per task: 5/10, 2/10 and 3/10. Physical execution evidence under task-specific 120/240/240-second success deadlines, with small trial counts. e-eval-headse-real-protocole-real-results |
| xArm Kitchen manipulation 65 demonstrations across five tasks; goal-conditioned BAKU; five trials per task. | Put yogurt 5/5; get yogurt 5/5; put ketchup 4/5; get tea 5/5; get water 3/5. Successes / trials | Put ketchup reaches 5/5 with BYOL and MoCo-v3; get water ties MoCo-v3. Several gains coexist with a loss and a tie; do not collapse them into universal superiority. e-eval-headse-real-protocole-real-results |
| Push-T across policy classes Same benchmark; compare representations within each policy column in Table 4. | VQ-BeT 0.66; Diffusion 0.73; chunking MLP 0.35; kNN 0.12. Final target coverage; maximum 1 | Strongest other entries: 0.57, 0.73, 0.30 and 0.09, respectively. Compatibility extends across heads, but Diffusion ties ImageNet; different heads have different training settings. e-policye-policy-training |
| Dynamics component ablation Table 6, frozen-feature downstream evaluation; full model normalized to 100%. | No stop-gradient: 1/5/9/0%; no bottleneck: 92/22/9/75%, in Kitchen/Block/Push-T/LIBERO order. Performance relative to full architecture | Full model: 100% for every environment. Target stabilization and bottleneck matter. The separate no-forward ablation also changes the model input, so it does not isolate only prediction direction. e-ablation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 2. A local diagnostic asks which scene details remain close in feature space. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the query column and its legend. Blue marks identify the end effector, red and green marks the blocks, and pale marks the retrieved neighbors. The top row overlays image-space matches; the bottom row makes positional differences easier to separate from the large robot silhouette. Read across the methods while tracking the red and green block locations, rather than judging only whether the arm looks similar. Section 3 says this example uses twenty nearest neighbors of a demonstration frame. DynaMo's matches cluster nearer the query's relevant object configuration, whereas several comparison columns spread the block positions over the workspace. e-neighborse-sim-results
What it supports. This example makes the representation-learning motivation concrete: matching the dominant arm can coexist with poor matching of the manipulated objects. DynaMo's embedding retrieves transitions' relevant scene features more faithfully in the displayed query, consistent with its stronger downstream Block Pushing score.
Where the evidence stops. This is one qualitative retrieval example, not an aggregate localization benchmark or a controlled causal attribution. The paper supplies no error distribution over queries here, so the picture cannot quantify general feature accuracy.
Table 5. Adding pretrained weights produces a task-dependent tradeoff. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the bottom two rows first to examine the two DynaMo training recipes. Then compare each with the plain ImageNet row to distinguish in-domain adaptation from using pretrained features unchanged. Kitchen and Block Pushing improve with the ImageNet-fine-tuned recipe. Push-T and LIBERO instead favor random-initialized DynaMo. Preserve the metric denominators in the header: the columns still measure different outcomes. Tables 8–11 add a crucial qualification: the recipes use different learning rates, and some target-encoder or dropout settings change by environment. Consequently, this table compares complete recipes rather than a strictly controlled swap of initialization alone. e-initializatione-sim-protocole-hyperparameters
What it supports. Fine-tuned DynaMo reports 3.82 Kitchen tasks and 0.67 blocks, compared with 3.64 and 0.65 from scratch. Its Push-T coverage drops from 0.66 to 0.50 and LIBERO success from 0.93 to 0.90. Pretrained initialization is compatible with DynaMo, but the table does not show a uniform benefit.
Where the evidence stops. The caption's positive summary applies to some columns. Differences in learning rate and other settings confound a causal claim about initialization itself. No seed-level uncertainty is given to establish the stability of the smaller differences.
Table 6. Relative scores expose collapse prevention and bottleneck sensitivity. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The retained caption defines the scale: each environment's full model is 100 percent. A cell is a fraction of that reference score, not an absolute task success rate. Read a row across environments to test whether a component matters consistently, then read a column downward to see which changes hurt that task. No stop-gradient removes target stabilization; no bottleneck sets the transition dimension equal to the embedding dimension. Section 4.6 explains that no forward replaces the future target with a same-step target and removes the observation embedding from the predictor input. The short-context row compares the longer setting with two frames where applicable. e-ablatione-objectivee-hyperparameters
What it supports. Removing target stabilization leaves only 1%, 5%, 9% and 0% of full performance across Kitchen, Block, Push-T and LIBERO. Removing the bottleneck leaves Push-T at 9%. In contrast, removing inverse dynamics leaves Push-T at 97%, so component importance is strongly task dependent.
Where the evidence stops. The no-forward row changes both target timing and predictor input; it cannot isolate a single causal factor. Rounded normalized scores and absent seed distributions also prevent reconstruction of precise raw scores or reliable effect uncertainty.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors leave larger-scale unlabeled training and manipulation outside laboratory settings untested. In-domain representation gains do not establish cross-embodiment or broad deployment generalization. e-limitationse-sim-protocole-real-protocol
Reported tables lack confidence intervals and training-seed variability; real tasks have only five or ten trials. Comparisons also mix ResNet18 and ViT-B backbones and different pretraining budgets. Statistical significance and objective-only attribution remain unresolved. e-sim-resultse-real-resultse-eval-headse-compute
ImageNet-fine-tuned DynaMo improves Kitchen/Block to 3.82/0.67 but reduces Push-T/LIBERO to 0.50/0.90 versus 0.66/0.93 from scratch. The initialization comparison also changes training settings, so it is not a pure initialization intervention. e-initialization
7.2 Questions for discussion
- Would transition bottlenecks still help when varied lighting or camera motion explains more visual change than the robot's action?
- Would a controlled initialization-by-learning-rate experiment retain the mixed gains in Table 5?
8. Reproducibility audit
8.1 Requirements and known gaps
A focused Push-T reproduction needs the 206 demonstrations rendered at 224×224, an eight-dimensional transition latent, five-frame pretraining context, SimSiam targets and zero forward dropout. Use the tabulated optimizer and cosine schedule with five warmup epochs. Reported pretraining cost is one hour on one NVIDIA A100. e-sim-protocole-hyperparameterse-compute
Budget separately for the frozen-feature policy: Push-T VQ-BeT uses 5,000 epochs, batch 512, five-frame observation and prediction windows, and learning rate 5.5×10^-5; the reported cost is seven hours on one NVIDIA A100. Allegro and xArm deployment rates are 4 Hz and 10 Hz, respectively. e-policy-traininge-real-protocol
Resolve exact dynamics-transformer dimensions, token-mask implementation, dataset partitions, seed counts and pinned software versions before claiming replication. The PDF gives substantial settings but not a complete executable specification; its release promise is not evidence of a working public package. e-flowe-computee-release
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether the latent bottleneck preserves useful transitions
Reader-proposed, not performed: on Push-T, compare the reported eight-dimensional latent with a 32-dimensional latent and one as wide as the observation embedding. Keep the same demonstrations, five-frame context, encoder, target stabilization, optimizer schedule and downstream VQ-BeT settings. Use five paired training seeds and the same 100 evaluation initializations per seed. Record dynamics loss, embedding variance and final coverage; add a control that shuffles transition latents across samples before forward prediction. The mechanism is supported if compact, correctly paired latents improve downstream coverage despite wider latents fitting training predictions well. Equal or better wide-latent performance, or no effect of shuffling, would weaken the proposed explanation. e-flowe-ablatione-sim-protocole-hyperparameterse-policy-training
Check 2: Separate initialization from the changed training recipe
Reader-proposed, not performed: repeat Push-T as a two-by-two comparison of random versus ImageNet initialization and learning rates 10^-4 versus 10^-5. Hold the SimSiam target scheme, zero forward dropout, latent dimension, data, epochs and policy training fixed. Evaluate five paired seeds with identical rollout initializations and report both coverage and uncertainty. Table 5's 0.66 versus 0.50 difference motivates the test but does not predict its result. If ImageNet remains worse at both matched learning rates, initialization is a plausible contributor; if the gap disappears or reverses, optimization settings explain at least part of the original recipe comparison. e-initializatione-sim-protocole-hyperparameterse-policy-training
8.3 Reading coverage
Visual audit: Personally inspected the title/byline/version page, Figures 1–7, Tables 1–15, Equation (1), method and evaluation text, and all appendix pages supporting implementation, compute and proposed checks. Inspected every final original crop. Figure 3's input directions and causal labels were cross-checked against Section 3.1; target stabilization was checked against Figure 1 and Equation (1). Full-paper text, including bibliography pages 11–16, was read in all seven supplied chunks; bibliography pages were not visually inspected. Rollouts were read as PDF still sequences only. External videos, code and separately hosted supplements remain uninspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 17, 18, 19, 20, 21, 22, 23. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and 1 Introduction (pp. 1–3)
- 2.1 Visual imitation learning and 2.2 Visual pretraining for policy learning (p. 3)
- 3 DynaMo and 3.1 Dynamics as a visual self-supervised learning objective (pp. 3–5)
- 4.1 Environments and datasets; 4.2 frozen-representation evaluation; 4.3 real robotics; 4.4 policy classes; 4.5 initialization; 4.6 ablations; 4.7 action-label variants (pp. 5–9)
- 5 Related works and 6 Discussion and Limitations (pp. 9–10)
- Acknowledgements and References (pp. 11–16)
- Appendix A.1–A.6: all environments and datasets (pp. 17–18)
- Appendix B.1–B.2: encoder and downstream policy implementation details (pp. 19–21)
- Appendix C: real robot rollout figures (pp. 22–23)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Poppler layout text is faithfully retained without manual repairs. Mathematical symbols, table ordering, figure labels and ligatures require inspection against the original PDF; successful extraction is not scientific reading.
- Separate supplemental material availability has not been fully verified.
- All seven supplied text chunks were read without sampling. Original PDF pages 1–10 and 17–23 were visually inspected, including every figure and table; bibliography pages 11–16 were read as text.
- Version scope: the inspected title page is arXiv:2409.12192v2, 30 October 2024, with a NeurIPS 2024 footer. Title and all five authors match the catalog. The catalog submission date is 18 September 2024; the earlier edition was not supplied, so revision contents were not compared.
- The supplied extraction warnings are preserved above. Equation layout, table ordering and retained graphic details were checked against PDF images.
- No code, external project pages, full robot videos or separately hosted supplements were inspected, and no experiments were reproduced. References were read as this paper's bibliography, not independently verified sources.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, byline, affiliation, arXiv margin stamp and venue footer
The title and five authors match the catalog. The PDF lists New York University, arXiv:2409.12192v2 dated 30 October 2024, and NeurIPS 2024.
Go to primary source ↓e-motivationPDF pp. 1–3, Abstract, Introduction and Sections 2.1–3
DynaMo learns visual representations from sequential in-domain demonstrations without action labels during pretraining; downstream imitation uses observation/action demonstrations. The paper motivates dynamics supervision by task-relevant features missed by frame-based objectives.
Go to primary source ↓e-neighborsPDF p. 3, Section 3, nearest-neighbor example; p. 4, Figure 2 and caption
A query frame and its 20 nearest neighbors compare DynaMo, BYOL, MoCo and TCN in Block Pushing. Pixel overlays and top-down locations illustrate better preservation of task objects in this example.
Go to primary source ↓e-flowPDF p. 2, Figure 1(a–b); p. 4, Figure 3 and Section 3.1; p. 5, final paragraph before Section 4
Individual frames are encoded by ResNet18. Causally masked transformer inverse dynamics infers transition latents; forward dynamics predicts next embeddings from embeddings and latents. The latent dimension is much smaller than the embedding dimension. Only the encoder is retained for downstream control.
Go to primary source ↓e-objectivePDF p. 5, Section 3.1, Equation (1) and following paragraphs
The objective adds cosine-distance dynamics prediction and off-diagonal covariance regularization with coefficient 0.04. Targets use stop-gradient embeddings or a momentum encoder. Multiple-view losses are averaged.
Go to primary source ↓e-sim-protocolPDF pp. 5–6, Section 4.1(a–d); p. 17, Appendix A.1–A.4
Franka Kitchen uses 566 demonstrations and mean tasks completed out of four; Block Pushing uses 1,000 scripted trajectories and mean blocks in targets out of two; Push-T uses 206 human demonstrations and final target coverage. Each has 100 evaluation rollouts. LIBERO Goal has 500 demonstrations across ten tasks and 100 total rollouts, ten per goal, reporting success rate. Simulated observations are 224×224 RGB.
Go to primary source ↓e-eval-headsPDF p. 6, Section 4.2; p. 7, Sections 4.3–4.4
Evaluation trains policies on frozen embeddings. Main simulated results use VQ-BeT; xArm uses goal-conditioned BAKU with a VQ-BeT action head; Allegro uses kNN and ImageNet-initialized encoders for all compared SSL methods. Multi-view embeddings are concatenated. MAE-style baselines use ViT-B; other baselines and DynaMo use ResNet18.
Go to primary source ↓e-sim-resultsPDF p. 7, Table 1, all method rows and benchmark columns
DynaMo scores 3.64, 0.65, 0.66 and 0.93 on Kitchen, Block Pushing, Push-T and LIBERO. Relevant comparators are BYOL 3.75 on Kitchen, RPT 0.52 on Block Pushing, MoCo-v3 0.57 on Push-T and ImageNet 0.93 on LIBERO. No uncertainty columns are supplied.
Go to primary source ↓e-real-protocolPDF p. 6, Section 4.1(e–f); p. 17, Appendix A.5–A.6; p. 18, Appendix A.6 continuation
Allegro uses 6 sponge, 7 teabag and 6 microwave demonstrations, with 543, 1,034 and 735 frames. Success deadlines are 120, 240 and 240 seconds. Its policy runs at 4 Hz, with 50 Hz Franka and 60 Hz hand demonstration collection. xArm has 65 demonstrations across five tasks, four camera views, 30 Hz collection and 10 Hz deployment.
Go to primary source ↓e-real-resultsPDF p. 7, Table 2 and Section 4.3; p. 8, Table 3
Table 2 reports DynaMo successes of 7/10, 5/10 and 9/10 for Allegro sponge, tea and microwave; xArm put yogurt, get yogurt, put ketchup, get tea and get water yield 5/5, 5/5, 4/5, 5/5 and 3/5. BYOL, BYOL-T and MoCo-v3 provide taskwise comparisons. Table 3 gives ImageNet and R3M Allegro baselines.
Go to primary source ↓e-policyPDF p. 7, Section 4.4; p. 8, Table 4
Push-T coverage with DynaMo is 0.66 for VQ-BeT, 0.73 for Diffusion Policy, 0.35 for chunking MLP and 0.12 for kNN. The strongest other entries in those columns are 0.57, 0.73, 0.30 and 0.09; Diffusion ties ImageNet.
Go to primary source ↓e-initializationPDF p. 8, Table 5 and Section 4.5; p. 19, Tables 8–11
ImageNet-fine-tuned DynaMo scores Kitchen 3.82, Block 0.67, Push-T 0.50 and LIBERO 0.90, compared with random-initialized DynaMo 3.64, 0.65, 0.66 and 0.93. Fine-tuning also changes the learning rate, and some environment-specific settings differ.
Go to primary source ↓e-ablationPDF p. 8, Section 4.6, forward-dynamics ablation; p. 9, Table 6 and Section 4.6 continuation
Table 6 normalizes full-model performance to 100%. No forward gives 34/8/44/33%; no inverse 72/35/97/41%; no bottleneck 92/22/9/75%; no covariance 94/62/85/59%; no stop-gradient 1/5/9/0%; short context 100/75/88/89%, in Kitchen/Block/Push-T/LIBERO order. No forward changes both target and input; no bottleneck sets latent dimension equal to embedding dimension.
Go to primary source ↓e-action-labelsPDF p. 9, Table 7 and Section 4.7
Supervised inverse dynamics alone reaches 100/54/70/11% of the base model; adding an action-prediction head to full DynaMo reaches 97/29/94/86%, in Kitchen/Block/Push-T/LIBERO order. Neither action-label variant improves these reported scores.
Go to primary source ↓e-limitationsPDF p. 10, Section 6, Discussion and Limitations
The authors propose larger unlabeled datasets as possible future work and explicitly leave effectiveness outside laboratory manipulation settings to further research.
Go to primary source ↓e-hyperparametersPDF p. 19, Appendix B.1, Tables 8–11 and final paragraph; p. 20, first paragraph
Random-initialized DynaMo uses AdamW, learning rate 10^-4, 40 epochs, batch size 64, zero weight decay, betas (0.9, 0.999), gradient clipping 0.1 and covariance weight 0.04. Kitchen uses two frames; other listed environments five. Push-T uses an eight-dimensional latent, SimSiam targets and zero forward dropout. Fine-tuning uses learning rate 10^-5. Schedules and target-encoder choices vary by environment.
Go to primary source ↓e-computePDF p. 20, Appendix B.1, implementation paragraphs, compute list and Table 12
The paper bases its transformers on nanoGPT, links baseline implementations, reports different SSL training budgets, and gives DynaMo training compute. Push-T pretraining takes one hour on one NVIDIA A100. It does not enumerate full dynamics-transformer dimensions or pinned software versions here.
Go to primary source ↓e-policy-trainingPDF p. 21, Appendix B.2, implementation and compute paragraphs, Tables 13–15
Push-T VQ-BeT uses 5,000 epochs, batch size 512, five-frame window, prediction window five and learning rate 5.5×10^-5; training takes seven hours on one NVIDIA A100. Diffusion uses observation/prediction/action horizons 2/10/8. MLP and Diffusion settings are separately tabulated.
Go to primary source ↓e-rolloutsPDF p. 18, Figure 5; p. 22, Figure 6; p. 23, Figure 7
Original image sequences depict the five xArm tasks and selected Allegro/xArm rollouts, explicitly including successful and failed examples. These are still-image sequences, not full videos.
Go to primary source ↓e-releasePDF p. 3, opening paragraph; pp. 19–21, Appendix B
The paper promises public datasets and training/evaluation code and supplies implementation references and hyperparameter tables. It does not itself establish a verified release, installation, complete split/seed specification or reproduced result.
Go to primary source ↓8.5 Primary sources
DynaMo: In-Domain Dynamics Pretraining for Visuo-Motor Control ↗
PDF · 9,454 extracted words
Source fingerprint
c85bd96f89aec8ffad1b4d09d1ee788bc5f2ed1c26574e10eede3c869377205c