A review of learning-based dynamics models for robotic manipulation
1. Paper overview
In one sentence: Choosing what to predict can make robot dynamics easier to learn, but stronger structure shifts difficulty into perception and constrains which interactions the model can represent. scoperepresentations-overviewtraining-control-figurestudy-tabletask-table
| At a glance | What to know |
|---|---|
| Research problem | Source description Manipulation needs predictions of action consequences, yet analytical simulation often depends on physical parameters and full states that are difficult to recover from real sensing. Learning from interaction data can absorb hard-to-model effects, but robot data are expensive and unconstrained state spaces are large. The review asks which structural assumptions retain sufficient task information while improving sample efficiency and generalization. Its scope is environmental dynamics external to the robot; analytical, nonlearned differentiable, and hybrid models are excluded from comprehensive coverage. scopeframeworkphysics-comparison |
| Core mechanism | Source description Organizes the literature into five state-representation families and connects each to perception requirements, dynamics architectures, and deployment costs. Figure 3 presents a qualitative abstraction spectrum rather than an experimentally established ordering. representations-overviewrepresentation-tradeoffs |
| Reading caution | Reader analysis There is no original benchmark, common data split, uncertainty estimate, or controlled ablation. Tables 1–2 summarize different cited systems rather than compare their scores under a shared protocol. Accordingly, no empirical result entries or strongest-result ranking are inferred here. scopestudy-tabletask-tabletask-illustration |
Core contributions
- Source description
Organizes the literature into five state-representation families and connects each to perception requirements, dynamics architectures, and deployment costs. Figure 3 presents a qualitative abstraction spectrum rather than an experimentally established ordering. representations-overviewrepresentation-tradeoffs
- Source description
Separates learning a transition predictor from using it for control. Figure 2 distinguishes training on physical interactions, planning through predicted trajectories, and training policies using model-generated interactions. training-control-figureplanningpolicy-learning
- Reader analysis
Maps sensing and material coverage in Table 1, and representation–model–controller combinations in Table 2. These are useful starting points for choosing a system design, but they cannot establish performance superiority. study-tabletask-table
Figure 1. Different manipulation problems suggest different levels of state abstraction. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right: an initial scene is abstracted into a state, a gray arrow connects it to a predicted change, and a later photograph shows the physical event. Row A retains many local elements to describe dough deformation. Row B tracks a small set of colored points and their structural arrangement during folding. Row C keeps discrete objects and their relationships as a supporting arrangement collapses. Then compare vertically: the source uses these examples to motivate particles, keypoints, and object-centric representations, respectively. The human-prediction column is an explanatory illustration of the chosen abstraction, not a learned model output. human-abstractionsscoperepresentations-overview
What it supports. The figure makes the review's selection problem concrete: a useful state preserves the distinctions needed for the interaction at hand. Dense geometry, sparse structural landmarks, and discrete objects expose different information. This motivates matching representation detail to the task before choosing a dynamics network.
Where the evidence stops. These are illustrative human-intuitive physics examples. The figure provides neither a human-subject experiment nor robot success measurements, and it does not establish that one representation is universally sufficient.
2. Motivation
2.1 The problem and the proposed response
Manipulation needs predictions of action consequences, yet analytical simulation often depends on physical parameters and full states that are difficult to recover from real sensing. Learning from interaction data can absorb hard-to-model effects, but robot data are expensive and unconstrained state spaces are large. The review asks which structural assumptions retain sufficient task information while improving sample efficiency and generalization. Its scope is environmental dynamics external to the robot; analytical, nonlearned differentiable, and hybrid models are excluded from comprehensive coverage. scopeframeworkphysics-comparison
2.2 What this reading follows
A robot shaping dough, folding cloth, or packing objects needs to anticipate how contact changes the scene. This review organizes learned predictors around their state representation: pixels, latent vectors, particles, keypoints, or objects. The central design question is how much structure to impose before learning dynamics, and whether the sensing pipeline can reliably recover that structure. The figures below connect the representation choice to training, planning, and policy learning. The tables map examples from the literature rather than report a common benchmark. Read this as a guide to choosing and evaluating a system, with separate attention to prediction quality, perception reliability, and executed control. scoperepresentations-overviewtraining-control-figurestudy-tabletask-table
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Related resources. View the current classification.
3.1 Evidence-based assessment
Classification assessment not applicable
The recorded foundational survey category fits the source. Architecture, prediction paradigm, and quadrant are not applicable because the article reviews many systems rather than proposing one architecture. Its perception–dynamics–control framework includes planning, inverse dynamics policies, and model-generated policy training; these alternatives cannot collectively establish a One Model or joint-prediction classification. scopetraining-control-figurepolicy-learningtask-table
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Trace the state before tracing the action
The review starts with a perception module because the robot does not directly observe everything that determines contact. Its state is an estimate from observation and action history, not necessarily a physical simulator's full state. Figure 2 then separates two questions. During dynamics learning, the current state and action predict the next state, and the subsequent observation supplies a learning target. During downstream control, a cost evaluates imagined consequences or a policy learns from simulated interactions. This division explains why a low prediction loss is not itself a manipulation policy. A planner still needs a task objective and an action-selection procedure; a learned policy still needs a training signal. The review further distinguishes inverse dynamics, which predicts an action connecting states, from the forward model that predicts consequences of an action. frameworktraining-control-figureplanningpolicy-learning
Figure 2. Learn a transition model from interaction, then use its predictions for planning or policy training. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the upper panel. Observation o₀ is encoded as s₀, which joins action a₀ at the dynamics model. Its predicted state is compared with s₁ encoded from the next observation. Black arrows show forward computation, and red dashes show the learning update through the prediction branch. In the lower-left branch, candidate action sequences produce predicted state trajectories, which the cost function evaluates before selection. In the lower-right branch, a policy interacts with the learned model to collect trajectories for policy updates. These branches distinguish optimizing actions with a model from learning a policy using a model; neither turns the predicted state itself into a motor command. training-control-figureframeworkplanningpolicy-learning
What it supports. The same learned transition model can play different roles at different stages. Planning repeatedly evaluates possible actions; policy optimization uses generated interactions to improve an action-producing policy. This distinction is central when deciding whether a system needs dynamics-model computation during deployment.
Where the evidence stops. The legend calls red dashes gradient backpropagation, but the lower-left arrow is labeled Select. The text separately describes sampling and gradient optimization. Preserve that distinction; this diagram does not make all trajectory selection differentiable or specify encoder freezing.
5.2 Locate the cost of adding structure
Reader interpretation: Figure 3 is most useful as a way to locate where a system pays for structure. Pixels avoid explicit object reconstruction but require learning in a large observation space. A latent objective can discard irrelevant variation, yet reward-based supervision can tie the representation to a particular task. Particles introduce geometry and local interactions, but their input pipeline must cope with occlusion and correspondence. Keypoints compress further by relying on stable task-relevant landmarks. Object-centric states expose relationships among discrete entities while depending on object discovery or segmentation. Thus, moving toward structure does not simply remove complexity; it can transfer complexity to perception and restrict the modeled phenomena. Table 1 helps expose this transfer through the sensors and material settings associated with concrete studies. The resulting design should be assessed end to end. representation-tradeoffspixelslatent-learningparticleskeypointsobject-centricstudy-table
Figure 3. The authors frame representation choice as a tradeoff between prediction structure and state-estimation difficulty. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the arrow from the pixel end toward the object-centric end. The labels associate greater abstraction with a compact state, stronger priors, and more demanding state estimation. Particles occupy an intermediate geometric position, while keypoints retain fewer task-relevant locations. The latent illustration reminds the reader that compact encodings can exist without explicit object or geometric structure. Read the endpoint lists together: the same choice that simplifies dynamics may make it harder to infer a reliable state from images. The surrounding discussion adds task suitability, sensing constraints, interpretability, and computational cost; the drawing compresses these dimensions into a conceptual spectrum. representation-tradeoffslatent-learningparticleskeypointsobject-centric
What it supports. The review's practical claim is that state structure can improve sample efficiency and generalization when its assumptions fit the task and its state can be estimated reliably. The gain is conditional: demanding perception or excluding relevant deformations can offset the advantage of a simpler predictor.
Where the evidence stops. The axis has no measured units, and stronger generalization is an author synthesis rather than a quantified ranking. The picture is not an ablation, and compact latent states prevent reading it as a strict ordering of dimensionality.
5.3 Evaluate the decisions that the model supports
Reader interpretation: the natural evaluation unit is the prediction-and-control pipeline under a specified task and sensing condition. The review warns that video-quality metrics may not track planning performance and that optimization can exploit inaccurate model predictions. A useful reading of Table 2 therefore asks which controller consumed the model, which action space it searched, and whether feedback corrected errors. The review's examples of model predictive control illustrate why replanning and a single imagined rollout are different settings. Meanwhile, the future-directions section argues for uncertainty, hierarchical abstraction, and learned search guidance without demonstrating a universal solution. This makes the survey a map for constructing comparisons, not a source of pooled performance estimates. Its strongest practical lesson is to measure whether imagined action rankings remain useful when those actions are executed. pixelstask-tableplanningpolicy-learningrobotic-tasksrobustnessfuture-control
5.4 Training and inference
During training
Figure 2A uses observations from interaction trajectories to construct current and next-state targets. The dynamics model receives the current state and action; a prediction loss compares its next-state estimate with the state obtained from the subsequent observation. The schematic does not specify a universal loss formula, encoder-freezing rule, optimizer, dataset size, or training schedule. training-control-figure
Pixel prediction is commonly trained by self-supervised maximum likelihood on future frames. For latent models, the review distinguishes reconstruction-based supervision from inverse dynamics, optical flow, contrastive, and reward-based learning. Recurrent state-space models can preserve history and represent uncertainty; deterministic alternatives include MLPs and CNNs. pixelslatent-learninglatent-dynamics
Structured representations can reduce learning cost, but the reported claim that some models train within a few GPU hours is a literature-level observation. It is not a standardized timing comparison with shared hardware, datasets, or preprocessing costs. representation-tradeoffsstudy-table
During inference
The review separates path planning, which searches collision-free state sequences, from trajectory optimization using dynamics. CEM and MPPI sample candidate actions; differentiable models also permit action updates using cost gradients. Online system identification can adapt predictions to uncertain physical properties. planning
An inverse dynamics model can become a goal-conditioned policy by learning the action linking a current state to a desired next state. Model-based reinforcement learning uses imagined transitions to improve a policy. Neither mechanism implies joint future/action prediction in one network, and model use during policy training does not establish that every deployed policy queries the model. policy-learningtraining-control-figure
Feedback matters: the review describes combining latent-space RRT with model predictive control for long-term planning and real-time corrections. It also warns that long rollouts accumulate errors and that policies or planners can exploit inaccurate predictions outside supported data regions. robotic-taskspolicy-learningrobustness
5.5 Implementation flow
- Estimate a decision-relevant state
The perception module summarizes observation and action history into the task-relevant state. The review treats state estimation and representation choice as coupled design problems: a compact state must still retain what determines future interaction outcomes. framework
- Choose the prediction domain
Pixel models predict action-conditioned observations with little explicit perception machinery. Latent models predict compressed encodings instead. Reconstruction retains observation information; reconstruction-free objectives emphasize action, motion, contrastive correspondence, or reward, with different task-dependence tradeoffs. pixelslatent-learning
- Introduce geometric or semantic structure
Particles discretize surfaces or volumes; graph nodes and local interactions support geometric dynamics, while occlusion complicates point tracking. Keypoints retain sparse task-relevant features and require consistent detection. Object-centric states model discrete entities and relations, enabling compositional reasoning but making object discovery or segmentation a bottleneck. particleskeypointsobject-centric
- Connect prediction to decisions
Trajectory optimization evaluates candidate actions through the learned model; policy learning instead uses simulated transitions to train an observation-to-action map. Table 2 shows that the same representation family can support multiple controller choices, so representation alone does not specify the complete control system. planningpolicy-learningtask-table
6. Experiments & results
This review explains learned dynamics for manipulation through the state that a robot chooses to predict. Pixels, latent vectors, particles, keypoints, and object-centric states redistribute difficulty between perception, prediction, and control. The article connects these choices to trajectory optimization and policy learning, then identifies gaps in partial observability, robustness, and scale. It offers a design taxonomy and literature synthesis rather than a new trained model or comparative benchmark.
The review contains four illustrative/conceptual figures and two literature-summary tables, all included here. Figure 2 is a general training/control framework rather than a new architecture; Figure 3 is a qualitative spectrum rather than an ablation. Tables 1–2 contain checkmarks or design labels, not benchmark scores, shared evaluation conditions, or uncertainty. Figure 4 is an attributed montage of cited systems. Consequently, the results visuals document application and design coverage; no original quantitative result table, controlled ablation, or featured empirical result is available from this source. scopetraining-control-figurerepresentation-tradeoffsstudy-tabletask-illustrationtask-table
6.1 Read the original evidence
Table 1. The sensing and material coverage of cited studies is a design inventory, not a scorecard. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Select a representation group, then read a study's row across the sensing, dynamics, and object-type columns. Checkmarks record the review's categorization. Its caption defines 2D and 3D as the space in which rotations, translations, and deformations are modeled, rather than the shape of a neural-network tensor. For example, Ai et al. (20) is marked for RGB, depth, multiview, and tactile sensing, with 3D dynamics and rigid, deformable, and multiobject settings. Manuelli et al. (13) is marked for RGB and depth, 3D dynamics, and rigid objects. Use such rows to identify compatible sensing and task requirements before comparing algorithms. study-tablerepresentation-tradeoffsperception-gaps
What it supports. Representation labels do not by themselves reveal the sensing burden or material coverage of a system. The matrix exposes those additional design choices. A researcher considering a structured predictor should examine the observation pipeline alongside the state abstraction, rather than treating the representation name as a complete specification.
Where the evidence stops. Blank cells are unmarked coverage, not failed trials. No success or runtime values appear. The p. 7 prose says Table 1 includes computational requirements, but the actual table has no compute column; that discrepancy remains unresolved.
Figure 4. The review illustrates applications across rigid, deformable, multiobject, and tool-mediated manipulation. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the time arrow across each row, then use the row labels to separate the manipulation problems. A repositions an object; B changes a rope configuration; C manipulates cloth; D deforms plasticine; E moves an object within a crowded container; F uses tools around dough. The labels below the sequences identify different cited studies. Read this montage alongside Table 2 to connect visible tasks with the review's representation and control categories. Rows are separate examples, so transitions between rows do not form one robot's long-horizon episode. The pictures show selected stages, without a shared duration or common task-success criterion. task-illustrationrobotic-taskstask-table
What it supports. The illustrated application range explains why the review resists a single default state representation. Repositioning, distributed deformation, crowded contact, and tool use place different demands on the information a predictor must preserve. The montage documents the survey's examples, while its accompanying discussion supplies the design context.
Where the evidence stops. Selected photographs cannot establish success frequency, prediction accuracy, robustness, or transfer across tasks. These systems belong to different cited works; the montage is not an evaluation of a single general-purpose model.
Table 2. A task is solved by a representation–dynamics–controller combination, not by representation alone. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Choose a task group and read across all three design columns. In plasticine manipulation, the table lists particle/GNN systems with gradient descent and with a learned policy, alongside a latent/Transformer system using CEM. The tool-use group likewise includes particle/GNN designs with a learned policy, gradient descent, or MPPI. These rows prevent collapsing the world model and controller into one choice. Compare vertically only to identify alternative designs used in the literature; the table does not hold environments, action spaces, or training data fixed. Its learned-policy entries also connect back to the policy-optimization branch of Figure 2 rather than implying trajectory search at every action. task-tabletraining-control-figurepolicy-learning
What it supports. The survey's task mapping supports a modular reading of manipulation systems. A representation shapes what is predicted, the dynamics architecture shapes how prediction is learned, and the controller determines how predictions influence actions. Multiple combinations are documented, without evidence here for a universally strongest combination.
Where the evidence stops. These are the review's descriptions of cited systems, not independently reproduced implementations. There are no shared splits, success metrics, uncertainty estimates, or ablations, so a within-task row comparison cannot isolate a representation or controller effect.
6.2 Results and evaluation conditions
No quantitative results are included in this reading.
6.3 Ablations and diagnostic examples
No ablation or diagnostic visual is included in this edition. The source-specific visual limitations are recorded above.
7. Analysis & limitations
7.1 What the evidence leaves open
There is no original benchmark, common data split, uncertainty estimate, or controlled ablation. Tables 1–2 summarize different cited systems rather than compare their scores under a shared protocol. Accordingly, no empirical result entries or strongest-result ranking are inferred here. scopestudy-tabletask-tabletask-illustration
Visual prediction metrics often fail to correlate with planning performance. Pixel models can hallucinate under partial observability; structured states can fail through occlusion, inconsistent correspondence, or segmentation errors. Better prediction under one representation is therefore insufficient evidence of better executed control. pixelsparticleskeypointsobject-centricrepresentation-tradeoffs
The authors identify multimodal distribution and sensing-frequency mismatches, limited action-labeled data, mostly local tabletop settings, and exploitation of model errors as unresolved barriers. Foundation-model priors, Gaussian representations, hierarchical models, and uncertainty-aware planning are research directions rather than demonstrated solutions in this review. perception-gapsrobustnessfuture-dynamicsfuture-control
A table/prose discrepancy remains: the comparison section says Table 1 includes computational requirements, but its displayed columns contain sensing, dynamics dimensions, and object types only. Compute comparisons should be traced to individual studies, not inferred from this table. representation-tradeoffsstudy-table
7.2 Questions for discussion
- How should a representation be selected when the sensing needed to estimate it costs more than the dynamics prediction?
- Which evaluation would distinguish visually plausible futures from action rankings that improve executed manipulation?
- Can hierarchical representations preserve local contact detail without making long-horizon planning intractable?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproducing a cited system requires that study's interaction data, sensing and state-estimation pipeline, action parameterization, loss, model configuration, controller cost, and evaluation protocol. This review supplies design categories and references, not a shared executable recipe or complete per-study implementation settings. training-control-figurestudy-tabletask-table
Reader-proposed first check: on a fixed manipulation task, compare representation families under equal interaction data and planning budgets, while separately measuring perception error, rollout error, end-to-end latency, and executed task success. Evaluate new geometries and controlled occlusion independently so perception failure is not mistaken for dynamics failure. representation-tradeoffsparticlesperception-gaps
Reader-proposed second check: keep a pixel predictor and planner protocol fixed while comparing model rankings by visual prediction quality against rankings by executed control. Separately test whether increasing planning effort improves predicted costs but worsens realized outcomes, which would expose exploitation of model errors. pixelsplanningrobustness
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate perception failure from dynamics failure in cloth control
Proposed experiment, not performed: compare a particle/GNN and a keypoint/GNN predictor for cloth manipulation, motivated by the alternatives in Table 2. Give both the same recorded RGB-D interactions, action parameterization, task cost, horizon, and planning wall-time budget. Hold out cloth configurations and vary occlusion separately. Measure state-estimation error, rollout error, end-to-end latency, and executed success under a preregistered goal-shape tolerance. Add a reference-state condition that supplies carefully reconstructed states instead of each method's ordinary perception output. If an apparent dynamics advantage disappears only with ordinary perception, sensing is the bottleneck; if it persists under reference states and matched control, representation and prediction deserve closer investigation. particleskeypointsrepresentation-tradeoffstask-tableperception-gaps
Check 2: Test whether better-looking futures lead to better action choices
Proposed experiment, not performed: train several checkpoints of the same action-conditioned pixel predictor on one fixed interaction dataset. On held-out action sequences, rank checkpoints by a fixed visual prediction metric; then run the same planner, candidate-action budget, horizon, and task cost with each checkpoint on matched manipulation starts. Compare that ranking with realized task cost and success, recording the model's predicted cost as well. In a separate controlled sweep, increase planning effort without changing the predictor. A widening predicted-versus-realized cost gap as planning effort grows would support the review's model-exploitation concern. Consistent rankings and matching cost improvements would weaken that concern for the tested distribution, without establishing general robustness. pixelsplanningpolicy-learningrobustness
8.3 Reading coverage
Visual audit: Visually inspected the title and affiliations on p. 1, the formulation and all representation/control discussions on pp. 2–12, Figures 1–4, Tables 1–2, and publication history/disclosures on p. 15. All six final crops were individually viewed with labels and relevant table headers retained. Figure 2's arrows were checked against its caption and the control discussion; its Select arrow is distinguished from gradient updates. Table 1's missing compute column is disclosed. The supplied PDF has no appendix. Reference-only pp. 13–14 were completely read as text but not visually inspected; no independent claims about their cited works are made. Separate supplement availability remains unverified.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 15. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Introduction and stated scope (p. 1)
- Learning-based dynamics models: background, perception, dynamics, control, and comparison with physics-based models (pp. 1–3)
- State representations: pixels, latent states, particles, keypoints, and object-centric states; comparison and selection (pp. 3–8)
- Connection to robotic control: motion planning, policy learning, and representative robotic tasks (pp. 7–11)
- Future directions: perception, dynamics learning, and robotic control (pp. 9–12)
- Conclusions (p. 12)
- References and notes; acknowledgments, author contributions, competing interests, and publication history (pp. 12–15)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This supplied extraction limitation was addressed by visually inspecting Figures 1–4, Tables 1–2, and the relevant page layouts.
- Separate supplemental material availability has not been fully verified.
- The supplied artifact is the published Science Robotics 10, eadt1497 review dated 17 September 2025, DOI 10.1126/scirobotics.adt1497. Its title and all ten authors agree with the catalog; the title page supplies the period in Henrik I. Christensen. No different edition or earlier revision was supplied for comparison.
- No appendix is present in the supplied 15-page PDF. Its reference list was read, but the referenced works were not independently opened or verified.
- All nine supplied text chunks were read individually. Pages 1–12 and 15 were visually inspected; the reference-only pages 13–14 were read as text.
- No code was inspected, no experiments were reproduced, and no external sources were acquired.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author list, affiliations, and publication footer; p. 15, publication history and DOI
The title and ten-author sequence match the catalog. The published review is Science Robotics 10, eadt1497, dated 17 September 2025, DOI 10.1126/scirobotics.adt1497. It records submission on 25 November 2024 and acceptance on 22 August 2025.
Go to primary source ↓scopePDF p. 1, abstract, Introduction, and opening of Learning-based dynamics models
The review focuses on learned environmental dynamics for robotic manipulation, their state representations, perception, and control. Analytical, nonlearned differentiable, and hybrid approaches are outside its comprehensive scope.
Go to primary source ↓frameworkPDF p. 2, Background, Eq. (1), Perception module, Dynamics module, and Control module
Defines observation, action, state, perception g, transition models, policy, cost, and horizon; gives the history-dependent perception relation and expected cumulative-cost objective. Control can output positions or forces.
Go to primary source ↓physics-comparisonPDF pp. 2–3, Comparison with physics-based dynamics models
Discusses hard-to-measure physical effects, state-estimation and identification errors, the sim-to-real gap, learned compensation, differentiability, and potential computational advantages.
Go to primary source ↓human-abstractionsPDF p. 2, Figure 1 panels A–C and caption
Illustrates particles for fine-grained geometry, keypoints for structural detail, and object-centric representations for multiobject interaction. The panels depict intuitive human predictions, not quantitative robot evaluations.
Go to primary source ↓training-control-figurePDF p. 3, Figure 2A–B, arrows, legend, and caption
Shows observation encoding, action-conditioned next-state prediction, and a prediction loss. Downstream branches use model rollouts to evaluate action trajectories or generate interaction data for policy updates. Black arrows mark forward propagation; dashed red arrows indicate updates, with the planning arrow specifically labeled Select.
Go to primary source ↓representations-overviewPDF pp. 3–4, opening State representations discussion; p. 7, Figure 3 and caption
Distinguishes pixels, latent encodings, particles, keypoints, and object-centric states as choices with different levels of structure and perception demands.
Go to primary source ↓pixelsPDF p. 4, 2D pixel representations and Dynamics learning
Describes action-conditioned observation prediction, self-supervised maximum likelihood, proprioceptive action labels or inferred latent actions, and appearance-based metrics that may not reflect planning quality. Notes partial observability, hallucination, temporal consistency, and compute challenges.
Go to primary source ↓latent-learningPDF p. 4, Latent representations and Perception and representation learning
Explains compact latent encodings, collapse avoidance, reconstruction, optical flow, inverse dynamics, contrastive objectives, reward supervision, and recurrent history aggregation.
Go to primary source ↓latent-dynamicsPDF pp. 4–5, Latent representations: Dynamics learning and summary
Contrasts probabilistic and deterministic latent prediction, action concatenation, recurrent state-space models, MLPs and CNNs, and task-specific versus task-agnostic learning tradeoffs.
Go to primary source ↓particlesPDF p. 5, 3D particle representations, perception, dynamics learning, and summary
Particles encode surfaces or volumes. Point-cloud sensing, occlusion and correspondence affect estimation; graph and convolutional approaches model local interactions. Describes dynamic graphs, hierarchical nodes, and actions supplied through controllable-object poses in SPNets.
Go to primary source ↓keypointsPDF pp. 5–6, Keypoint representations, perception, dynamics learning, and summary
Sparse points retain task-relevant geometry or semantics. Extraction may be supervised, reconstruction-based, or based on pretrained vision models. Ordered points can use MLPs, and graph models also apply; consistent detection and occlusion remain concerns.
Go to primary source ↓object-centricPDF pp. 6–7, Object-centric representations, perception, dynamics learning, and summary
Describes segmentation, unsupervised object discovery, inverse rendering, and relational dynamics. Compositional structure supports varying object configurations, but perception and continuous-material modeling remain difficult.
Go to primary source ↓representation-tradeoffsPDF p. 7, Comparing and selecting state representations and Figure 3
Contrasts expressiveness, perception complexity, generalization, interpretability, and compute. Mentions training some structured models within a few GPU hours and graph-edge scaling, but provides no common timing benchmark. The text says Table 1 includes computational requirements.
Go to primary source ↓study-tablePDF p. 8, Table 1, caption, all column headers and representation groups; Ai et al. (20) and Manuelli et al. (13) rows
The table records sensing modalities, dynamics in 2D or 3D, and object types using checkmarks. Ai et al. has RGB, depth, multiview, tactile, 3D, rigid, deformable, and multiobject checks; Manuelli et al. has RGB, depth, 3D, and rigid checks. There are no runtime, accuracy, or success-rate columns.
Go to primary source ↓planningPDF pp. 7–8, Connection to robotic control: Motion planning; p. 9, opening continuation
Separates path planning from trajectory optimization; discusses RRT/PRM paths, CEM/MPPI action sampling, gradient-based updates, and online system identification.
Go to primary source ↓policy-learningPDF p. 9, Policy learning
Model-generated current-state, goal-state, and action tuples support inverse dynamics policies; simulated transitions also support reinforcement learning. Discusses rollout errors, multimodal actions, model exploitation, and real-data fine-tuning.
Go to primary source ↓robotic-tasksPDF p. 9, Representative robotic tasks: Object repositioning, Deformable object manipulation, Multiobject manipulation, and Tool-use manipulation
Surveys planning and policy integrations for repositioning, deformables, packing, and tools, including RoboPack with MPPI and action priors, and latent-space RRT combined with model predictive control.
Go to primary source ↓task-illustrationPDF p. 10, Figure 4 panels A–F, panel attributions, and caption
Shows sequential examples of object repositioning, rope, cloth, plasticine, multiobject, and tool-use manipulation. The montage illustrates different cited systems without performance scores or comparative trials.
Go to primary source ↓task-tablePDF p. 11, Table 2, all headers and task groups; plasticine and tool-use rows for Shi et al. (16, 17), Bauer et al. (111), Wang et al. (22), and Ai et al. (20)
Maps each listed task to representation, dynamics model class, and controller. Plasticine rows include particle/GNN/gradient descent, particle/GNN/learned policy, and latent/Transformer/CEM. Tool-use rows include particle/GNN with learned policy, gradient descent, or MPPI. No common metrics or evaluation splits are supplied.
Go to primary source ↓perception-gapsPDF p. 9, Future directions: Partially observable domains and Multimodal perception
Identifies occlusion and hidden physical properties, history and active sensing, multimodal statistical differences, mismatched sensing frequencies, and unresolved signal fusion.
Go to primary source ↓robustnessPDF pp. 9–10, Future directions: Robust dynamics models
Warns that planning and reinforcement learning exploit model inaccuracies; broad state-action coverage is difficult. Suggests simulation coverage, physical priors, and probabilistic treatment of aleatoric uncertainty.
Go to primary source ↓future-dynamicsPDF p. 10, Foundation dynamics models; Dynamics priors from foundation models; Emerging representations from graphics research; Large-scale scene representations
Discusses scarce action-labeled data, latent-action inference, material priors, NeRF and Gaussian representations, and the gap between local tabletop models and coherent large scenes.
Go to primary source ↓future-controlPDF pp. 10–12, Opportunities in robotic control: Hierarchical dynamics modeling and planning, Learning to plan, Performance guarantees; p. 12, Conclusions
Proposes spatial and temporal abstraction, learned search guidance and action proposals, uncertainty estimation, and stronger planning guarantees; emphasizes that generalizable and scalable models remain open challenges.
Go to primary source ↓8.5 Primary sources
A review of learning-based dynamics models for robotic manipulation ↗
PDF · 15,011 extracted words
Source fingerprint
4b3141f6ef863ff98be6d9247bec435d3d72e2c9b9648162c64bc5826964d7ad