PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation
1. Paper overview
In one sentence: PIN-WM turns image-based rigid-body identification into a training environment for a separate robot policy, then uses nearby physics and appearance variations to improve transfer. e02e03e05e07e11e12e17
| At a glance | What to know |
|---|---|
| Research problem | Source description Pushing and poking are sensitive to friction, inertia and restitution. The paper asks whether a small amount of task-agnostic interaction can identify a useful training environment for these contact-rich skills. It assumes known object geometry and targets a specific environment, rather than learning broadly reusable dynamics from large video collections. e02 |
| Core mechanism | |
| A key reported result | Real-world T-shaped push and mug flip: PADC: push 75%, 37.5 steps; flip 65%, 11.3 steps. Success rate; reported #Steps. Direct deployment; 20 trials under matched initial conditions; no real-world fine-tuning. Without PADC: 65%/60%, 45.2/13.2 steps. ASID + rendering: 40%/10% success. PADC adds 10 and 5 percentage points over the identified twin. These are robot executions, but the small evaluation lacks reported confidence intervals. e07e12 |
| Reading caution |
Core contributions
Figure 2. A fitted renderer supplies image gradients to physics identification; a separate policy learns in the resulting simulators. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left half from top to bottom. Panel (a) fits the 2D Gaussian representation from several views. In panel (b), the black arrow carries the predicted next state from the LCP physics block to the renderer. The green return arrow carries the physics gradient; the snowflake marks the renderer as frozen during this stage, as the caption and Eq. (8) confirm. On the right, panel (c) trains one policy using several physics/rendering variants, with rendered observations and rewards returning through the training loop. Panel (d) replaces that simulated loop with real observations and executed actions. e03e04e05e07e11
What it supports. The architecture separates three operations: appearance fitting, physical identification, and policy learning. Only the trained policy is shown in the real deployment loop. This supports the dual-system reading: the world model supplies an interactive training environment, while a distinct policy chooses robot actions from visual feedback.
Where the evidence stops. The overview simplifies preprocessing. Real implementation still uses a known mesh, FoundationPose for the initial pose and SAM 2 segmentation. Section III.C narrows the appearance randomization to spherical-harmonic parameters.
2. Motivation
2.1 The problem and the proposed response
Pushing and poking are sensitive to friction, inertia and restitution. The paper asks whether a small amount of task-agnostic interaction can identify a useful training environment for these contact-rich skills. It assumes known object geometry and targets a specific environment, rather than learning broadly reusable dynamics from large video collections. e02
2.2 What this reading follows
A robot can watch a simple push without learning how to complete a difficult manipulation task. PIN-WM uses that distinction: static views establish what an object looks like, an interaction video constrains how it moves, and reinforcement learning discovers useful actions inside the fitted simulator. The paper then relaxes the idea that one fitted simulator is exact, training across nearby digital cousins. Read the architecture first to separate parameter fitting from policy learning, then compare simulation accuracy with actual robot success. The experiments support useful transfer for the tested rigid objects, while initial pose estimation, known geometry and ambiguous physical parameters remain essential qualifications. e02e03e05e07e11e12e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded dual-system, other-mechanisms and outside-quadrants classification fits Figure 2: a physics-and-rendering world model supplies an environment for a separate PPO policy. Actions condition forward dynamics; they are not inferred by inverse dynamics or jointly generated with future observations. Multiview 3D representation and world-model-assisted RL are supported, while policy post-training should not imply adaptation of a pretrained generalist policy. e02e03e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Why freeze appearance before fitting motion?
Begin with the object at rest. The robot captures multiple viewpoints, and 2D Gaussian splats are optimized so their rendered images agree with those observations. The source then freezes these appearance parameters before fitting physics. During a push, the simulator predicts the object pose, a rigid transformation moves the splats, and the renderer predicts the next image. Equation (8) compares that prediction with the observed interaction video; gradients pass back to the physical parameters through both rendering and dynamics. Reader interpretation: freezing appearance reduces one route by which image fitting could conceal an incorrect trajectory. It does not make the physical estimate unique. Moreover, the real pipeline still obtains an initial pose with FoundationPose and uses a known mesh, so avoiding state supervision along the trajectory does not mean eliminating geometric initialization. e03e04e05e11e17
5.2 How an end-effector translation becomes a predicted image
The action is an end-effector translation, not a measured force. PIN-WM converts that displacement over the action duration into imposed end-effector velocity. The physical model represents positions, orientations and twist velocities for the object, robot end-effector and floor, while treating the floor as stationary and neglecting changes to the robot’s own dynamics. A velocity-based linear complementarity problem resolves joint, contact and friction constraints. Its resulting velocities update position and quaternion orientation through semi-implicit integration. Because an action can last longer than a simulation step, the paper subdivides it and propagates derivatives through the sequence. The predicted object pose then moves the Gaussian representation. This creates the differentiable path from physical parameters to pixels, but its validity depends on the rigid-body and imposed-robot-motion assumptions. e03e04e05e06
5.3 Why cousins matter after a useful twin has been identified
A fitted simulator can predict the observed interaction reasonably well and still omit effects relevant to control. PADC samples local variations around identified physics and spherical-harmonic appearance parameters, using a ten-percent multiplicative perturbation. PPO learns one policy across those variations; the simulation setup resamples each cousin after an episode. At deployment, the real image feeds the policy and the robot executes its translation action, with the next real image closing the loop. Reader interpretation: the randomized simulator family serves as a training distribution, rather than an inference-time planning ensemble. Table IV helps explain why its center and width matter: oracle-centered, narrower ranges improve domain randomization. That diagnostic motivates the design, but the with/without PADC comparisons do not separate the contribution of physics variation from appearance variation. e03e07e08e12e16
5.4 Training and inference
During training
Simulation identification uses one straight-line pushing trajectory. Physics-based comparisons share this data; Dreamer V2 receives 100 task-agnostic trajectories, while Diffusion Policy receives 100 successful expert trajectories. These protocols probe low-data learning with different priors and supervision. e08
World-model fitting precedes policy optimization. Appendix A states that RL comparisons share architecture, hyperparameters, reward and stopping criterion; the handcrafted reward is the negative sum of translation and rotation distances to the target. e03e15
During inference
The trained policy observes real RGB feedback and issues end-effector translations, with no real-world fine-tuning. Figure 2(d) closes the observation/action loop through the physical environment. It does not show simulator rollouts or online parameter fitting during deployment. e03e07
5.5 Implementation flow
- Align appearance first
Fit surface-attached 2D Gaussian splats to static views. The objective combines image reconstruction and D-SSIM with depth-distortion and normal-consistency regularization. Rigid transforms move the splats when the simulated object pose changes. Real preprocessing uses FoundationPose for the initial pose and SAM 2 for segmentation, qualifying the broad claim of avoiding state estimation. e04e11
- Identify contact dynamics through pixels
Freeze rendering parameters and replay the same translations in simulation and the observed sequence. The state contains position, orientation and twist velocity; learned physics comprises mass/inertia, restitution and friction. An LCP enforces joint, contact and friction constraints, followed by pose integration. Gradients traverse rendering and successive physics steps. The floor stays fixed, and the end-effector is treated as an imposed motion source. e03e05e06
- Train across nearby environments
Perturb identified physics and each splat’s spherical-harmonic appearance coefficients by a multiplicative uniform range of ±10%. Train PPO in these cousins. The simulation experiment uses Bullet with 32 parallel environments, randomized initial poses and a fresh cousin after each episode. e07e08
6. Experiments & results
PIN-WM fits an object-specific rigid-body simulator from multiview images and simple interaction video, then trains a separate visual manipulation policy in nearby randomized simulators. Its strongest evidence is real robot transfer for pushing and flipping; the central tradeoff is dependence on known geometry, visual alignment and restrictive physical assumptions.
6.1 Read the original evidence
Table I. Simulation transfer improves further when the identified model is expanded into digital cousins. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task as a pair of columns: success should increase, while the reported number of steps should decrease. The final two rows provide the closest comparison because both use PIN-WM and differ in PADC. The middle rows compare physics-based alternatives, including ASID and 2D Physics augmented with the paper’s renderer. Section IV.A evaluates each method for 100 episodes, with limits of 100 steps for pushing and 25 for flipping. The data-driven methods use different training supervision and more trajectories, so those rows answer a broader low-data comparison rather than a clean architecture ablation. e08e09
What it supports. PADC raises simulated push success from 92% to 97% and flip success from 70% to 83%. Reported steps fall from 32.1 to 30.1 for push and 12.0 to 11.4 for flip. PIN-WM also retains strong performance without PADC, so the gain cannot be attributed solely to randomization.
Where the evidence stops. These are target-domain simulator tests, not robot success rates. Uncertainty across training seeds is not reported, and the paper does not fully specify the aggregation convention for the #Steps statistic.
Table II. Lower one-step errors support the dynamics model, but do not prove recovery of unique physical parameters. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first pair of numerical columns describes pushing; the second describes flipping. Trans. is translation error in meters, while Ori. is orientation error in radians, as defined in the evaluation text. Read every column with lower values preferred. The diagnostic applies the same randomly sampled action at the same object surface point to the learned and target-domain models, then compares the resulting object state. This is a different endpoint from policy success: a policy may repeatedly correct local errors, and a model may fit observed transitions while remaining inaccurate under other contacts. The separate Figure 4 plots training errors. e10e17e18
What it supports. PIN-WM reports 0.017 m and 0.13 rad for push, versus ASID’s 0.030 m and 0.40 rad. The corresponding flip entries are 0.014 m and 0.3 rad, versus 0.14 m and 1.6 rad. These numbers support improved prediction under the sampled one-action evaluation protocol.
Where the evidence stops. Do not treat this as a long-horizon or parameter-identifiability test. Appendix Table VI prints the zero-noise flip orientation error as 0.27 instead of 0.3; the source does not explain that precision difference.
Table III. The main physical deployment result is 75% push success and 65% flip success. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this table separately from the simulation table. Section IV.B describes a real T-shaped object pushed toward a target and a side-lying mug turned upside down, tested under identical initial conditions across 20 trials. The last two rows isolate the addition of the complete PADC package to PIN-WM. The other rows test several simulator-based alternatives, with the paper’s rendering function added where indicated. The source excludes Dreamer V2 and Diffusion Policy from this real comparison because of their data requirements. Compare success first, then consult the reported step counts without assuming they describe successful trials alone. e07e12
What it supports. With PADC, PIN-WM achieves 75% push success and 65% flip success, compared with 65% and 60% without PADC. Its reported steps also decrease, from 45.2 to 37.5 for push and 13.2 to 11.3 for flip. This is evidence of physical policy transfer for the tested tasks.
Where the evidence stops. Twenty trials provide limited resolution, and confidence intervals or repeated-training variability are not reported. The PADC comparison perturbs both physics and appearance, so it cannot assign the gain to either component individually.
Figure 8. Representative robot sequences make the three-dimensional flip task concrete. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the time arrow from left to right within a row. Each row names a different method, and each starts with the mug lying on its side. The task described in Section IV.B is to turn it upside down, so inspect the final mug orientation rather than simply how far it moves. The PIN-WM row shows a successful change to the inverted pose; the baseline rows illustrate other outcomes in these selected sequences. Use the photos to understand the contact and orientation changes that distinguish flipping from planar pushing, then return to Table III for frequencies across trials. e03e12e13
What it supports. The selected PIN-WM sequence illustrates actual robot execution of a three-dimensional reorientation, consistent with the paper’s emphasis on 3D rigid-body dynamics. It supplies a qualitative example of the behavior counted in the quantitative evaluation, where PIN-WM with PADC reports 65% mug-flip success.
Where the evidence stops. This montage contains selected examples, not a success distribution. The frames have no timestamps or action magnitudes, so they do not establish control frequency, completion time or a quantitative contact-force comparison.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Real-world T-shaped push and mug flip Direct deployment; 20 trials under matched initial conditions; no real-world fine-tuning. | PADC: push 75%, 37.5 steps; flip 65%, 11.3 steps. Success rate; reported #Steps | Without PADC: 65%/60%, 45.2/13.2 steps. ASID + rendering: 40%/10% success. PADC adds 10 and 5 percentage points over the identified twin. These are robot executions, but the small evaluation lacks reported confidence intervals. e07e12 |
| Simulated push and flip Target-domain evaluation over 100 episodes per method; limits of 100 push and 25 flip steps. | PADC: push 97%, 30.1 steps; flip 83%, 11.4 steps. Success rate; reported #Steps | Without PADC: 92%/70%. ASID + rendering: 58%/11%; 2D Physics + rendering: 55%/8%. Both identification and cousin training are associated with stronger transfer in this restricted simulation protocol. e08e09 |
| One-step dynamics prediction Simulated object transitions under matched sampled actions and surface contact points. | Push: 0.017 m, 0.13 rad. Flip: 0.014 m, 0.3 rad. Translation error (m); orientation error (rad), lower is better | ASID: push 0.030 m, 0.40 rad; flip 0.14 m, 1.6 rad. Supports local predictive accuracy, without establishing long-horizon accuracy or unique parameter recovery. Appendix Table VI reports zero-noise flip orientation as 0.27 rather than Table II’s 0.3; the source does not explain this precision difference. e10e17e18 |
| Randomization-range diagnostic Simulation; oracle DR ranges shrink around ground-truth physics. | PADC 97%/83%; direct ground truth 98%/89%. Push/flip success rate | DR at R: 33%/32%; R/2: 56%/43%; R/4: 78%/61%. A well-centered narrow training distribution helps. Ground-truth controls require privileged information and are not deployable identification methods. e16 |
| Dynamics prediction with noisy geometry Appendix source-labeled noise level sigma=3.0; compared with ASID using perfect geometry. | Push: 0.029, 0.30; flip: 0.077, 1.5. Translation error (m); orientation error (rad) | ASID: push 0.030, 0.40; flip 0.14, 1.6. PIN-WM remains better on these entries, although noise substantially degrades flip prediction. The source’s noise variance convention is ambiguous. e10e18 |
| Additional slippery-plane pushes and cube flip Real-world appendix evaluations; separate trial count not stated. | Slippery T push 45%; slippery cube push 40%; cube flip 60%. Success rate | ASID: 5%, 0%, 5%, respectively. Broadens the demonstrated settings, while the slippery-task success rates show substantial remaining failure. e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table IV. Shrinking a well-centered randomization range improves policy success. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the rightmost DR(R) column and move left through DR(R/2) and DR(R/4). Those columns reduce the randomization range around ground-truth physical parameters, according to Appendix B.1. The GT column uses the ground-truth parameters directly. PIN-WM with PADC instead builds its local range around identified parameters. The two rows test pushing and flipping separately. This layout helps distinguish a broad training distribution from a useful neighborhood around the target dynamics. It also exposes the oracle information in the controls: knowing the center of that neighborhood is itself part of the real system-identification problem. e07e16
What it supports. Reducing DR from R to R/4 raises push success from 33% to 78% and flip success from 32% to 61%. PADC reaches 97% and 83%, while direct ground-truth physics reaches 98% and 89%. The diagnostic supports concentrating training around relevant dynamics rather than assuming any wide randomization is sufficient.
Where the evidence stops. The DR controls use privileged ground-truth centering. Appendix B.1 frames the difference as a physics-range change, whereas Section III.C includes appearance perturbation in PADC; the table does not independently isolate that appearance contribution.
7. Analysis & limitations
7.1 What the evidence leaves open
Moving robot shadows can bias image losses and corrupt identified physics. Elasticity, plasticity and deformable objects are outside the rigid-body formulation. e03e14
Multiple physical parameter sets can explain the same video. Table V estimates one parameter at a time with all others fixed to ground truth and omits inertia; it cannot establish joint recovery of the full physical system. e17
PADC simultaneously changes physics and appearance, so its with/without comparison does not isolate either component. The real-world evaluation omits Dreamer V2 and Diffusion Policy, and provides no seed variation or confidence intervals. e07e12
7.2 Questions for discussion
- How much of PADC’s gain comes from physics perturbation versus spherical-harmonic appearance perturbation?
- Which additional contacts would distinguish equally image-consistent physical parameter estimates?
8. Reproducibility audit
8.1 Requirements and known gaps
Required ingredients include calibrated multiview images, known meshes, initial pose estimates, object masks, synchronized translation/video sequences, differentiable rendering and LCP simulation, and a Bullet/PPO training environment. e04e05e06e08e11
The supplied paper does not enumerate hardware models, training compute, numerical PPO settings, exact success pose tolerances, identification optimization schedules, or numerical H/h values. Appendix A’s shared-defaults statement is insufficient for exact replication. Appendix B describes geometry noise using a variance proportional to a length, whose implementation needs clarification. e06e08e11e15e18
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate physics and appearance robustness
Proposed check, not performed: identify one fixed twin, then train matched PPO policies under four conditions: no perturbation, physics-only perturbation, spherical-harmonic-only perturbation, and both. Use the paper’s delta=0.1, identical training budgets, rewards, initial-pose distributions and multiple policy seeds. Evaluate all four on the same held-out real push and flip starts, reporting success counts and uncertainty. Independently vary lighting and contact friction in controlled tests. A physics-specific benefit should concentrate under friction changes, while an appearance-specific benefit should concentrate under lighting changes; failure of that separation would weaken a simple mechanism attribution. Specify pose tolerances and failure handling before evaluation because the supplied paper does not fully define them. e07e08e12e14e15
Check 2: Test whether image-consistent physics transfers to new contacts
Proposed check, not performed: fit all physical parameters jointly from the same straight-line push using several initial parameter guesses, keeping geometry, renderer and optimization budget fixed. Measure training image loss, then apply held-out pushes at different surface points and directions and a flipping interaction. Compare one-step translation/orientation error and policy success, alongside the paper’s easier diagnostic that fits one parameter while others remain at ground truth. Similar training image losses with divergent held-out predictions would expose ambiguity hidden by the fitting trajectory. As a controlled follow-up, add one deliberately different interaction and test whether held-out errors contract. This tests predictive sufficiency without equating a good image fit with recovery of unique physical constants. e05e08e10e17
8.3 Reading coverage
Visual audit: Visually inspected the title, author and v2 date on p. 1; all method pages and Eqs. (1)–(12) on pp. 3–6; evaluation text, tables and training curves on pp. 7–9; all real trajectories on pp. 10–11; and Appendix A–C, Tables IV–VII and Figures 9–11 on p. 14. Figure 2’s forward state arrow, backward physics gradient and frozen-rendering snowflake agree with its caption and Eq. (8); Section III.C specifies that appearance perturbations use spherical-harmonic parameters. Every final crop was viewed after extraction, with labels and table rules intact. Reference-only pp. 12–13 were read as text. All six text chunks were read completely. No supplementary video, external project material or code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract and I. Introduction
- II. Related Work (A–C)
- III. Method (A–C), Eqs. (1)–(12)
- IV. Results and Evaluations (A–B)
- V. Conclusions and limitations
- VI. Acknowledgements and References
- Appendix A. Implementation Details for Baselines
- Appendix B. More Experimental Results (1–3)
- Appendix C. Further Real-World Evaluations
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting PDF pages 1–11 and 14, including all figures and tables. Reference-only pages 12–13 were read as text.
- Revision scope: this report reviews v2 dated 3 May 2025. The catalog records submission on 23 April 2025; v1 was not supplied, so changes between versions were not compared. The observed title and author list match the catalog.
- No code, project website or separate supplement was inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author affiliations and arXiv version stamp
The title and seven authors match the catalog. The artifact is arXiv:2504.16693v2, 3 May 2025. Four institutions are printed; Wenxuan Li and Hang Zhao have equal-contribution marks, and Ruizhen Hu and Kai Xu are corresponding authors.
Go to primary source ↓e02PDF pp. 1–3, Introduction; Section II.B, final paragraph; Section III.A
Contact-sensitive pushing and flipping motivate an environment-specific world model. Geometry is assumed known; the authors explicitly distinguish this model from a general-purpose world foundation model.
Go to primary source ↓e03PDF p. 4, Figure 2 and caption; Sections III.A–B, Eq. (2)
The pipeline separately aligns rendering, freezes its parameters during physics fitting, and trains a policy in perturbed simulators. State includes positions, orientations and twist velocities of the object, end-effector and floor. Physics parameters cover mass/inertia, restitution and friction.
Go to primary source ↓e04PDF p. 5, Section III.B, Rendering Alignment, Eqs. (3)–(7)
Static multiview images fit 2D Gaussian splats using image/D-SSIM loss and depth-distortion/normal-consistency regularizers. Object-pose transformations move the splats for subsequent rendering.
Go to primary source ↓e05PDF p. 5, Section III.B, Identification of Physics Parameters, Eq. (8)
With rendering parameters fixed, physics parameters minimize summed squared image discrepancies along a task-agnostic interaction sequence. Actions are end-effector translations rather than measured wrenches.
Go to primary source ↓e06PDF pp. 5–6, Section III.B, Eqs. (9)–(11) and following paragraphs
A velocity-based LCP with joint, contact and friction constraints yields twist velocities; semi-implicit Euler updates poses. A primal-dual interior-point solver supports differentiation. Robot dynamics are neglected relative to object dynamics, the floor is stationary, and action duration H is subdivided into H/h simulation steps.
Go to primary source ↓e07PDF p. 6, Section III.C, Eq. (12)
PADC uniformly perturbs identified physics and per-splat spherical-harmonic rendering parameters multiplicatively with delta 0.1. PPO learns the policy, which is deployed without real-world fine-tuning.
Go to primary source ↓e08PDF pp. 7–8, Section IV.A, Experiment setup, Evaluation metrics, Baseline methods and policy comparison
Simulation identification uses one straight-line pushing trajectory. Bullet runs 32 parallel cousins, resampled after episodes. Policies observe RGB; initial poses are randomized. Evaluation uses 100 episodes per method and limits of 100 push or 25 flip steps. Physics-based baselines share the interaction data; Dreamer V2 and Diffusion Policy receive 100 task-agnostic and 100 task-completion trajectories, respectively.
Go to primary source ↓e09PDF p. 7, Table I, PIN-WM, ASID, 2D Physics and other method rows
With PADC, push/flip success is 97%/83%, with 30.1/11.4 steps. Without PADC it is 92%/70%, with 32.1/12.0 steps. ASID plus rendering achieves 58%/11%; 2D Physics plus rendering achieves 55%/8%.
Go to primary source ↓e10PDF p. 8, Table II and system-identification comparison; PDF p. 7, Evaluation metrics
One-step translation/orientation error uses the same sampled action at the same surface point, in meters/radians. PIN-WM reports push 0.017/0.13 and flip 0.014/0.3; ASID reports 0.030/0.40 and 0.14/1.6. Figure 4 on p. 8 plots training errors, not these aggregate test measurements.
Go to primary source ↓e11PDF pp. 8–9, Section IV.B, Experiment setup; p. 8, Figure 5
The real system has an eye-in-hand and an eye-to-hand camera. Known mesh geometry and FoundationPose supply the initial object pose. Camera poses are recorded, SAM 2 segments the object, static views fit rendering, and a straight-line push video fits physics.
Go to primary source ↓e12PDF p. 9, Section IV.B, policy comparison and Table III
Real tests use identical initial conditions across 20 trials: a T-shaped push and a mug flip. PADC yields 75%/65% success and 37.5/11.3 steps; no PADC yields 65%/60% and 45.2/13.2. ASID plus rendering yields 40%/10%, Domain Rand plus rendering 10%/25%, and 2D Physics plus rendering 35%/5%. Data-driven baselines are excluded from this real-world comparison.
Go to primary source ↓e13PDF p. 11, Figure 8; PDF p. 9, Section IV.B, task definition
Five rows show representative real mug-flip sequences for Domain Rand, RoboGSim, ASID, 2D Physics and PIN-WM. Time advances left to right. The goal is to turn a side-lying mug upside down; the figure does not provide frame timestamps or trial frequencies.
Go to primary source ↓e14PDF p. 9, Section V, Limitation and future work
Robot-motion shadows can corrupt rendering loss and physical-property estimates. The framework models rigid bodies; differentiable relighting and deformable-body simulation are future directions.
Go to primary source ↓e15PDF p. 14, Appendix A, Implementation Details for Baselines
The appendix states that RL policies share PPO architecture, rewards, hyperparameters and a success-based stopping criterion. Reward is minus translation distance minus rotation distance. Some baselines use official implementations with defaults. Numerical PPO settings, full architecture, training budget and exact success pose tolerances are not enumerated.
Go to primary source ↓e16PDF p. 14, Appendix B.1 and Table IV
Shrinking domain-randomization ranges around ground-truth physics improves push/flip success from 33%/32% at R, to 56%/43% at R/2, to 78%/61% at R/4. Direct ground-truth parameters achieve 98%/89%; PIN-WM with PADC achieves 97%/83%. Ground-truth centering is acknowledged to be unavailable in real applications.
Go to primary source ↓e17PDF p. 14, Appendix B.2 and Table V
The authors call system identification ill-posed. The parameter diagnostic estimates one parameter at a time with the rest fixed to ground truth, and excludes inertia. PIN-WM estimates push mass 0.76 kg and flip mass 1.19 kg against 1.00 kg ground truth.
Go to primary source ↓e18PDF p. 14, Appendix B.3 and Table VI
At the source-labeled geometry-noise level sigma=3.0, PIN-WM has push translation/orientation errors 0.029/0.30 and flip errors 0.077/1.5, compared with ASID on perfect geometry at 0.030/0.40 and 0.14/1.6. The source calls sigma percent of bounding-box diagonal L a variance, leaving its dimensional convention unclear. The zero-noise flip orientation entry is 0.27 here versus 0.3 in Table II.
Go to primary source ↓e19PDF p. 14, Appendix C, Table VII and Figures 9–11
Further real-world success is 45% for slippery-plane T pushing, 40% for slippery-plane cube pushing and 60% for cube flipping, versus ASID at 5%, 0% and 5%. This subsection supplies no separate trial count or uncertainty estimates.
Go to primary source ↓8.5 Primary sources
PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation ↗
PDF · 10,639 extracted words
Source fingerprint
c8fc165446f13815ee35322e5123c92773494ff6803cebc690349357eb0c9379