ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation
1. Paper overview
In one sentence: Material-aware particle attention and hybrid geometric supervision improve the tested dynamics and control pipelines, while requiring scene-specific training and external perception. e03e05e06e07e09e11e12e13e17
| At a glance | What to know |
|---|---|
| Research problem | Source description Multi-material manipulation couples directly actuated objects to passive motion: lifting cloth moves granular objects, and a rope can transmit contact to other objects. The paper targets prediction of these interactions from partial 3D observations. Its motivation is that fixed graph neighborhoods constrain information propagation, while dense tracked-particle supervision makes real-data preparation expensive. Avoiding that tracking does not eliminate stereo reconstruction or segmentation. e02e03e08 |
| Core mechanism | |
| A key reported result | Multi-material simulation dynamics prediction: 0.0023 / 0.0013 / 0.0008 MSE, lower is better; units not supplied. FleX Cloth Gathering, Rope Maneuver and Rope Sweeping, in that order; ground-truth particle states are available. Evaluation split size and prediction horizon are unspecified. Chamfer-only ablation: 0.0042 / 0.0037 / 0.0013; GBND trained with hybrid loss: 0.0076 / 0.0069 / 0.0029. ParticleFormer has the lowest listed MSE on all three tasks. These are dynamics errors, not robot success rates; the table reports no uncertainty. e08e09e15 |
| Reading caution | Reader analysis The authors explicitly limit demonstrated generalization: models are trained per scene, and transfer across diverse environments or robots is unestablished. External segmentation failures can corrupt predictions. Novel target configurations should therefore not be equated with scene-agnostic learning. e13e12 |
Core contributions
Figure 2. Stereo-derived particle states become displacement predictions, supervised by the next observed geometry. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the stereo pair on the left and follow preprocessing through stereo reconstruction and segmentation. The concatenation label identifies the three particle inputs: position x, material m and applied motion u. Read upward through the state projector, Transformer encoder and state predictor, then follow the curved arrow to the addition symbol. Equations (4)–(5) clarify that the predictor produces displacement and the addition updates current coordinates. The separate right branch processes the next stereo pair into ground-truth states for the hybrid loss. That branch supplies training targets; the predictor does not require future camera images when forecasting. e03e04e05e06e07
What it supports. The architecture preserves a particle-wise representation while allowing attention across the whole set. It removes the explicit neighbor graph from the learned dynamics stage and avoids correspondence-based tracking supervision. It still reconstructs 3D geometry from stereo and depends on segmentation, so its simplification concerns tracking and interaction modeling rather than all preprocessing.
Where the evidence stops. The lower shortcut is schematic: Equation (5) adds current coordinates to predicted displacement, not the concatenated material/motion features. Figure 2 omits the separate MPPI action-selection loop described in Section 3.4.
2. Motivation
2.1 The problem and the proposed response
Multi-material manipulation couples directly actuated objects to passive motion: lifting cloth moves granular objects, and a rope can transmit contact to other objects. The paper targets prediction of these interactions from partial 3D observations. Its motivation is that fixed graph neighborhoods constrain information propagation, while dense tracked-particle supervision makes real-data preparation expensive. Avoiding that tracking does not eliminate stereo reconstruction or segmentation. e02e03e08
2.2 What this reading follows
When a robot lifts cloth or moves a rope, objects it never touches directly can still move. ParticleFormer represents these coupled interactions as a set of 3D particles, predicts their displacements with a Transformer, and lets a separate planner choose actions. The central design couples flexible particle attention with a loss that penalizes both average geometric mismatch and large deviations. Read the architecture first, then use the prediction table to separate the contributions of attention and supervision. The physical rollouts test whether useful forecasting translates into control; the graph-neighborhood diagnostics expose an accuracy–memory tradeoff. The evidence supports these particular tasks, with scene-specific training and incomplete reproduction settings. e03e05e06e07e09e11e12e13e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system / Other mechanisms / Outside quadrants classification is supported: an action-conditioned 3D dynamics predictor feeds a separate MPPI planner. It neither jointly predicts actions and futures nor extracts actions through inverse dynamics. Stereo-derived point clouds support the 3D multiview tag in the limited sense of stereo perception, not demonstrated arbitrary-camera generalization. e03e04e05e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn heterogeneous materials into a common prediction interface
Consider Rope Sweeping: the robot acts on the rope’s ends, while granular objects respond through contact. The source represents both object and end-effector geometry as particles, then attaches material and applied-motion information to each particle. Object motion inputs are zero because these particles are not directly commanded; the end-effector input uses its positional change from forward kinematics. The projector turns these heterogeneous inputs into embeddings, and three Transformer layers allow interactions across the entire set. A shared predictor then returns particle displacements, preserving an explicit geometric output for planning. This information flow distinguishes a learned dynamics function from an action policy. The material labels and segmented point clouds are inputs to that function, not semantic categories or complete scene geometry that the network is shown discovering autonomously. e03e04e05e08
Figure 3. One-step prediction examples link visible object geometry to learned attention patterns. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task row from the observed state and blue robot-motion arrows to the three prediction columns. Compare the arrangement of predicted points, especially the box outline and the rope’s continuity, rather than treating the overlaid images as full trajectories. The rightmost heatmaps use attending particle on the vertical axis and attended particle on the horizontal axis, matching the caption’s row-i-to-column-j convention. In Rope Sweeping, the authors identify a diagonal rope-related band and broader granular-related blocks. Follow the color scale within each heatmap; the displayed scales differ between tasks. e08e10e06
What it supports. The examples illustrate the proposed failure mechanism: explicit graph connectivity can yield implausible local distortions, while the CD-only variant can underpredict motion transferred indirectly through cloth or rope. The full model’s examples look more coherent. These visual diagnoses complement the numerical table but do not establish their frequency across the test distribution.
Where the evidence stops. The paper interprets granular-to-rope attention as consistent with rope-to-granular force transfer. Attention direction is not a measured force direction. The heatmaps are descriptive model internals, and the figure supplies selected examples rather than a causal intervention.
5.2 Read the loss ablation as a geometric tradeoff
The authors motivate hybrid supervision by arguing that average nearest-neighbor matching can neglect indirectly moved particles. The Hausdorff approximation adds sensitivity to large deviations, and training applies the combined objective over five autoregressive steps. The discriminating evidence is the loss ablation: on Cloth Gathering, the full model improves MSE from 0.0042 to 0.0023 and CD+HD from 0.749 to 0.484, although CD rises from 0.082 to 0.096. Reader analysis: the mechanism is therefore better described as reallocating geometric error than improving every notion of similarity simultaneously. Table 2 repeats the CD-only advantage on all three single-material tasks. The examples are consistent with improved passive motion, but neither the table nor the heatmaps isolate a causal contribution for individual material groups. Such a claim needs a targeted check. e06e09e10e16
5.3 Close the loop without turning the predictor into a policy
During deployment, MPPI proposes candidate action sequences and the world model predicts their consequences. Costs compare those predicted object states with target geometry and penalize infeasible actions or collisions. Figure 5 then tests the resulting controller on physical tasks with feedback, using normalized CD+HD to track target mismatch. The task-specific targets can change while the scene-trained dynamics model remains the same. Reader analysis: this supports the catalog’s dual-system interpretation, because action selection is performed by a separate optimization procedure rather than an action output learned jointly with the next state. The distinction also bounds the generalization claim. Reaching a previously unseen target within an established scene does not demonstrate one dynamics model transferring across cameras, robots or environments; the authors explicitly leave such broad generalization unshown. e04e07e12e13
5.4 Training and inference
During training
Randomly select a starting frame and autoregressively unroll five future steps, feeding predictions back as inputs. Supervise with a weighted combination of Chamfer Distance and a differentiable Hausdorff approximation. The authors argue that the latter discourages neglect of indirectly moved particles; point-to-point correspondence is unnecessary. e06
Training is per scene. Simulation supplies 1,000 training episodes per task. Real data comprise 500-second episodes of random interactions; their count is unspecified. An NVIDIA A100 is reported for dynamics learning. The real camera records 1280×720 stereo images at 10 Hz. e13e14
During inference
At each control step, sample action sequences over horizon H, predict their trajectories and use MPPI to seek low-cost actions. The objective combines target mismatch with infeasibility and collision penalties; experiments use normalized CD+HD for target matching. Figure 5 demonstrates closed-loop physical execution. The network predicts consequences; action selection belongs to the planner. e07e12
5.5 Implementation flow
- Recover a partial geometric state
FoundationStereo reconstructs a dense point cloud; GroundingDINO and Segment Anything isolate relevant objects. Object points and end-effector points form the state. These external perception stages provide geometric inputs rather than a single scene-level image embedding. e03e04
- Encode material and actuation
Concatenate each particle’s position, material code and motion before a learned projector. Object motion inputs are zero; end-effector motion is the current-minus-previous end-effector position from forward kinematics. Object materials use one-hot codes; end-effector material codes are zero. This is an applied-motion input, not a supplied velocity for every object particle. e04
- Propagate interactions and predict motion
Three multi-head Transformer encoder layers let every particle attend to all others without an explicit adjacency graph. Spatial and motion information already enter the tokens, so no extra positional encoding is used. A shared decoder predicts displacements, which are added to current particle positions. e05
6. Experiments & results
ParticleFormer predicts action-conditioned 3D particle motion using material-aware Transformer attention and a Chamfer–Hausdorff training loss. Stereo reconstruction and segmentation supply point sets without requiring tracked particle correspondences. A separate MPPI planner uses predicted futures for manipulation. Experiments support better MSE and combined geometric error than the tested baselines, with a Chamfer-only tradeoff; deployment remains scene-specific.
6.1 Read the original evidence
Table 1. Hybrid supervision improves combined geometry and MSE, with exceptions on Chamfer-only error. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read one task column within one metric block before comparing methods. All arrows point downward: smaller values indicate less prediction error. The MSE block has no DINO-WM row because that pipeline does not produce corresponding particles. The two set-distance blocks include DINO-WM after its predicted images are reconstructed with Depth Anything V2. GBND is trained with the hybrid loss, while Ours w/o Hybrid uses only CD, so those comparisons address different design changes. The table’s bold entries make the CD exceptions visible; they should take precedence over the broader performance wording in Section 4.3. e08e09e15
What it supports. The full model’s MSE is 0.0023, 0.0013 and 0.0008 across the three tasks. On Cloth Gathering, hybrid training lowers CD+HD from 0.749 to 0.484 while increasing CD from 0.082 to 0.096. This supports a tradeoff between the reported geometric objectives, rather than universal superiority on every metric.
Where the evidence stops. These are simulation prediction errors, not manipulation success rates. The table does not provide metric units, evaluation split size, prediction horizon or uncertainty. DINO-WM’s scores also depend on the added image-to-3D reconstruction stage.
Figure 5. Closed-loop physical manipulation tests whether the learned dynamics help reach new targets. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the initial-and-target column: blue dashed regions identify initial object states and green dashed regions identify desired states. Move across the rollout composites to compare the three world-model choices, then examine the corresponding error curve at the right. Red denotes full ParticleFormer, blue GBND, and green the CD-only ablation. The vertical axis is normalized CD+HD, while the horizontal axis counts rollout steps rather than seconds. Each task has a different displayed step range. Section 4.4 states that MPPI uses the learned model to plan toward targets absent from training. e07e12e13
What it supports. The full-model curve ends below both baselines for all three real tasks. This connects improved geometric prediction to executed feedback control in the reported setup. The result is stronger than a visually plausible forecast alone, but the measured outcome remains final target-state mismatch, with no reported binary success criterion.
Where the evidence stops. The caption reports three rollout trials, and the statistical meaning of the shaded bands is unspecified. Exact endpoints are not tabulated. Models remain trained per scene, so new target geometry does not demonstrate transfer to new robots or arbitrary environments.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Multi-material simulation dynamics prediction FleX Cloth Gathering, Rope Maneuver and Rope Sweeping, in that order; ground-truth particle states are available. Evaluation split size and prediction horizon are unspecified. | 0.0023 / 0.0013 / 0.0008 MSE, lower is better; units not supplied | Chamfer-only ablation: 0.0042 / 0.0037 / 0.0013; GBND trained with hybrid loss: 0.0076 / 0.0069 / 0.0029. ParticleFormer has the lowest listed MSE on all three tasks. These are dynamics errors, not robot success rates; the table reports no uncertainty. e08e09e15 |
| Hybrid-supervision ablation on Cloth Gathering (simulation) Table 1 compares full ParticleFormer with its Chamfer-only variant on the same task. | CD+HD 0.484; CD 0.096 CD+HD and CD, lower is better | Chamfer-only: CD+HD 0.749 and CD 0.082; GBND: 0.886 and 0.152; DINO-WM: 1.053 and 0.187. Hybrid supervision improves combined error but sacrifices Chamfer error here. Rope Sweeping likewise favors Chamfer-only on CD (0.061 versus 0.067). Section 4.3’s claim of superiority across all metrics overstates Table 1. e09 |
| Single-material simulation dynamics prediction Appendix B.1: Rope, Granular Object and Cloth, in that order; quantitative evaluation split and horizon are unspecified. | 0.00017 / 0.00152 / 0.00160 MSE, lower is better | Chamfer-only: 0.00025 / 0.00173 / 0.00182; GBND: 0.00043 / 0.00196 / 0.00194. The MSE advantage extends to these single-material tasks, while Table 2 gives the Chamfer-only variant the best CD on all three. e16 |
| Goal-directed manipulation with MPPI (real robots) Box Pushing, Cloth Gathering and Rope Sweeping toward novel targets; Figure 5 caption states three rollout trials. | The full model’s plotted curves finish below both compared baselines on all three tasks; exact endpoints are not tabulated. Normalized target-state CD+HD over rollout steps | GBND and ParticleFormer without hybrid supervision. This is physical closed-loop control evidence. The shaded bands lack a stated statistical definition, and neither a success threshold nor a success rate is reported. Appendix C adds qualitative simulation rollouts. e12e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 4. The tested graph model becomes more accurate as its allowed neighborhood grows. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each task group, compare the three blue GBND bars in the legend’s order: TopK 5, 10 and 15. TopK limits the maximum number of adjacent particles. The decreasing bar heights show that changing this graph construction setting changes dynamics accuracy. Then compare the red ParticleFormer bar, which is lower in all three groups and does not require TopK. Use the companion memory visual to assess the cost of increasing connectivity. Table 1 provides exact main-comparison values; this diagnostic adds a neighborhood sweep rather than an additional task or success measure. e08e09e11e17
What it supports. The pattern supports the authors’ concern that graph construction can limit useful interaction propagation. Expanding GBND’s neighborhood improves the tested model but does not close the plotted gap to ParticleFormer. This is evidence about a particular graph implementation and sweep, with the hybrid-loss baseline described in Section 4.1.
Where the evidence stops. TopK is a GBND ablation, not a test that removes attention from ParticleFormer. The figure supplies no error bars or parameter-budget matching details. Absence of graph tuning does not mean that Transformer training has no hyperparameters.
Figure 8. The accuracy benefit of larger GBND neighborhoods carries a measured memory cost. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the axis carefully: this panel measures GPU memory in MB, not MSE or elapsed time. As in the accuracy diagnostic, red is ParticleFormer and the three blues are GBND with TopK 5, 10 and 15. Appendix B.2 specifies input clusters of 200 scene particles for this comparison. Within each task, the GBND bars rise with TopK, while the ParticleFormer bar lies below them. Compare the ordering with Figure 4: more graph neighbors improve prediction there but consume more memory here. The original legend overlaps part of the upper plotting area and is retained faithfully. e05e11e14e17
What it supports. At the reported particle count, ParticleFormer uses less memory than every plotted GBND setting. Read together with the accuracy sweep, this supports an attractive empirical operating point for the authors’ implementation. It does not establish how either architecture behaves when particle count, batch size or implementation efficiency changes.
Where the evidence stops. The paper does not report a particle-count scaling curve, wall-clock latency or uncertainty here. Appendix A names A100 hardware for dynamics learning, but full training settings are absent. This memory comparison cannot establish a universal computational advantage of attention over graphs.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors explicitly limit demonstrated generalization: models are trained per scene, and transfer across diverse environments or robots is unestablished. External segmentation failures can corrupt predictions. Novel target configurations should therefore not be equated with scene-agnostic learning. e13e12
DINO-WM predictions are converted from 2D images into point clouds with Depth Anything V2 and lack point correspondences. Its geometric scores include this reconstruction stage; MSE is not reported for it. This is a pipeline comparison, not an isolated dynamics-backbone comparison. e09
TopK and MaxDist diagnostics support sensitivity of the tested GBND implementation. Memory measurements use 200 scene particles; they do not establish a general scaling law or control latency. Attention heatmaps suggest interactions but do not measure physical forces. e10e11e17e18
7.2 Questions for discussion
- Would the hybrid loss retain its advantage if indirect granular motion were scored separately from cloth or rope motion? [e06, e09]
- How much of the graph comparison survives matched training budgets, parameter counts and larger particle sets? [e11, e17]
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the FleX tasks, material labels, stereo/segmentation preprocessing and synchronized end-effector poses. Physical tests use one xArm-6 for box/cloth tasks, two for rope sweeping, a ZED-2i and approximately 70×55 cm workspace. Control runs on an RTX 3090. Software versions and exact calibration/preprocessing parameters are not specified. e03e04e14
The PDF leaves optimizer, learning rate, batch size, training duration, embedding width, attention-head count, perception freezing/fine-tuning policy, alpha, HD smoothing, evaluation split sizes and seeds unspecified. MPPI horizon, sampling budget, action bounds and cost coefficients are also missing. A faithful implementation must disclose these choices and the timing convention connecting motion inputs to candidate actions. e04e05e06e07e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does hybrid supervision specifically recover indirect granular motion?
Reader-proposed check, not performed: use identical Cloth Gathering simulation episodes, particle sampling, model architecture, initialization seeds and training budget for CD-only and hybrid training. Keep the source’s five-step autoregressive horizon. Because alpha and HD smoothing are unspecified, disclose their definitions and run a prespecified alpha sweep rather than silently selecting a favorable setting. On held-out episodes, score whole-scene CD and MSE, granular-only MSE, and worst geometric deviation before and after cloth lifting induces granular movement. A repeatable improvement in passive granular motion despite slightly worse aggregate CD would support the proposed mechanism. If gains occur only on the directly moved cloth, the passive-motion explanation would be weakened. e06e08e09e14
Check 2: Does the attention advantage survive matched resources and more particles?
Reader-proposed check, not performed: compare ParticleFormer with hybrid-trained GBND on the three multi-material simulation tasks, matching training data, optimization steps, parameter budget, batch size, numerical precision and GPU. Repeat TopK 5/10/15 at the reported 200 scene particles, then repeat at a proposed 400 particles to test sensitivity beyond the published setting. Record held-out MSE, peak GPU memory and rollout latency with identical measurement procedures. Declare all graph-distance settings; on Cloth, separately repeat the published MaxDist 0.3, 0.7 and end-effector-to-all variants. If the accuracy–memory ordering persists after matching, the architectural account gains support; if it disappears, implementation or resource allocation is a plausible contributor. e05e08e11e14e17e18
8.3 Reading coverage
Visual audit: Visually inspected the title/author page; all ten figures, both tables, method and loss equations; main evaluation and limitations pages; and Appendix A–C setup, diagnostics and simulation control pages. The six final original crops were individually inspected, including the corrected Table 1 crop. Figure 2’s residual path was checked against Equations (4)–(5), and Figure 3’s attending/attended axes against its caption and Section 4.2. Table 1’s CD exceptions are preserved despite Section 4.3’s broader wording. All retained method, numerical, hardware, training, evaluation and proposed-check premises are supported by these declared pages. Reference-only pages 10–12 were read as text; external references and linked videos were not opened.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15, 16, 17. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related work
- 3 Method: 3.1 Problem Formulation; 3.2 Transformer-Based Neural Dynamics Learning; 3.3 Hybrid Supervision; 3.4 Model-Based Planning and Control
- 4 Experiment: 4.1 Setup and Task Design; 4.2 Qualitative Evaluation; 4.3 Quantitative Evaluation; 4.4 Model-Based Planning and Control
- 5 Conclusion; Limitations; Acknowledgment
- References
- Appendix A: A.1 Setup and A.2 Task Design
- Appendix B: B.1 Dynamics Prediction Comparison and B.2 Hyperparameter Analysis
- Appendix C: Additional Results for Model-Based Planning and Control
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only figure limitation was addressed by visually inspecting all ten figures and both tables in the supplied PDF. Reference-only pages 10–12 were read as text, without a separate visual pass.
- The complete supplied 17-page CoRL 2025 paper and its internal appendices were read. No separate supplements, linked videos, code, or external references were inspected; no experiments were reproduced.
- Identity/version scope: the title and all five authors match the catalog. Page 1 identifies CoRL 2025, Seoul, Korea. No numbered revision or revision date is established by the supplied PDF; no comparison with another edition is claimed.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author block and conference footer
Title and authors match the catalog: Suning Huang, Qianzhong Chen, Xiaohan Zhang, Jiankai Sun and Mac Schwager. Zhang is at RAI Institute; the other authors are at Stanford University. The footer identifies CoRL 2025, Seoul, Korea.
Go to primary source ↓e02PDF pp. 1–3, Abstract and Section 1; Figure 1, p. 2
The motivation contrasts graph-neighborhood sensitivity and tracked-particle supervision with a Transformer point-cloud model and hybrid geometric supervision for multi-material interactions.
Go to primary source ↓e03PDF p. 3, Figure 2 and caption; p. 4, Section 3.2, Observation Embedding
Stereo reconstruction and segmentation generate particle states; FoundationStereo, GroundingDINO and Segment Anything are named. Future stereo-derived states supervise predicted geometry.
Go to primary source ↓e04PDF p. 4, Sections 3.1–3.2, Equations (1)–(2)
Inputs comprise object/end-effector points, externally applied motion and one-hot object material codes. Object motion inputs and end-effector material codes are zero; end-effector motion is the current-minus-previous position. A projector embeds concatenated features.
Go to primary source ↓e05PDF p. 4, Section 3.2, Dynamics Transition, Equation (3); p. 5, Equations (4)–(5)
Three Transformer encoder layers provide all-particle self-attention without added positional encoding. A shared decoder predicts each particle displacement and adds it to the current position.
Go to primary source ↓e06PDF p. 5, Section 3.3, Equation (6) and Training paragraph
The weighted CD/HD approximation loss supervises five-step autoregressive training without point correspondences. Alpha is constrained to [0,1], but its selected value and the HD approximation formula are absent.
Go to primary source ↓e07PDF p. 5, Section 3.4; p. 8, Section 4.4
MPPI samples action sequences over horizon H and evaluates world-model rollouts with target, feasibility and collision costs. The experiments use normalized CD+HD for target matching. Numerical planner settings are not given.
Go to primary source ↓e08PDF p. 6, Section 4.1, task and Baselines paragraphs
Real tasks are Box Pushing, Cloth Gathering and Rope Sweeping; the latter tests a changed rope/granular arrangement. GBND is retrained with hybrid loss, the ablation uses CD only, and DINO-WM is a 2D scene model.
Go to primary source ↓e09PDF p. 7, Table 1, all metric rows and task columns; Section 4.3
MSE for Ours is 0.0023/0.0013/0.0008 versus 0.0042/0.0037/0.0013 without hybrid loss and 0.0076/0.0069/0.0029 for GBND. Combined error favors Ours, but CD favors the ablation for Cloth Gathering (0.082 versus 0.096) and Rope Sweeping (0.061 versus 0.067). DINO-WM uses Depth Anything V2 reconstruction and has no MSE entry.
Go to primary source ↓e10PDF p. 6, Figure 3 and caption; pp. 6–7, Section 4.2
One-step prediction examples cover box, cloth/granular and rope/granular dynamics. The caption defines attention from row i to column j; the axes label attending and attended particles. The authors interpret rope-related attention structures as learned interactions.
Go to primary source ↓e11PDF p. 7, Figure 4 and caption; pp. 7–8, Section 4.3
GBND error decreases as TopK rises from 5 to 10 to 15 but remains above ParticleFormer in the three plotted multi-material tasks. The sweep changes maximum allowed adjacent nodes.
Go to primary source ↓e12PDF p. 8, Section 4.4 and Figure 5 with caption
Real-robot MPPI rollouts target unseen configurations; blue dashed regions show initial states and green regions targets. The full-model curves end lowest on normalized CD+HD. The caption mentions three rollout trials but does not define the shaded bands.
Go to primary source ↓e13PDF p. 9, Limitations paragraph
Models are trained per scene; broad environment/robot generalization is unshown. External object-mask failures may propagate to dynamics predictions.
Go to primary source ↓e14PDF p. 13, Appendix A.1, Setup, Simulation and Real World
A100 training and RTX 3090 control are reported. Simulation collects 1,000 episodes per task. Real experiments use xArm-6 arm(s), a ZED-2i at 10 Hz and 1280×720, approximately 70×55 cm workspace and 500-second episodes; the number of real episodes is not stated.
Go to primary source ↓e15PDF pp. 14–15, Appendix A.2, Task Design; p. 13, Figure 6
Six simulation tasks include Rope, Granular Object, Cloth, Cloth Gathering, Rope Maneuver and Rope Sweeping. Rope Maneuver uses two arms and a rope to indirectly move a rigid box; the real tasks are Box Pushing, Cloth Gathering and Rope Sweeping.
Go to primary source ↓e16PDF p. 15, Appendix B.1, Table 2, all rows
For Rope/Granular Object/Cloth, Ours has MSE 0.00017/0.00152/0.00160, the CD-only ablation 0.00025/0.00173/0.00182 and GBND 0.00043/0.00196/0.00194. Ours wins MSE and CD+HD; the ablation wins CD in all three columns.
Go to primary source ↓e17PDF p. 15, Appendix B.2, opening paragraph; p. 16, Figure 8 and caption
Memory comparison uses input clusters of 200 scene particles. GPU memory in MB rises with GBND TopK; ParticleFormer uses less memory than all three plotted GBND settings across the tasks. No particle-count scaling sweep is reported.
Go to primary source ↓e18PDF p. 16, Appendix B.2, MaxDist paragraph and Figure 9
On Cloth, GBND is compared with end-effector-to-all-cloth connections and MaxDist 0.7 or 0.3. Error depends strongly on these settings; ParticleFormer is lower in the plotted comparison.
Go to primary source ↓e19PDF pp. 16–17, Appendix C and Figure 10 with caption
Qualitative simulation control sequences cover Cloth Gathering, Rope Maneuver and Rope Sweeping toward novel target states marked in red. They compare Ours, GBND and Ours without hybrid loss without a numerical control-results table.
Go to primary source ↓8.5 Primary sources
ParticleFormer: A 3D Point Cloud World Model for Multi-Object, Multi-Material Robotic Manipulation ↗
PDF · 7,292 extracted words
Source fingerprint
38f2a627c3c36658ffb6d966882b1b6800d67de33c24c05d99e39669dc2ad18c