PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation
1. Paper overview
In one sentence: PointWorld makes robot motion a geometric input to fast 3D scene prediction, enabling planner-based manipulation while depending on calibrated observations and accurately realized robot trajectories. e-modele-coste-robot-resultse-deploymente-limitations
| At a glance | What to know |
|---|---|
| Research problem | Source description Manipulation requires predicting contact-driven motion from partial observations across changing scenes and robot geometries. PointWorld seeks a reusable dynamics model whose action interface carries physical contact geometry, so heterogeneous robot datasets can supervise the same prediction task without a task-specific reward or demonstration policy. e-probleme-model |
| Core mechanism | Source description A shared 3D representation turns robot motion into geometric conditioning for dense scene-flow regression, with ten future steps predicted together. e-model |
| A key reported result | Zero-shot physical manipulation across eight tasks: Tissue box 70; book 20; scarf 80; pillow 40; microwave 30; drawer 90; duster 60; broom 60. Task success rate (%).. One jointly pretrained model, single RGB-D capture, manual goals, ten feasible randomized configurations per task; no post-training or replanning. No competing physical controller baseline is reported; unsafe plans count as failures. This establishes executed behavior across task families, with substantial task-dependent failures. Pre-grasped deformable/tool setups exclude grasp acquisition; confidence intervals and numerical success tolerances are not provided. e-robot-resultse-deployment |
| Reading caution | Source description The static initial input omits scene velocity; robot flows assume rigid links and realized trajectories. Contact-induced tracking errors or compliant gripper deformation can violate that abstraction. Photometric changes and explicit conservation constraints are absent. e-limitations |
Core contributions
- Source description
A shared 3D representation turns robot motion into geometric conditioning for dense scene-flow regression, with ten future steps predicted together. e-model
- Source description
The paper combines a large annotation pipeline, backbone/objective/observability ablations, transfer experiments and zero-shot physical manipulation with manually specified goals. e-real-datae-sim-datae-roadmape-viewse-transfere-robot-resultse-deployment
Figure 2. Known gripper motion conditions a learned forecast of scene motion. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top half as input construction: the RGB-D observation supplies scene geometry, while joint actions and the URDF supply the magenta robot trajectories. Both occupy the concatenated cloud. The lower half separates their features before the PTv3 backbone produces scene flow. The DINOv3 box is shorthand: the text specifies projecting scene points into calibrated RGB views and sampling frozen image features, not running an image encoder directly on 3D coordinates. Temporal embeddings distinguish robot points at different future times. Figure 2 and Section 3.1 agree that robot motion is given to the model; the forecasted output is the scene response. e-modele-featurese-coste-limitations
What it supports. The architecture makes candidate actions geometrically comparable across robot embodiments while preserving their contact surfaces. One forward pass predicts ten future steps, each spaced by 0.1 seconds. This provides the dynamics function that a planner can repeatedly query; the diagram does not depict a learned policy selecting actions.
Where the evidence stops. The initial scene is assumed static, and only observed scene points are propagated. Robot geometry follows forward kinematics as if the proposed trajectory were realized; contact-induced tracking error and gripper deformation are outside that assumption.
2. Motivation
2.1 The problem and the proposed response
Manipulation requires predicting contact-driven motion from partial observations across changing scenes and robot geometries. PointWorld seeks a reusable dynamics model whose action interface carries physical contact geometry, so heterogeneous robot datasets can supervise the same prediction task without a task-specific reward or demonstration policy. e-probleme-model
2.2 What this reading follows
Imagine deciding how to fold a scarf from one RGB-D capture. PointWorld turns that problem into predictions about where the observed surface points will move for each contemplated gripper path. Its distinctive choice is to express both the scene and the robot's future motion as points in the same 3D space. A pretrained image encoder supplies visual features, a point-cloud network forecasts a short motion chunk, and a separate optimizer selects a path toward manually specified targets. The figures below trace this information flow, the evidence for its design choices, and the substantial gap between accurate short-horizon prediction and reliable physical execution. e-modele-coste-robot-resultse-deploymente-limitations
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system / Other mechanisms / Outside quadrants classification is supported at the prediction-control level: one learned forward dynamics model evaluates proposed actions, while a separate sampling optimizer selects executable trajectories. Shared 3D state/action inputs do not make this joint future-and-action generation. No learned inverse-dynamics decoder is used; uncertainty primarily regularizes training. e-modele-coste-losse-deployment
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn a candidate action into contact geometry
Begin with a calibrated RGB-D observation and a robot description. Back-project non-robot pixels into scene points, then propagate sampled gripper surface points along a proposed joint trajectory using forward kinematics. The resulting robot flows tell the model where the gripper would be, even where the camera cannot see it. PointWorld combines this temporal geometry with the initial scene cloud, frozen DINOv3 image features and raw geometric features. PTv3 then predicts displacements for the observed scene points. This is an action-conditioned forward model: robot motion is a condition, while scene motion is the prediction. A separate planner must still propose useful actions. Its geometric interface enables different embodiments to share a network, but the rigid-link and realized-trajectory assumptions remain part of that interface. e-modele-featurese-coste-limitations
5.2 Make noisy movement supervision usable
The training problem is unbalanced because most scene points remain static. Movement weighting uses reference displacements to increase the influence of moving points, but noisy real-world labels can then become unusually influential. The Huber residual and learned log-variance address that second problem: the residual is scaled by predicted inverse variance, while an additive log-variance term discourages assigning unlimited uncertainty. Real correspondences must also pass visibility and depth-validity masks. Simulation needs a different treatment. With nearly zero residuals, freely learned variance can collapse toward zero and produce unstable gradients, so the appendix replaces simulated uncertainty with a constant. Reader interpretation: these mechanisms concern learning from mixed-quality supervision; they do not by themselves constitute a calibrated measure of downstream planning risk. e-losse-featurese-sim-variance
5.3 Understand exactly what the planner demonstrated
For deployment, the user selects scene points and target positions. MPPI samples end-effector trajectories, converts them into robot flows and rolls out their predicted scene consequences. Candidate costs combine point-to-goal distances with control regularization, then weighted updates refine the trajectory. The reported implementation plans 30 steps through three model chunks and sends the selected trajectory through inverse kinematics and a robot controller. Crucially, Appendix A.6.1 says the experiments do not replan. Reader interpretation: the success rates test whether a single observed scene and an optimized path suffice under the chosen initial configurations; they do not test recovery after an unexpected disturbance. Manual goals and pre-grasped starts also narrow the autonomy claim, even though no task-specific model finetuning is required. e-coste-deploymente-robot-resultse-limitations
5.4 Training and inference
During training
The curated real/simulated corpus totals approximately two million trajectories and 500 hours. DROID supplies pseudo-labels through FoundationStereo depth, robot-aligned extrinsic refinement and CoTracker3 correspondences lifted to 3D. Occluded or invalid real points receive no supervision. BEHAVIOR-1K supplies simulator-derived link trajectories. Its appendix includes negative clips with robot motion but no object motion, qualifying the main text's contact-and-motion-only summary. e-probleme-real-datae-sim-datae-features
Movement weighting emphasizes the small moving subset, while Huber residuals and predicted uncertainty temper noisy labels. Synthetic samples use constant uncertainty to avoid variance collapse. Preprocessing uses 1.5 cm voxels and up to 12,000 scene points, with geometric/color augmentation disabled at evaluation. e-losse-sim-variancee-features
The 411M main-experiment configuration reports AdamW at 0.0001, 200 epochs, batch 176 and seven days on eight H100s. The 1B configuration reports 300 epochs, batch 1,920 and 20 days on 128 H100s. These are distinct training budgets; both keep the image encoder frozen. e-traininge-features
During inference
One dynamics pass covers ten 0.1-second steps without a separate point tracker. Physical planning spans 30 steps using three chunks, with 256 samples per refinement and 20 refinements. Full planning takes a few seconds. Although framed as MPC, the demonstrated procedure does not replan. GUI/SAM2 goals, PyBullet inverse kinematics and the Deoxys controller complete execution; deformable and tool-use tasks begin pre-grasped. e-modele-coste-deploymente-robot-results
5.5 Implementation flow
- Construct scene and action geometry
Mask robot pixels and back-project the remaining RGB-D observations. Sample robot surface points once, attach them to rigid links, and propagate them with forward kinematics. The practical action representation concentrates points on grippers. Scene observations remain partial, while contemplated robot geometry can include hidden contact surfaces. e-model
- Fuse features and predict a chunk
Concatenate the initial scene cloud with robot points stacked across time. Frozen DINOv3 features describe scene appearance; temporal embeddings mark robot motion. Raw features additionally include normals, gripper openness, robot velocity/acceleration and distances to robot points. PTv3 and a shared MLP predict scene displacements for the complete chunk. e-modele-features
- Separate prediction from action selection
MPPI generates candidate end-effector paths, converts each into robot flows and scores PointWorld rollouts against task and control costs. It refines a nominal trajectory using exponentiated negative-cost weights. The model forecasts consequences; it does not jointly emit an action policy or infer actions through inverse dynamics. e-coste-deployment
6. Experiments & results
PointWorld forecasts how observed scene points move under a proposed robot trajectory. It places RGB-D scene geometry and forward-kinematic gripper motion in one point-cloud representation, then predicts a short motion chunk with a pretrained visual encoder and learned dynamics backbone. Large-scale real/simulated training improves prediction and supports physical manipulation through a separate planner. The central tradeoff is explicit geometric action conditioning versus assumptions about static initial scenes, reliable calibration and accurately realized robot motion.
6.1 Read the original evidence
Table 1. PTv3 scaling improves point-flow accuracy with a measurable latency cost. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the final two columns: they separate errors on moving and static points, measured in meters under the paper's evaluation protocol. The latency column is in milliseconds; parameter, memory and FLOP columns are ratios to GBND, not absolute resource counts. Compare the four PTv3 rows to see scaling within one backbone family, then compare against the other architectures. A smaller mover error alone is insufficient to judge a model that also disturbs background points, which is why the static column matters. These rows are Table 1's backbone comparison; Figure 7's progressive recipe is a different sequence and should not supply substitute baseline numbers. e-backbonese-evaluatione-roadmape-traininge-deployment
What it supports. PTv3-50M reports 0.0331 m mover error, improving to 0.0312 m for PTv3-1B; GBND reports 0.0390 m. The 1B model also reduces static error to 0.0056 m. Its 123.65 ms forward latency exceeds GBND's 13.46 ms, so accuracy gains come with a runtime tradeoff.
Where the evidence stops. The DROID confidence filter has conflicting descriptions on pages 8 and 29. Forward-pass latency also excludes the full sampling optimizer: physical planning takes a few seconds. Training hardware counts on page 28 do not establish the timing setup for this table.
Table 2. Real-world pretraining transfers to a held-out lab; simulation alone leaves a larger zero-shot gap. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Decode the arrows before comparing cells: D is DROID, B is BEHAVIOR-1K, and H is the held-out real CLVR lab. An arrow indicates the pretraining domain followed by the evaluation domain. The upper block is zero-shot evaluation, while the lower block permits adaptation. The gray From Scratch column is a specialist trained on the held-out lab, not another zero-shot model. For H, the lab data have a 90% training and 10% test split, and the finetuned variants use only the training subset. Each adaptation run uses one twentieth of the original training updates. Compare mover and static rows separately within a common target domain. e-transfere-evaluation
What it supports. For held-out real scenes, mixed DROID+B1K pretraining gives 0.0300 m zero-shot mover error, close to DROID-only at 0.0305 m and the specialist at 0.0293 m. After adaptation, DROID-only and mixed models reach 0.0271 and 0.0272 m. B1K-only starts substantially worse at 0.0531 m.
Where the evidence stops. Mixed pretraining does not improve every metric: its zero-shot held-out static error is 0.0063 m versus 0.0049 m for DROID-only. Reduced adaptation updates are not a comparison of total training cost, and the confidence-filter discrepancy still applies to real-data scores.
Figure 8. Physical trials demonstrate diverse behaviors, with strongly uneven success across tasks. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the bars as separate task-level success rates, not a continuous scale of task difficulty. The photographs illustrate pushing, deformable manipulation, articulated-object interaction and tool use; their arrows indicate intended motion rather than feedback or policy architecture. Appendix A.6 specifies ten randomly sampled, kinematically feasible initial configurations per task. A human restores each scene and starts execution; unsafe optimized plans count as failures. The robot uses one RGB-D capture and a jointly pretrained model with manual scene masks and target positions. Deformable and tool-use trials begin pre-grasped, so their rates evaluate the subsequent manipulation rather than autonomous grasp acquisition. e-robot-resultse-deploymente-open-details
What it supports. Drawer closing reaches 90% and scarf manipulation 80%, while book pushing and microwave opening reach 20% and 30%. Tissue-box pushing is 70%, pillow placement 40%, and both tools 60%. These are real executions using the learned dynamics model, showing a broader capability than motion-prediction accuracy alone, but also substantial failures.
Where the evidence stops. Despite the MPC wording, Appendix A.6.1 explicitly says no replanning was performed. Each rate comes from ten trials, without reported confidence intervals or a competing physical controller baseline; detailed numerical success tolerances are also absent.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| DROID held-out point-flow prediction and backbone scaling One-second prediction; shared expert-filtered DROID evaluation, with mover/static subsets defined from reference motion. | PTv3-1B: 0.0312/0.0056 m, 123.65 ms; PTv3-411M: 0.0315/0.0059 m, 102.47 ms. Mean mover/static Euclidean error (m); forward-pass latency (ms). | GBND: 0.0390/0.0066 m, 13.46 ms; PTv3-50M: 0.0331/0.0067 m, 59.60 ms. Larger PTv3 models improve prediction while costing latency. These are model-forward timings, not end-to-end planning times. Figure 7's progressive recipe uses different configurations. e-backbonese-evaluatione-roadmape-deployment |
| Transfer to held-out CLVR real-world scenes CLVR excluded from pretraining; its 90%/10% split supports finetuning and testing. Finetuning uses one twentieth of original updates. | DROID-only zero-shot/finetuned: 0.0305/0.0271; DROID+B1K: 0.0300/0.0272. Mean mover Euclidean error (m), lower is better. | B1K-only: 0.0531/0.0299; specialist trained from scratch: 0.0293. Mixed pretraining modestly improves zero-shot mover error, but does not beat the specialist until finetuned. Fewer adaptation updates do not mean less total pretraining compute. e-transfere-evaluation |
| Cross-domain real/simulation point-flow transfer DROID-to-BEHAVIOR-1K and BEHAVIOR-1K-to-DROID evaluation on unseen destination samples; adaptation uses one twentieth of original updates. | DROID-to-B1K: 0.1460 zero-shot, 0.0107 finetuned; B1K-to-DROID: 0.0558 zero-shot, 0.0378 finetuned. Mean mover Euclidean error (m). | In-domain B1K and DROID models report 0.0087 and 0.0315 m respectively. Adaptation substantially narrows both gaps but does not equal the corresponding in-domain result. Real and simulated scores use different supervision quality and filtering, so their absolute values are not directly comparable. e-transfere-evaluation |
| Action representation and chunked-rollout ablations Figure 11 jointly trains on DROID+B1K; Figure 12 compares training/inference strategies across ten future steps. | Gripper-only flows have the lowest plotted errors in both domains. Matched chunk training/inference has the lowest late-horizon drift. Mover Euclidean error; qualitative ordering from plotted curves. | Whole-robot and low-dimensional actions perform worse; sliding-window and autoregressive rollout incur more drift. The plots support contact-focused action geometry and matched temporal training. Exact bar values are not tabulated; the dense full-robot point count conflicts between text and legend. e-action-ablatione-chunk-ablation |
| Zero-shot physical manipulation across eight tasks One jointly pretrained model, single RGB-D capture, manual goals, ten feasible randomized configurations per task; no post-training or replanning. | Tissue box 70; book 20; scarf 80; pillow 40; microwave 30; drawer 90; duster 60; broom 60. Task success rate (%). | No competing physical controller baseline is reported; unsafe plans count as failures. This establishes executed behavior across task families, with substantial task-dependent failures. Pre-grasped deformable/tool setups exclude grasp acquisition; confidence intervals and numerical success tolerances are not provided. e-robot-resultse-deployment |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 11. Concentrating action points on grippers performs best in the joint-training comparison. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the colored symbols to associate each bar with an action representation; they serve as method labels rather than a separate measurement series. Both panels report mover error, but their vertical scales differ, so compare bar ordering within a panel. The two low-dimensional baselines use end-effector poses or joint positions plus gripper openness, whereas flow-based variants explicitly place robot points beside scene points. All variants are trained jointly on DROID and BEHAVIOR-1K. The gripper-only representation concentrates spatial resolution near likely contacts. The full-robot alternatives spread points over inactive body regions or increase the point budget, changing what the network must process. e-action-ablation
What it supports. Gripper-only flows have the lowest plotted mover error on both datasets. In simulation, whole-robot flows still outperform low-dimensional actions; in DROID, whole-robot variants underperform those baselines. This supports the particular contact-focused representation under the reported joint-training setup, while the authors' explanations about sparse signals remain mechanistic interpretations.
Where the evidence stops. The dense full-robot count is unresolved: page 9 says 2,000 points, but this original legend says 3,000. Exact bar values are not tabulated. Joint-training comparisons alone do not isolate the benefit of cross-embodiment data sharing.
Figure 12. Training and inference should agree on how predicted states are reused. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each legend arrow as training strategy followed by inference strategy. AR means autoregressive; sf means self-feeding and tf means teacher forcing, which uses reference inputs during training. The green and blue curves reuse a chunk-trained model through sliding windows of one or five steps. The solid red curve predicts the full chunk in the same way during training and inference. The horizontal axis advances in 0.1-second steps and the vertical axis measures mover error. The informative comparison is how the curves separate later in the horizon: repeatedly feeding predictions back can magnify deviations from the input distribution used during training. e-chunk-ablatione-modele-deployment
What it supports. Matched chunk training and inference yields the lowest late-horizon error among these settings while needing one forward pass for the ten-step prediction. Reusing the chunk model one step at a time produces the largest drift; a five-step window recovers part of the loss. Thus, a fast chunk model is not automatically a reliable single-step simulator.
Where the evidence stops. The plot covers the trained one-second horizon. Physical planning chains three chunks over 30 steps, and the source does not provide an equivalent long-horizon drift curve here. This ablation does not establish robustness to arbitrary repeated rollout.
7. Analysis & limitations
7.1 What the evidence leaves open
The static initial input omits scene velocity; robot flows assume rigid links and realized trajectories. Contact-induced tracking errors or compliant gripper deformation can violate that abstraction. Photometric changes and explicit conservation constraints are absent. e-limitations
The authors explicitly limit the interpretation to predictive correlations under observed action-conditioned data, rather than recovered causal mechanisms. Thin-object calibration errors can corrupt supervision. e-limitations
DROID evaluation is conditioned on an expert-derived mask. Page 8 retains the top 80%, while page 29 rejects confidence below the 0.8 quantile, ordinarily leaving about 20% before voxel filtering. The supplied source does not resolve this conflict; exact benchmark replication depends on it. e-evaluation
Deployment evidence does not establish closed-loop recovery, automatic goal specification or superiority over another controller. Qualitative benefits of mixed training are not a controlled physical success-rate ablation. e-deploymente-robot-resultse-mixture
7.2 Questions for discussion
- Do architecture rankings persist under both interpretations of the expert-confidence filter?
- How much does closed-loop replanning improve failures caused by inaccurate contact or controller tracking?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires synchronized stereo streams, calibrated geometry/URDFs, the annotation pipeline, fixed train/test masks, DINOv3/PTv3 configurations and a matching controller stack. The PDF's release statements vary between future promise and present dataset availability; artifact availability remains unverified. e-real-datae-sim-datae-featurese-traininge-evaluatione-deploymente-release
Unresolved details include movement-weight parameters, several B1K filtering thresholds, exact DINO layer selection, mixture proportions and success tolerances. Resolve the 2,000-versus-3,000-point ablation label and confidence-filter conflict before asserting a faithful reproduction. e-open-detailse-action-ablatione-evaluation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether model rankings depend on the confidence-filter interpretation
Reader-proposed check, not performed: freeze one held-out DROID split, one expert model and identical evaluated predictions. Compare masks retaining the highest-confidence 80% against masks rejecting confidence below the 0.8 quantile, following the same voxel exclusion procedure. Add an unfiltered evaluation and a small independently checked correspondence subset. Report retained mover/static counts, error and model rankings for each mask. If rankings reverse or gains disappear on the checked subset, the reported improvements are sensitive to label selection; stable rankings would support a more robust conclusion. Resolve the source conflict before calling either implementation a faithful reproduction. e-evaluatione-backbones
Check 2: Separate model-prediction error from the absence of replanning
Reader-proposed check, not performed: repeat matched tissue-box and drawer trials with the same checkpoint, manual goals, feasible initial configurations, cost terms and controller. Compare the reported single-plan execution against a variant that reacquires RGB-D and warm-starts MPPI after each executed chunk. Record total planning time, success, scene-flow error and commanded-versus-measured robot motion; retain unsafe plans as failures. Test both undisturbed scenes and predefined small object displacements. Improvement after disturbances would support feedback as a remedy for stale predictions. Persistent failures with large robot tracking deviations would instead challenge the assumption that commanded geometry is realized. e-deploymente-robot-resultse-limitationse-chunk-ablation
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block; all main figures and both quantitative tables; appendix limitations, annotation equations and filtering, feature/training tables, uncertainty treatment, evaluation and deployment details; and Figures 14–21. All six final original-PDF crops were separately viewed with labels and legends intact. Figure 2's input/output connections were checked against Sections 3.1 and A.4; Figure 11's 2,000/3,000-point discrepancy, the confidence-filter conflict and the no-replanning deployment qualification are disclosed. All 12 text chunks were read. Reference-only pages 13–19 and contents page 20 were not rendered. No linked interactive visualizations, separate supplements, code or experiments were inspected or executed.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work
- 3. Method (3.1–3.2)
- 4. Dataset Curation and Evaluation Protocol
- 5. Experiments (5.1–5.4)
- 6. Conclusion
- Acknowledgments and References
- A.1. Extended Discussions on Limitations
- A.2. DROID 3D Annotation Pipeline (A.2.1–A.2.4)
- A.3. BEHAVIOR-1K Data Generation (A.3.1–A.3.3)
- A.4. Model Training Details
- A.5. DROID Evaluation Protocol
- A.6. Real-Robot Experiment Details (A.6.1–A.6.4)
- A.7. Additional 3D Annotation Examples
- A.8. Additional Model Rollouts
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Version scope: arXiv:2601.03782v1, 7 January 2026, verified on the title page; observed title and all seven authors match the catalog. No other revision or edition was supplied for comparison.
- The acquisition notes that text extraction does not reconstruct figure images; this was addressed by inspecting the retained PDF, including all figures and tables.
- Separate supplemental material availability has not been fully verified; none was supplied. Linked interactive visualizations and release artifacts were not inspected.
- Code was not inspected and experiments were not reproduced. All 12 supplied text chunks were read; PDF pages 1–12 and 21–38 were visually inspected. Reference-only pages 13–19 and the appendix contents page 20 were read as text.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block and arXiv margin
The title and seven authors match the catalog. The artifact identifies arXiv:2601.03782v1 [cs.RO], 7 January 2026, and affiliations Stanford University and NVIDIA.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section 1
PointWorld learns action-conditioned 3D scene dynamics from RGB-D and robot geometry; the authors report approximately two million trajectories and 500 hours spanning DROID and BEHAVIOR-1K.
Go to primary source ↓e-modelPDF pp. 3–4, Figure 2 and Section 3.1, State Representation, Action Representation and Dynamics Prediction
A static scene cloud and time-stacked forward-kinematic robot points share one point-cloud input. Frozen DINOv3 scene features and temporal robot embeddings feed a backbone and shared displacement head. A forward pass predicts ten steps at 0.1 seconds per step; inference requires no separate tracker.
Go to primary source ↓e-lossPDF pp. 4–5, Training Objective, Figure 4 and Equation (1)
Ground-truth motion defines normalized sigmoid movement weights. The weighted Huber residual is multiplied by exponentiated negative predicted log-variance, with an additive log-variance regularizer. Unreliable occluded correspondences are excluded.
Go to primary source ↓e-costPDF p. 5, Section 3.2 and Equation (2)
MPPI perturbs candidate end-effector trajectories, converts them to robot flows, predicts scene flows, and weights candidates exponentially by cost. Task cost is mean squared distance of selected scene points to specified targets; control cost includes path length and reachability.
Go to primary source ↓e-real-dataPDF pp. 6 and 8, Section 4; pp. 22–23, Sections A.2.1–A.2.4
DROID annotations use FoundationStereo depth, VGGT-initialized extrinsics refined against robot geometry, and CoTracker3 tracks lifted to 3D. Visibility and depth-validity flags accompany flows; DBSCAN rejects spatial outliers. The main text reports recovery of over 60% of DROID, nearly 200 hours.
Go to primary source ↓e-sim-dataPDF p. 8, Simulation paragraph; p. 24, Sections A.3.1–A.3.3 and Equation (3)
Simulation supervision uses local visible link points and known rigid-link trajectories; privileged state is not exposed to the model. Contrary to the main text's contact-and-motion-only shorthand, Equation (3) also retains specified robot-motion clips without object motion and excludes trunk/arm collisions.
Go to primary source ↓e-featuresPDF pp. 25–26, Tables 3–4 and Section A.4
Preprocessing includes 1.5 cm voxels, a 12,000 scene-point cap, and geometric/color augmentation, disabled for evaluation. Raw features include normals, robot velocity/acceleration, gripper openness and scene-to-robot distances. Frozen DINOv3 ViT-L/16 multilayer features are averaged over depth-consistent views.
Go to primary source ↓e-sim-variancePDF p. 27, Section A.4, Aleatoric Uncertainty on Simulation Data
Near-zero synthetic residuals can drive predicted variance toward zero and destabilize mixed training. The uncertainty head is learned on real samples; simulated log-variance is replaced by a batchwise constant related to real-sample variance, or a small fixed value for simulation-only training.
Go to primary source ↓e-trainingPDF p. 28, Tables 5–8; p. 26, Training Configuration
The 1B model uses AdamW, learning rate 0.0001, 300 epochs, global batch 1,920, 128 H100 GPUs and 20 days. The main-experiment 411M configuration uses 200 epochs, batch 176, eight H100 GPUs and seven days. Both list weight decay 0.01, gradient norm cap 5, Huber delta 5.0 and ten-step prediction.
Go to primary source ↓e-evaluationPDF p. 8, Model Evaluation Protocol and Interpretation of the Metric; p. 29, Section A.5
Evaluation averages per-point Euclidean position error across timesteps, separately for movers and static points. DROID uses shared masks from an expert trained only on evaluation data, without sharing parameters or training data with evaluated models. Page 8 says retain the top 80%; page 29 says reject confidences below the 0.8 quantile. The retained fraction is unresolved. Means are reported; the authors state standard errors at most 0.00001 m.
Go to primary source ↓e-backbonesPDF p. 9, Table 1, GBND and PTv3 rows; Section 5.1
Mover/static errors are 0.0390/0.0066 m for GBND, 0.0331/0.0067 for PTv3-50M, 0.0315/0.0059 for PTv3-411M and 0.0312/0.0056 for PTv3-1B. Latencies are 13.46, 59.60, 102.47 and 123.65 ms respectively. Parameter, memory and FLOP columns are relative to GBND.
Go to primary source ↓e-roadmapPDF pp. 8–9, Figure 7 and Section 5.1; p. 9, Figure 9; p. 11, scaling paragraph
Figure 7 is a progressive design sequence with unadopted hatched variants; it differs from Table 1's configurations. Figure 9 separately sweeps data fraction from 5% to 100% and capacity from 50M to 1B, supporting the authors' approximate log-linear scaling description within these ranges.
Go to primary source ↓e-transferPDF p. 10, Table 2; p. 11, Section 5.3
DROID-to-held-out-CLVR zero-shot mover error is 0.0305 m, mixed DROID+B1K is 0.0300, B1K-only is 0.0531, and the specialist is 0.0293. Finetuned values are 0.0271, 0.0272 and 0.0299 respectively. CLVR is split 90% training and 10% testing; finetuning uses one twentieth of original updates. Cross-domain D-to-B changes from 0.1460 to 0.0107 m; B-to-D from 0.0558 to 0.0378 m.
Go to primary source ↓e-action-ablationPDF pp. 9–10, Section 5.2, Action Representations; p. 10, Figure 11 and legend
Joint DROID+B1K training compares gripper-only flows, sparse/dense whole-robot flows, end-effector poses and joint positions. Gripper-only has the lowest mover error in both plotted domains. Page 9 describes the dense full-robot baseline as 2,000 points, whereas the Figure 11 legend labels it 3,000 points.
Go to primary source ↓e-chunk-ablationPDF p. 10, Section 5.2, Chunked Prediction; p. 11, Figure 12
The figure compares self-feeding and teacher-forced autoregressive training, chunk-trained sliding-window inference with W=1 or 5, and matched chunk training/inference over ten steps. Matched chunking has the lowest late-horizon error and needs one forward pass instead of two to ten.
Go to primary source ↓e-viewsPDF pp. 10–11, Section 5.2, Partial Observability and Figure 13
Models trained with one, two, three or randomized camera counts are evaluated at all three counts. Randomized-view training gives the lowest plotted errors across evaluation settings, supporting robustness to varying observed coverage.
Go to primary source ↓e-robot-resultsPDF p. 9, Figure 8; pp. 11–12, Section 5.4; p. 30, Sections A.6.2–A.6.3
Reported physical success rates are tissue box 70%, book 20%, scarf 80%, pillow 40%, microwave 30%, drawer 90%, duster 60% and broom 60%. Each task has ten feasible randomized initial configurations; unsafe plans count as failures. Deformable and tool-use tasks start pre-grasped.
Go to primary source ↓e-deploymentPDF p. 30, Section A.6 and Sections A.6.1–A.6.2
A Franka with unseen fin-ray gripper geometry and one RealSense D435 uses FoundationStereo depth, GUI/SAM2 masks and world-frame targets. MPPI uses 256 samples per iteration, 20 iterations, a 30-step horizon, beta 0.05 and EMA 0.9. Planning takes a few seconds, and no replanning is performed. PyBullet IK targets are followed by a Deoxys joint-impedance controller at 20 Hz.
Go to primary source ↓e-limitationsPDF p. 21, Section A.1, all eight limitation paragraphs
The authors assume a static initial scene, rigid kinematic robot geometry and realized commanded trajectories. They acknowledge manual reward specification, calibration errors for thin objects, correlation rather than causal identification, no photometric dynamics and no explicit physical conservation constraints.
Go to primary source ↓e-mixturePDF p. 30, Section A.6.4
Real-only predictions are qualitatively conservative; simulation-only predictions can move clutter with targets. Mixed training is described as more balanced, while a controlled deployment study of real/simulation proportions remains future work.
Go to primary source ↓e-open-detailsPDF pp. 5, 24, 26–30, Equations (1), (3), Section A.4 and Sections A.5–A.6
The supplied text leaves movement-weight parameters, several simulation clip thresholds, exact DINO layer selection, numerical success tolerances and a fully specified real/simulation mixture unresolved. The confidence-filter discrepancy also prevents unambiguous implementation of evaluation.
Go to primary source ↓e-qualitativePDF p. 7, Figure 6; pp. 31–38, Figures 14–21 and captions
Additional figures compare reconstructed DROID annotations and predicted versus reference rollouts for real and simulated interactions. Figure 6 identifies green reference points as occluded tracker regions and states that point-cloud visualizations are nearest-neighbor upsampled from 1.5 cm grid sampling.
Go to primary source ↓e-releasePDF p. 1, final Abstract sentence; p. 2, Contributions; p. 6, Section 4
The abstract promises future open-sourcing of code, data and checkpoints, while body text describes the dataset as open-sourced. The PDF alone does not establish the actual release state.
Go to primary source ↓8.5 Primary sources
PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulation ↗
PDF · 19,525 extracted words
Source fingerprint
368ee87a0aeb2b784026d4eb0bd47e38937759473a2762f0ac338f9ad06a4b08