Evaluating Real-World Robot Manipulation Policies in Simulation
1. Paper overview
In one sentence: SIMPLER makes simulated evaluation useful for selecting real robot policies by matching visual inputs and calibrating action effects, while accepting that absolute success and difficult contact dynamics may still differ. e02e04e05e10e11e12e15e23
| At a glance | What to know |
|---|---|
| Research problem | Source description Real robot evaluation becomes expensive and difficult to standardize as policies cover more tasks. SIMPLER seeks a scalable improvement signal: preserve relative performance without requiring a full-fidelity digital twin or replacing physical evaluation. e02 |
| Core mechanism | |
| A key reported result | Google Robot policy ranking: three-task protocol comparison: Visual Matching: 0.056 / 0.924. Average MMRV ↓ / Pearson r ↑. Table 1: six checkpoints across Pick Can, Move Near and Drawer; matched real/sim evaluation. Variant Aggregation: 0.143 / 0.778; action MSE: 0.375 / 0.308. Best printed averages favor Visual Matching. These are not success rates; RT-1 MSE samples training data, and Table 10 changes drawer values. e07e10e20 |
| Reading caution | Source description Rigid-object tasks, RGB/proprioception, fixed-camera green screening, imperfect shadows and manual scene curation limit scope. Preliminary Gaussian-splatting scenes were inadequate; contact-free calibration may fail for fast collisions or dexterity. e23 |
Core contributions
Section 4.2, unnumbered visual-matching illustration. Matching what the policy sees requires separate treatment of background and foreground. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from left to right: the real scene, raw simulation, background replacement, then background replacement plus texture matching. The rows illustrate tabletop objects, an articulated drawer and a WidowX block setup. In Section 4.2, the foreground mask selects the simulated robot and interactable objects, while the complementary mask selects the cleaned real background. This explains why the cabinet and robot can still look wrong after the surrounding scene improves. The last column additionally changes foreground appearance using aligned real-image textures. These are alternative observation constructions for evaluation; the panels do not depict a learned policy progressively improving during training. e04e05e09e23
What it supports. The image makes the division of labor concrete: background compositing restores much of the surrounding appearance, while foreground curation addresses robot and object mismatch. The paper combines this visual treatment with a separately calibrated controller so both observations and action effects resemble the real setup.
Where the evidence stops. Visual resemblance alone does not establish ranking accuracy or correct contact physics. Appendix G limits green screening to fixed cameras and acknowledges missing shadow fidelity; manual asset work also remains necessary.
2. Motivation
2.1 The problem and the proposed response
Real robot evaluation becomes expensive and difficult to standardize as policies cover more tasks. SIMPLER seeks a scalable improvement signal: preserve relative performance without requiring a full-fidelity digital twin or replacing physical evaluation. e02
2.2 What this reading follows
A simulator can render plausible manipulation scenes yet rank robot policies incorrectly. SIMPLER treats evaluation fidelity as a separate engineering problem: make a real-data-trained policy see comparable observations and make its actions produce comparable movements, then test the resulting rankings against physical rollouts. Its Google Robot and WidowX experiments support that strategy for rigid-object manipulation. The most revealing evidence combines a protocol comparison with component ablations and a simulated arm-texture sensitivity followed by a real intervention. Reading these results requires keeping task completion separate from grasping, correlation separate from calibration, and the main paper’s policy roster separate from its expanded appendix tables. e02e04e05e10e11e12e15e23
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator classification fits a physics-based evaluation suite for existing policies. No learned world/action architecture is proposed for One Model, joint future/action prediction or inverse-dynamics classification; the architecture, paradigm and quadrant are appropriately not applicable. e02e04e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Trace the observation–action loop before judging realism
Begin with the evaluated policy, whose behavior was learned from real demonstrations. During simulated evaluation, it receives a language instruction and current visual observations, with proprioception where applicable. Its output becomes a controller command that changes the simulated robot and objects, producing the next observation. A visual mismatch can therefore change the chosen action, while a control mismatch can change what the same action accomplishes. SIMPLER addresses both routes. Foreground masks retain simulated interaction geometry while real backgrounds and transferred textures improve observation appearance. Separate system identification adjusts action effects. Variant Aggregation offers a different evaluation choice: average over visually varied scenes instead of closely matching one real setup. Neither route trains a new world/action model inside the evaluation loop. e02e04e05e06e08e23
Figure 3. MMRV asks how costly a wrong ordering is in real performance. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the axes: horizontal position is real success and vertical position is simulated success. Each circle is a policy. In the first two panels, the red horizontal arrows span the real-performance differences of misordered pairs; their direction does not represent a robot action. Equation (1) weights a violated ordering by this horizontal margin, takes each policy’s largest violation, then averages. The third panel preserves the ordering despite a nonlinear relationship. The last panel shows why Pearson correlation can change greatly when policies have similar real performance. Read these as explanations of complementary metrics, not measurements from the robot benchmark. e03
What it supports. MMRV gives a small penalty when confused policies have nearly equal real success and a larger penalty when the simulator reverses a consequential performance gap. Pearson r remains useful for linear association, but the paper does not require a linear mapping to obtain a helpful model-selection signal.
Where the evidence stops. The panels are schematic. MMRV still depends on noisy real-success estimates and the chosen policy set; it is not a confidence measure. The printed formula uses strict comparisons, so a reproducer should preserve its treatment of ties.
5.2 2. Separate simulator calibration from policy learning
System identification uses 20 existing offline trajectories with different task instructions. The real action sequence is replayed open-loop from the same starting pose, and simulated end-effector positions and rotations are compared with the recorded trajectory. Three rounds of simulated annealing search stiffness and damping values. This calibration precedes the policy’s closed-loop evaluation. The appendix then matters: Google Robot actions pass through inverse kinematics and constrained joint motion planning at 3 Hz control with 501 Hz simulation; WidowX uses 5 Hz control with 500 Hz simulation and a different controller. These details determine the meaning of an action. The calibration ablation connects worse pose tracking with worse ranking fidelity, while Appendix G warns that contact-free trajectories may not calibrate high-speed collisions or dexterous contact. e04e08e16e23
5.3 3. Ask what a favorable correlation actually licenses
Reader analysis: the strongest interpretation is that SIMPLER supplies a useful relative-performance signal within its tested domains. MMRV asks whether an ordering mistake matters in real-success terms; Pearson r asks a different question about linear association. Neither requires exact success calibration. Figure 4 combines multiple task outcomes, including partial grasps, whereas Table 1 averages a particular three-task comparison. Table 2 adds OpenVLA, so its metrics describe another policy set. The arm-texture experiment supplies a more discriminating behavioral test because a simulated vulnerability motivates a physical intervention. Even there, the simulated range is not a confidence interval or a calibrated forecast. A reproduction should therefore retain policy rosters, separate task outcomes and measure absolute gaps alongside ranking metrics, rather than reporting a single favorable correlation. e03e10e11e12e15e22
5.4 Training and inference
During training
The main workflow evaluates existing RT-1 checkpoints, RT-1-X, RT-2-X and Octo. Simulator calibration fits parameters; it does not train a joint future/action model. A separate appendix experiment trains RT-1 solely on can-picking demonstrations. e04e07e21
During inference
The policy receives current observations/instructions and emits translation, rotation and gripper commands. Embodiment-specific controllers apply inverse kinematics and joint targets, yielding new simulated observations. RT-1 uses 3 Hz control/501 Hz simulation; WidowX uses 5/500 Hz. Drawer-apple instructions switch at termination or half the time limit. e07e08
5.5 Implementation flow
- Calibrate action effects
Replay identical actions from matched initial poses. Minimize translation and rotation discrepancy by tuning stiffness and damping on 20 existing offline trajectories with different instructions, using three narrowing rounds of simulated annealing. e04
- Construct policy observations
Replace the rendered background with a cleaned real image using foreground masks. Align real object images to meshes and transfer textures; tune robot colors. Variant Aggregation instead averages over changed backgrounds, lights, distractors and table textures. e05e06e09
- Run matched tasks
Match instructions, poses and success criteria. Planned Google trial counts are 75 for can picking, 60 for move-near, 54 for drawers and 27 for drawer-apple; Appendix A lists 24 per WidowX task. Average Google simulations over four arm colors and Octo simulations over three seeds. e07
- Score evaluation fidelity
Compare success rates using MMRV and Pearson r. Keep partial grasp success separate from completion, and retain each comparison’s policy roster and task aggregation. e03e10e11e13
6. Experiments & results
SIMPLER builds purpose-specific physics simulations to evaluate real-data-trained manipulation policies. It calibrates action execution and visual observations, then tests whether simulated successes preserve real policy rankings and robustness trends. Its strongest evidence concerns rigid-object tasks on Google Robot and WidowX; useful ranking signals coexist with absolute-success errors and reporting inconsistencies (e02, e10–e13, e23).
6.1 Read the original evidence
Figure 4. Ranking agreement appears across two embodiments, with visible absolute-success discrepancies. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the Google Robot and BridgeData V2 panels separately. Both plot measured real success horizontally and Visual Matching success vertically. The shape legends identify checkpoints; colors identify tasks. Bridge colors further distinguish intermediate grasping from completing the placement or stacking instruction, so a high grasp point is not necessarily a high task-success point. The dashed trend lines summarize association rather than proving equality of success probabilities. Use the printed MMRV and r values as Figure 4’s own summaries. Table 1 has a different task aggregation, and Appendix Table 2 contains an expanded policy roster, so those values should not be substituted into these panels. e07e11e12e13
What it supports. The figure prints MMRV/r of 0.049/0.929 for Google Robot and 0.014/0.890 for BridgeData V2. These support useful relative-performance correspondence within the evaluated settings. Individual points remain displaced from equal real and simulated success, and some task-specific comparisons are weaker than the combined picture.
Where the evidence stops. Table 2 adds OpenVLA beyond this legend. Real RT-1 Begin drawer testing was stopped after two trials, as disclosed on p. 20. Neither plotted correlations nor low MMRV establish that every checkpoint’s absolute success is calibrated.
Table 1. Visual Matching yields the strongest average ranking agreement in the main protocol comparison. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper and lower halves together: MMRV should decrease and Pearson r should increase. Rows represent alternative ways to assess the same six named checkpoints, while columns separate three Google Robot task groups and their average. The first row uses action prediction error as a proxy for policy quality; the other rows execute policies in simulation. For the simulation-free comparison, Appendix F.1 correlates negative MSE with real performance because smaller error is intended to mean better behavior. Do not read the average correlation as an average task-success rate. The Move Near column is also a useful reminder that the default recommendation does not win every individual metric. e10e20
What it supports. Visual Matching reports average MMRV 0.056 and r 0.924, compared with 0.143/0.778 for Variant Aggregation and 0.375/0.308 for MSE. Variant Aggregation nevertheless has higher Move Near r, 0.887 versus 0.855, with tied MMRV of 0.111. The support is strongest for the printed averages.
Where the evidence stops. The label “Validation MSE” needs qualification: Appendix F.1 uses RT-1 training demonstrations because its validation split is unavailable. Table 10 also gives different drawer metrics. Preserve this table’s values and protocol instead of merging the two summaries.
Table 6. A simulated appearance sensitivity motivates a concrete physical test. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across each policy row. The simulation column is a range across tuned and untuned arm colors, not an uncertainty interval. The two real columns compare the original arm with a changed appearance created using gift wrapping paper. Octo-Base spans zero to 0.293 in simulation and falls from 0.293 to zero in the real test. RT-1-X varies over a narrower simulated interval and retains substantial success under the real appearance change. Section 5.3 describes the simulated observation as the motivation for this new real-world intervention. The correspondence is about relative sensitivity of policies to appearance, not exact matching of the colors or success probabilities across domains. e15
What it supports. RT-1-X real success declines from 0.760 to 0.520, while Octo-Base declines from 0.293 to 0.000. The printed simulated ranges are [0.507, 0.653] and [0.000, 0.293], respectively. This provides a concrete example of simulation suggesting a vulnerability later observed on physical hardware.
Where the evidence stops. This is one task and two policies. The simulated and real perturbations are not pixel-identical treatments, and RT-1-X’s original real success lies outside its simulated range. The table does not provide confidence intervals or establish a universal robustness predictor.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Google Robot policy ranking: three-task protocol comparison Table 1: six checkpoints across Pick Can, Move Near and Drawer; matched real/sim evaluation. | Visual Matching: 0.056 / 0.924. Average MMRV ↓ / Pearson r ↑ | Variant Aggregation: 0.143 / 0.778; action MSE: 0.375 / 0.308. Best printed averages favor Visual Matching. These are not success rates; RT-1 MSE samples training data, and Table 10 changes drawer values. e07e10e20 |
| Cross-embodiment performance correlation Figure 4, Visual Matching; Google tasks and Bridge grasp/completion outcomes. | Google: 0.049 / 0.929; Bridge: 0.014 / 0.890. MMRV ↓ / Pearson r ↑ | Real success is the reference, with policy shapes and task colors distinguished. Association does not establish equal success probabilities. These summaries differ from Table 1 and the expanded Table 2 roster. e11e12e13 |
| Pick Coke Can under novel arm textures RT-1-X and Octo-Base; simulated texture variants and real gift-wrapped arms. | RT-1-X: 0.760 → 0.520, sim [0.507, 0.653]; Octo-Base: 0.293 → 0.000, sim [0.000, 0.293]. Real original → OOD success; simulated success range | Real drops are 24.0 and 29.3 percentage points, respectively. Simulation anticipates policy-specific sensitivity; its range is not a confidence interval or calibrated real-success prediction. e15 |
| Controller-calibration ablation Google Robot Visual Matching with perturbed stiffness/damping. | SysID: 0.131 / 0.031. Control loss ↓ / MMRV ↓ | Perturbed settings: 0.267 / 0.070 and 0.432 / 0.100. Worse replay tracking accompanies worse ranking fidelity; this does not validate all contact dynamics. e16e23 |
| Drawer visual-matching ablation Section 5.4 component combinations; checkpoint naming conflicts with Table 7. | All components: 0.050 / 0.136. MMRV ↓ / real-sim success gap ↓ | No components: 0.087 / 0.272; green screen plus drawer-only matching: MMRV 0.142. Combined curation helps; adding components is not uniformly beneficial. Text names RT-1 Begin, but Table 7 names RT-1 15%. e17 |
| Sensitivity to can mass and gripper friction Table 8(a): six Google policies; masses 10–80 g and friction coefficients 0.25–2.0. | MMRV 0.031 throughout; r 0.957–0.990. MMRV ↓ / Pearson r ↑ | Across the tested physical-parameter grid. Ranking tolerates these approximate rigid-body parameters; dexterous and soft-object transfer remains untested. e18e23 |
| Isaac Sim portability Table 9: five policies, Variant Aggregation, Pick Coke Can and Move Near. | Can average: 0.064 / 0.973; Move Near: 0.053 / 0.865. MMRV ↓ / Pearson r ↑ | Against paired real success; the paper also reports SAPIEN evaluations. This supports portability on a restricted subset, not interchangeable absolute success across simulators. e19 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Section 5.4, unnumbered visual-matching ablation table. The combination helps, but additional matching does not improve every configuration. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the three checkmark columns before comparing either metric. The first row disables all three components; the last enables all three. The rightmost columns measure different outcomes: relative ranking fidelity and the gap in absolute success. Green screening alone reduces the latter while leaving MMRV unchanged in this summary. Adding only drawer matching to the green-screened setup worsens MMRV, whereas matching both foreground components produces the lowest reported values. Section 5.4 hypothesizes that inconsistent appearance between scene parts can hurt transfer. That explanation is an author hypothesis; this seven-row table does not directly measure appearance consistency or provide every cell of a full factorial comparison. e05e17
What it supports. The complete combination reports MMRV 0.050 and success gap 0.136, compared with 0.087 and 0.272 without matching. Green screen plus drawer matching alone instead reaches MMRV 0.142. The results support evaluating whole observation configurations rather than assuming each isolated visual improvement necessarily helps.
Where the evidence stops. The main text names RT-1 Converged, Begin and RT-1-X, but Appendix Table 7 names Converged, 15% and RT-1-X. This unresolved checkpoint mismatch limits exact reproduction; the original table is retained without silently reconciling the labels.
7. Analysis & limitations
7.1 What the evidence leaves open
Rigid-object tasks, RGB/proprioception, fixed-camera green screening, imperfect shadows and manual scene curation limit scope. Preliminary Gaussian-splatting scenes were inadequate; contact-free calibration may fail for fast collisions or dexterity. e23
Inconsistencies remain: Table 2 adds OpenVLA, Table 10 changes drawer metrics, Table 7 changes an ablation checkpoint, and the augmented robustness plot gives MMRV 0.016 versus Table 4’s 0.041. RT-1 Begin drawer testing stopped after two real trials. e12e14e17e20
Tables provide point estimates without correlation confidence intervals. Non-significant Kruskal-Wallis tests do not establish equivalence, and favorable pooled correlations can conceal weaker individual tasks. e10e11e13e22
7.2 Questions for discussion
- Would ranking fidelity survive a new policy family without retuning textures?
- How much paired real evaluation distinguishes simulator bias from uncertain rankings?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires exact checkpoints, paired real rollouts, matched cameras/assets and aggregation rules. Preserve Temporal Gauss-Seidel, controller frequencies, Octo seeds and arm-color averaging. The PDF leaves software pins, compute budget, annealing bounds/schedule and absolute episode time limits unspecified. e07e08e09e24
Proposed checks should separately vary calibration and appearance matching with fixed policies and initial states, measuring ranking and absolute gaps with repeated-trial uncertainty. Resolve the drawer checkpoint mismatch first. e04e16e17
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does held-out control fidelity predict ranking fidelity?
Reader-proposed experiment; not performed here. Use the Google Pick Coke Can pose/orientation grid and a fixed policy roster, camera, meshes and arm-color aggregation. Fit stiffness/damping with the source’s 20-trajectory procedure, reserving additional real trajectories only for evaluation. Compare calibrated parameters with explicitly recorded perturbations, without retuning visual assets. Measure held-out replay loss, paired real/sim task-success gaps, MMRV and Pearson r, with trial-resampling uncertainty reported separately from arm-color variation. The proposed mechanism predicts that improved held-out replay fidelity accompanies improved ranking fidelity. Equal or worse rankings despite reliably better replay would challenge the sufficiency of contact-free calibration, especially if errors concentrate at grasp contact. Record annealing settings because the PDF does not fully specify them. e04e07e16e23e24
Check 2: Complete the drawer appearance comparison with a fixed roster
Reader-proposed experiment; not performed here. Declare the Table 7 roster—RT-1 Converged, RT-1 15% and RT-1-X—and explicitly note its disagreement with the main text. Evaluate all eight combinations of green screening, drawer-texture matching and robot-texture matching, including the green-screen/robot-only combination absent from the seven-row summary. Hold controller gains, initial poses, camera and evaluation budget fixed; keep open and close outcomes separate before aggregation. Compare MMRV and absolute real-sim success gaps with paired-trial uncertainty. The appearance-consistency hypothesis predicts that matching both foreground components benefits the green-screened setup more reliably than matching one. If the complete combination loses its advantage under this controlled roster, the original interpretation weakens. This check reproduces a declared appendix protocol, without claiming to resolve the source’s historical checkpoint identity. e05e07e17
8.3 Reading coverage
Visual audit: Visually inspected the title/author page, all method and evaluation pages, Figures 1–5, unnumbered method/robustness/ablation graphics, Tables 1–12, Algorithms 1–2 and Appendix G. Reference-only pages 9–14 were read in the complete text but not rendered. All six final PNG crops were opened and checked for readable labels, complete legends/headers and exclusion of surrounding prose. Figure 3’s horizontal violation arrows were checked against Eq. (1), and the visual-matching column order against Section 4.2’s compositing equation. Main/appendix differences in policy roster, drawer metrics, ablation checkpoint and augmented-robustness MMRV remain explicit. No external supplement, linked code or experimental execution was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 3 Using Physics Simulators for Evaluation of Robot Manipulation Policies
- 4 SIMPLER: Simulated Manipulation Policy Evaluation for Real Robots
- 4.1 Mitigating the Real-to-Sim Control Gap via Offline System Identification
- 4.2 Mitigating the Real-to-Sim Visual Gap via Visual Matching and Variant Aggregation
- 5 Experimental Results
- 5.1 Experimental Setup
- 5.2 SIMPLER Environments Show Strong Performance Correlations with Real Evaluations
- 5.3 SIMPLER Evaluations Effectively Model Policy Robustness to Distribution Shifts
- 5.4 Ablation Studies
- 6 Conclusion
- Acknowledgments
- References
- A Full Environment and Evaluation Protocol Details
- B More Implementation Details of Our Real-to-Sim Evaluation System
- B.1 Robot Controllers
- B.2 Robot and Object Assets
- B.3 SIMPLER-Variant Aggregation
- C Full Results for Real-and-Sim Relative Policy Performance Correlation Experiments
- D Full Results for Real-and-Sim Policy Behavior Correlation Experiments under Environment Distribution Shifts
- E Full Results for Main Paper Ablation Experiments
- F More Experiment Results
- F.1 More Ablations
- F.2 Other Metrics: Kruskal Wallis
- G Limitations of Our Current Work
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All eight supplied text chunks were read individually, covering all 24 PDF pages including references and Appendices A–G. All substantive evidence pages and final crops were visually inspected; reference-only pages 9–14 were read as text.
- Text extraction does not reconstruct figure images; the retained PDF was inspected to resolve figure and equation/table layout.
- Separate supplemental material availability has not been fully verified. No separate supplement was supplied. Code, linked resources and external revision histories were not inspected; experiments were not reproduced.
- Identity/edition note: the title and all 16 author identities match the catalog, but the PDF orders Karl Pertsch before Oier Mees. Its footer says CoRL 2024; the supplied catalog identifies proceedings publication in 2025. No numbered revision or dated revision chain is established; this report identifies the supplied proceedings artifact by its raw PDF hash.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and conference footer
The title matches the catalog. The 16 authors appear with Karl Pertsch before Oier Mees. Affiliations are UC San Diego, Stanford University, UC Berkeley and Google Deepmind. The footer identifies CoRL 2024, Munich, Germany; no numbered revision is printed.
Go to primary source ↓e02PDF pp. 2–3, Figure 2 and Sections 1 and 3, problem formulation
SIMPLER evaluates policies trained on real data in simulated Google Robot and WidowX/BridgeData V2 environments through a Gym interface. Its target is relative-performance correlation, complementing physical evaluation.
Go to primary source ↓e03PDF p. 4, Figure 3, Section 3 and Eq. (1)
MMRV averages each policy’s maximum misranking penalty, weighted by the real-performance margin. Pearson r measures linear association. The figure depicts small/large horizontal real-performance gaps, nonlinear rankings and noise sensitivity.
Go to primary source ↓e04PDF pp. 4–5, Section 4.1, system-identification loss and optimization paragraphs
Open-loop replay compares real/simulated end-effector positions and rotation matrices under identical actions. Twenty offline trajectories with different instructions and three rounds of simulated annealing tune stiffness and damping without collecting new calibration data.
Go to primary source ↓e05PDF p. 5, Section 4.2, unnumbered visual-matching figure and compositing equation
Real backgrounds with foreground removed are combined with simulated foreground masks. Object textures are projected from aligned real images; tuned arm colors address appearance changes during rollouts.
Go to primary source ↓e06PDF p. 6, Section 4.2 continuation; PDF p. 18, Figure 5 and Appendix B.3
SIMPLER uses SAPIEN and rigid-object task setups. Variant Aggregation averages over visual variants, with two variations along each of background, lighting, distractors and table texture; it need not visually match reality.
Go to primary source ↓e07PDF p. 6, Section 5.1; PDF pp. 15–16, Appendix A, task lists and averaging protocol
Paired tests match instructions, poses and success criteria. Planned Google trial counts are 75, 60, 54 and 27 for can, move-near, drawer and drawer-apple tasks; Appendix A lists 24 per WidowX task. Google simulations average four arm colors; Octo simulations average three seeds. Longer tasks switch instructions at termination or half the time limit.
Go to primary source ↓e08PDF pp. 16–17, Algorithms 1–2 and Appendix B.1
RT-1 uses 501 Hz simulation and 3 Hz policy control with inverse kinematics and constrained joint trajectory planning; Ruckig is named. WidowX uses 500 Hz simulation and 5 Hz control and retains the previous arm goal. Algorithm 1 filters gripper actions with absolute value below 0.01.
Go to primary source ↓e09PDF pp. 17–18, Appendix B.2, Robot and Object Assets
The pipeline uses robot URDFs, camera intrinsics, curated visual/collision meshes and Temporal Gauss-Seidel for Google Robot grasping. Texture alignment uses segmentation, pose adjustment and differentiable rendering; collision meshes use CoACD. Construction remains partly manual.
Go to primary source ↓e10PDF p. 7, Table 1, six-checkpoint caption and MMRV/Pearson r rows
Across Pick Can, Move Near and Drawer, average MMRV/r are 0.056/0.924 for Visual Matching, 0.143/0.778 for Variant Aggregation and 0.375/0.308 for action MSE. Move Near r favors Variant Aggregation, 0.887 versus 0.855.
Go to primary source ↓e11PDF p. 7, Figure 4, both panels and legends
Figure 4 prints MMRV/r of 0.049/0.929 for Google Robot and 0.014/0.890 for BridgeData V2. Shapes identify policies; colors identify tasks. Bridge includes both grasping and full-task success.
Go to primary source ↓e12PDF p. 19, Table 2, policy/metric rows; PDF p. 20, footnotes 2–3
Table 2 adds OpenVLA-7B to the six policies named in Table 1 and prints different aggregates. RT-1 Begin real drawer testing stopped after two trials due to potential damage. A footnote says continued evaluation could lower the reported MMRV.
Go to primary source ↓e13PDF p. 19, Table 3, grasp and Success columns
WidowX results distinguish partial grasping from completion. Put Carrot on Plate full success has MMRV 0.111 and r 0.575, despite stronger agreement on some other tasks; rankings are not uniformly perfect.
Go to primary source ↓e14PDF p. 7, Section 5.3 and distribution-shift plot; PDF p. 20, Tables 4–5
Robustness tests vary background, lighting, distractors, table texture and camera pose for RT-1 with/without augmentation. The plot prints augmented MMRV 0.016, but Table 4 prints 0.041 for the corresponding average. Table 4 uses absolute changes; Table 5 includes variants with improved success.
Go to primary source ↓e15PDF pp. 7–8, Section 5.3, novel distribution shifts; PDF p. 20, Table 6
For Pick Coke Can, real original/OOD arm-texture success is 0.760/0.520 for RT-1-X and 0.293/0.000 for Octo-Base. Simulated texture ranges are [0.507,0.653] and [0.000,0.293]. Physical arms were changed using gift wrapping paper.
Go to primary source ↓e16PDF p. 8, Section 5.4, Effect of system identification and adjacent table
SIMPLER SysID has control loss 0.131 and MMRV 0.031. Perturbed settings give 0.267/0.070 and 0.432/0.100, respectively.
Go to primary source ↓e17PDF p. 8, Section 5.4, visual-matching ablation table/text; PDF p. 21, Table 7, checkpoint headers and component rows
The main ablation gives MMRV/success gap 0.087/0.272 with no matching and 0.050/0.136 with all components. Green screen plus drawer matching alone gives MMRV 0.142. Main prose lists RT-1 Begin, while Table 7 labels RT-1 15%; this discrepancy is unresolved.
Go to primary source ↓e18PDF p. 21, Table 8(a–b), physical-property ablations
Can masses 10–80 g and gripper friction coefficients 0.25–2.0 retain MMRV 0.031, with r spanning 0.957–0.990. Drawer joint-friction tests also retain relatively low ranking violations. The experiments concern six Google policies and specified rigid-object tasks.
Go to primary source ↓e19PDF p. 8, simulator comparison; PDF p. 21, Table 9, five-policy Isaac Sim Variant Aggregation results
Isaac Sim tests cover Pick Coke Can and Move Near with five policies. Table 9 gives MMRV/r 0.064/0.973 and 0.053/0.865 for average success, respectively; absolute success differs from reality.
Go to primary source ↓e20PDF p. 22, Appendix F.1, simulation-free comparison and Table 10
MSE uses 25 Bridge validation trajectories but 25 RT-1 training trajectories because no public RT-1 validation split exists. Correlation uses negative MSE. Table 10 changes drawer values relative to Table 1: MSE 0.346/0.264 and Visual Matching 0.055/0.915 for MMRV/r.
Go to primary source ↓e21PDF p. 22, Table 11 and Appendix F.1; PDF p. 23, F.1 continuation
An added RT-1 policy is trained solely on can-picking demonstrations. Its real/sim success is 0.680/0.403, and the expanded comparison reports MMRV 0.027 and r 0.959.
Go to primary source ↓e22PDF p. 23, Table 12 and Appendix F.2
Kruskal-Wallis tests compare per-policy real and simulated trial-success distributions. Table 12 counts policies with p below 0.05; some tasks have detected shifts despite favorable ranking results.
Go to primary source ↓e23PDF pp. 23–24, Appendix G, limitations (1)–(5)
Limitations include rigid-object tasks, fixed-camera green screening, imperfect shadows, manual scene curation, RGB/proprioceptive inputs and simple contact-free identification. Preliminary Gaussian-splatting scenes had geometry/rendering problems. High-speed collisions and dexterous manipulation require stronger calibration.
Go to primary source ↓e24PDF p. 5, Section 4.1; PDF pp. 15–18, Appendices A–B
The PDF gives trajectory counts, controller frequencies, solver and asset tools, but no numeric annealing search bounds, complete annealing schedule, absolute episode time limits, pinned software environment or hardware/compute budget for reproducing the pipeline.
Go to primary source ↓8.5 Primary sources
Evaluating Real-World Robot Manipulation Policies in Simulation ↗
PDF · 14,013 extracted words
Source fingerprint
1cb4de1a58da4eeaf4f1761a488bb452c042ea3bec1e9840cff5acc51a7c7167