RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies
1. Paper overview
In one sentence: RoboDojo makes manipulation failures easier to measure through complementary simulation and physical tests, while standardizing interfaces and reducing evaluation cost. problemsuiteinterfacesimulation-resultsphysical-resultsrandomizationefficiency
| At a glance | What to know |
|---|---|
| Research problem | Source description A high aggregate manipulation score can hide failures in memory, precision, instruction grounding or long sequences. Simulation supplies scalable diagnosis but omits physical disturbances; real testing adds deployment evidence yet varies with hardware, lighting and resets. RoboDojo tries to control those measurement conditions. problem |
| Core mechanism | Source description A five-dimension simulation suite and three-embodiment physical suite evaluate distinct manipulation demands. Open tasks withhold task-specific demonstrations; Generalization separates standard scenes from randomized ones. suitesim-datasim-protocol |
| A key reported result | Five-dimension simulation manipulation: Hy-Embodied-0.5-VLA: 13.07 / 8.80%. Average partial-progress score / task success rate. Frozen 3 July 2026 leaderboard: 42 tasks; normally three training seeds and 50 episodes per task per seed; dimensions weighted equally. Spatial Forcing: 12.38 / 8.04%; π0.5: 11.41 / 6.91%; expert teleoperation: 80.42 / 76.03%. The leading policy remains far below expert completion. The printed table supplies no standard deviations, despite the protocol description. simulation-resultssim-protocol |
| Reading caution |
Core contributions
- Source description
A five-dimension simulation suite and three-embodiment physical suite evaluate distinct manipulation demands. Open tasks withhold task-specific demonstrations; Generalization separates standard scenes from randomized ones. suitesim-datasim-protocol
- Source description
XPolicyLab standardizes policy adapters and remote evaluation; heterogeneous simulation and RealEval provide the corresponding execution environments. The paper compares existing policies rather than introducing a shared learned controller. interfacesimulation-stackphysical-platform
Table 12. One external policy contract serves both physical trials and parallel simulation. Original paper, p. 35 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward through the Stage column, then compare the two deployment columns. In the single-environment path, policy.reset clears episode state, update_obs receives the latest observation, and get_action returns a chunk of actions. The execution row belongs to RoboDojo: env.step applies each action until termination or chunk exhaustion. On the right, the active index set selects the environments participating in batched observation updates, action prediction and execution. Finished environments leave that set. Appendix H describes the same sequence, so the table should be read as a runtime interface rather than a neural-network architecture. interfacesimulation-stack
What it supports. The shared contract makes heterogeneous policies evaluable through a common client. The action-chunk prediction row and environment-execution row are distinct: returning an action is not evidence of successful manipulation. Batched queries support the independently configured simulation environments described in Appendix F.
Where the evidence stops. The table does not specify each policy’s internal state, world-model objective or numerical chunk length. A common interface therefore cannot establish One Model architecture or an inference-time imagination mechanism.
2. Motivation
2.1 The problem and the proposed response
A high aggregate manipulation score can hide failures in memory, precision, instruction grounding or long sequences. Simulation supplies scalable diagnosis but omits physical disturbances; real testing adds deployment evidence yet varies with hardware, lighting and resets. RoboDojo tries to control those measurement conditions. problem
2.2 What this reading follows
RoboDojo asks what a generalist robot policy can actually complete when evaluation spans more than familiar pick-and-place scenes. Its simulation suite separates five capability dimensions, while a physical suite tests three bimanual embodiments under controlled hardware and reset conditions. XPolicyLab connects both through the same external policy interface. This unification concerns evaluation infrastructure: the tasks are not paired sim-to-real transfer tests, and the evaluated policies keep their own architectures. Read the leaderboards alongside the randomization diagnostic and throughput comparison. Together they show a large gap to expert teleoperation, sensitivity to scene changes, and the practical value of making repeated evaluation faster. problemsuiteinterfacesimulation-resultsphysical-resultsrandomizationefficiency
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator classification is supported by task suites, measurement protocols and deployment infrastructure. The sim-to-real tag should mean complementary sim-and-real assessment here, not matched transfer. One Model/Two Models and joint-prediction/inverse-dynamics quadrants are not applicable to the benchmark itself; evaluating WAM baselines supplies no architecture classification for RoboDojo. suiteinterfacesimulation-stack
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Start with what is held out
The five simulation dimensions define different tests rather than five interchangeable groups of scenes. Generalization changes backgrounds, clutter, lighting and objects; Memory requires using earlier observations; Precision stresses accurate control; Long-Horizon requires completing dependent steps; Open withholds task-specific demonstrations. The training set therefore has 34 task datasets plus auxiliary DLC, whereas evaluation covers 42 tasks. For Generalization, each seed contributes 25 standard and 25 random episodes per task. The aggregate first summarizes each dimension and then gives the five dimensions equal weight. This prevents the larger Generalization group from dominating, but it also means the headline score is not simply the fraction of all episodes completed. The physical suite follows a different task distribution and should be read as complementary deployment evidence. suitesim-datasim-protocol
Figure 7. Physical repeatability begins with fixed geometry and controlled reset conditions. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the enclosure in the center, then follow the dashed callouts to the supporting components. The left panels identify continuous LED lighting, the workstation and replaceable ARX X5, Piper X and Piper embodiments. The right panel labels one Gemini 335L head camera and two Gemini 305 wrist cameras, together with the structural fixtures and touchscreen. The assembly imagery concerns mechanical placement, not policy information flow. Section 4.2 and Appendix G explain the accompanying reset procedure: operators compare a transparent reference-image overlay with the live camera view and move objects back into alignment before execution. physical-platformstabilityscoring-gap
What it supports. The platform standardizes several nuisance variables that otherwise change between physical evaluation sessions: robot placement, camera pose, illumination and initial object layout. Its 1.2-by-1.2-meter table fits within the labeled 1.5-by-1.5-meter footprint. These controls make repeated physical measurements more interpretable.
Where the evidence stops. The hardware graphic does not itself quantify reproducibility. Tables 6 and 9 show that low aggregate variability can coexist with substantial task-level variation; scoring-disagreement filtering also remains underspecified.
5.2 Follow an observation through the execution boundary
XPolicyLab standardizes the outside of a policy without imposing a common neural architecture. Dataset adapters prepare each policy’s training format and normalization; at evaluation time the environment client communicates with a policy server through WebSocket and MessagePack. After reset, the client sends the latest observation, the server predicts an action chunk, and the environment executes its actions until the chunk ends or the episode terminates. Batched simulation carries active environment indices through observation updates, action queries and stepping, removing completed environments. RealEval adds fixed hardware geometry and visual layout replay, and Pink converts end-effector targets when needed. Consequently, an interface-compatible action prediction is only one stage: actual task success still depends on execution, contact and feedback through later observations. interfacephysical-platformreal-protocol
5.3 Read a diagnosis without turning it into a causal result
The standard/random comparison reveals a concrete weakness: Hy-Embodied-0.5-VLA’s score falls from 21.98 to 1.57, while Spatial Forcing retains 6.98 from 21.25. The latter also exceeds π0.5’s randomized score of 5.82, but attributing the difference solely to spatial grounding would require stronger controls. The paper’s training appendix helps assess comparability, yet cross-policy changes can still combine representation, optimization and implementation differences. My reading is that these results identify useful interventions to test, rather than prove why one policy fails. Likewise, the physical stability study reports low overall variation while store_in_safe remains much noisier. A credible improvement should therefore survive repeated trials on the relevant capability or task, instead of relying only on an aggregate ranking. randomizationtrainingstabilitydiagnosis
5.4 Training and inference
During training
Simulation provides 3,500 demonstrations: 34 training tasks plus auxiliary DLC, each with 100 trajectories, totaling 20.66 hours at 25 Hz. Open has no task-specific training data. Simulation demonstrations use automated annotation-grounded skills with cuRobo v2 or VR teleoperation. Physical training supplies 1,800 demonstrations, 17.91 hours, collected by four operators using matching leader/follower embodiments. sim-datareal-datasimulation-stack
Policy objectives and internals remain model-specific. Hy-Embodied-0.5-VLA uses batch 128 for 200K simulation steps and six historical images sampled every 20 steps. Spatial Forcing and π0.5 share pi05_base initialization, batch 256 and 60K simulation steps; π0.5 uses 30K physical-training steps. These are differing fine-tuning recipes, not a compute-matched architectural ablation. training
During inference
The environment resets policy state, sends an observation to the policy server and requests an action chunk. It executes returned actions until termination or chunk exhaustion, then repeats. Batched simulation queries only active environment indices. WebSocket and MessagePack carry observations/actions; they do not determine the policy’s prediction mechanism. interface
For physical end-effector commands, Pink converts targets into robot motions. Simulation horizons derive from the 90th-percentile demonstration length with a 1.2 multiplier, or 1.5 for short autogenerated tasks; physical horizons use 1.5. Unsafe physical trials may terminate early. real-protocolsim-protocol
5.5 Implementation flow
- Define the evaluation target
Simulation separates Generalization, Memory, Precision, Long-Horizon and Open. For example, match_and_pick_from_conveyor requires retaining an object identity after it disappears. Physical tasks instead stress deployment across ARX X5, Piper and Piper X; there are no matched sim/real task pairs. suite
- Instantiate reproducible environments
MagicSim, Isaac Sim and Isaac Lab instantiate YAML-defined assets, layouts and success conditions. Disabling strict physics replication lets each parallel environment retain independent scenes and state while sharing batched stepping; multiple GPU processes partition episode seeds. simulation-stack
- Control physical nuisance variables
RealEval fixes camera and robot mounts, lighting and workspace geometry. Operators overlay a reference layout on live images before each trial; the touchscreen manages execution, recording and emergency stops. physical-platform
- Measure progress and completion separately
Simulation uses 50 episodes per task per seed, with 25 standard and 25 random Generalization episodes, then weights the five dimensions equally. Real testing uses ten trials per task and averages across 18 tasks. Three blinded raters score physical videos; scoring-disagreement handling remains ambiguous. sim-protocolreal-protocolscoring-gap
6. Experiments & results
RoboDojo couples 42 simulation tasks with 18 physical tasks through XPolicyLab and standardized RealEval hardware. Its contribution is an evaluation system: capability-specific scores expose brittle robot execution, while parallel simulation reduces measurement cost. The suites are complementary, not paired transfer tests.
6.1 Read the original evidence
Table 1. Low average completion coexists with different leaders across capability dimensions. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read every cell as partial-progress score followed by success rate, not two success measurements. Move across a policy row to see its capability profile, then compare the Average column. That average weights the five dimensions equally despite their different task counts. Blue highlighting and underlining identify the best and second-best policy values; the human row is a separate reference excluded from ranking. The printed snapshot is frozen on 3 July 2026. Section 5.1 describes three training seeds for most policies and 50 episodes per task per seed, but this table prints point values without the promised standard deviations. simulation-resultssim-protocoltrainingdocumentation-gaps
What it supports. Hy-Embodied-0.5-VLA leads the average at 13.07 score and 8.80% success, versus the expert reference’s 80.42 and 76.03%. Leadership varies: X-VLA reaches 12.00% Precision success, while π0.5’s leading Open success is only 1.67%. The average alone hides these different weaknesses.
Where the evidence stops. Training budgets and initial checkpoints vary, and the expert reference helped construct the dataset. Without the missing uncertainty values and clear seed exceptions, small leaderboard gaps should not be treated as statistically established differences.
Table 2. The physical suite supplies direct execution evidence under a distinct task distribution. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First choose a policy block, then read its three embodiment rows. Task 1 through Task 6 are not identical tasks across robots: use the retained task-order footnote to identify each column. As in the simulation table, cells list score before success rate; the real-world table omits percent signs in its numerical entries. The embodiment average summarizes six tasks, while the rightmost cell summarizes all 18. Section 5.2 specifies one training seed per embodiment and ten trials per task. These results come from the standardized physical reset and video-scoring protocol, not simulated trajectories. physical-resultsreal-protocolsuitetraining
What it supports. π0.5 leads the reported physical average with score 22.9 and success 12.8%, followed by InternVLA-A1 at 12.0 and 7.2%. π0.5’s embodiment success rates range from 3.3% on Piper X to 21.7% on Piper. The human reference completes every reported task.
Where the evidence stops. Embodiments have different tasks, so their gap cannot isolate robot kinematics. The real and simulation suites are also unpaired and use different fine-tuning settings; their aggregate difference is not a measured sim-to-real transfer loss.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Five-dimension simulation manipulation Frozen 3 July 2026 leaderboard: 42 tasks; normally three training seeds and 50 episodes per task per seed; dimensions weighted equally. | Hy-Embodied-0.5-VLA: 13.07 / 8.80%. Average partial-progress score / task success rate | Spatial Forcing: 12.38 / 8.04%; π0.5: 11.41 / 6.91%; expert teleoperation: 80.42 / 76.03%. The leading policy remains far below expert completion. The printed table supplies no standard deviations, despite the protocol description. simulation-resultssim-protocol |
| Physical manipulation across three embodiments 18 tasks; one training seed per embodiment; ten trials per task, 180 trials per policy; replayed layouts and three-rater scoring. | π0.5: 22.9 / 12.8%. Overall score / success rate | InternVLA-A1: 12.0 / 7.2%; GalaxeaVLA: 9.0 / 4.4%; expert reference: 100.0 / 100.0%. Physical execution remains unreliable under this protocol. Different tasks, fine-tuning and participating policies prevent interpreting the simulation/real difference as a transfer rate. physical-resultsreal-protocolsuite |
| Generalization under scene randomization The 12 Generalization tasks, standard versus random episodes under the simulation protocol. | Hy-Embodied-0.5-VLA: 21.98 / 1.57; 92.9% drop. Standard score / random score; relative score drop | Spatial Forcing: 21.25 / 6.98; 67.2%. π0.5: 20.92 / 5.82; 72.2%. High standard-scene performance does not ensure robustness. Several scene factors change together, so this diagnostic does not isolate their individual effects. randomizationsim-protocol |
| Simulation evaluation throughput Eight RTX 4090 simulation/rendering GPUs; one process and ten environments per GPU; 1,640,000 frames; π0.5 served by LAN A800. | Heterogeneous: 77.4 with zero actions; 64.0 with π0.5. Interactions per second | Non-heterogeneous: 40.0 and 39.2, respectively; reported speedups 1.94× and 1.63×. This infrastructure comparison supports faster evaluation, not stronger policies. RoboTwin comparisons use different rendering resolutions, process counts and workloads. efficiency |
| Physical evaluation time and repeatability Timing: π0.5 on LAN RTX 4090, all 180 full trials. Separate stability study: three repeated rounds for three policies. | 202.0 minutes including reset, inference and execution; maximum overall success SD 1.3 percentage points and score SD 1.2. Wall-clock time; across-round standard deviation | π0.5 on store_in_safe has task-level success SD 23.1 percentage points and score SD 12.9. Stable aggregates can conceal noisy individual tasks. Timing and stability are separate experiments, and repeated evaluation does not measure uncertainty across independently trained physical policies. physical-timestabilityreal-protocol |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. A strong standard-scene score can collapse under combined scene variation. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each named policy has a standard score, a randomized score and a parenthesized relative drop. The table header supplies that order; its caption defines the drop as standard minus random, divided by standard. Compare absolute randomized scores before comparing percentages: a large relative improvement from an almost-zero standard score need not mean useful performance. Under the benchmark protocol, each Generalization task contributes 25 episodes in each setting per seed. Figure 8 and the adjacent text show that randomization combines background, lighting, clutter, object appearance and scene changes, so the table is a multi-factor robustness diagnostic. randomizationsim-protocoltraining
What it supports. Hy-Embodied-0.5-VLA falls from 21.98 to 1.57, a reported 92.9% relative loss. Spatial Forcing drops from 21.25 to 6.98, versus π0.5’s 20.92 to 5.82. Spatial Forcing retains more performance in these comparisons, but even its absolute randomized score remains low.
Where the evidence stops. This is not an ablation that changes only spatial grounding or one nuisance factor. Relative drops concern scores, not percentage-point changes in success. The table does not report uncertainty for these comparisons.
Table 4. Heterogeneous parallel evaluation improves throughput with and without policy inference. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the first two rows to examine raw simulation under zero actions, then compare the next two rows to include π0.5 inference. Those four RoboDojo rows each process 1,640,000 frames. The last two columns separate elapsed wall-clock time from normalized interactions per second. The surrounding experimental description fixes simulation and rendering to eight RTX 4090 GPUs, with one RoboDojo process and ten environments per GPU; π0.5 is served on a LAN A800 machine. Appendix F explains the implementation: environments keep distinct scenes and task state while sharing batched stepping, and episode seeds are distributed across GPU processes. efficiencysimulation-stack
What it supports. Heterogeneous throughput is 77.4 rather than 40.0 interactions per second under zero actions and 64.0 rather than 39.2 with π0.5. The reported gains are 1.94× and 1.63×. Faster repeated evaluation is the supported outcome; the table measures no policy-learning improvement.
Where the evidence stops. RoboTwin uses a different workload, two processes per GPU and lower image resolution, so its row is not a matched backend ablation. The π0.5 timing also includes a particular LAN inference configuration rather than arbitrary remote-network latency.
7. Analysis & limitations
7.1 What the evidence leaves open
Reported failure modes include jitter, uncorrected alignment errors and weak memory-conditioned execution. These observations motivate mechanisms to investigate, but cross-policy rankings do not causally isolate memory, world modeling or spatial grounding. diagnosistraining
Expert teleoperators helped collect or construct the benchmark data. Open is held out from benchmark fine-tuning, but the PDF does not audit foundation-model pretraining overlap. Dexterous, tactile, mobile and whole-body manipulation remain future extensions. sim-protocolreal-protocolsim-datafuture
Source ambiguities remain: Table 1 lacks promised deviations; seed exceptions are not identified; Appendix K omits five listed policies and full optimizer/software/frozen-module settings. Several Open instructions contain blank slots. DLC’s generic pick-and-place description differs from its letter-arrangement specification, and Table 8’s 35+ policy count differs from the 30-policy frozen table. documentation-gapstrainingsource-variation
7.2 Questions for discussion
- Which randomization factor drives the largest absolute score loss when factors are varied separately?
- How would policy rankings change with uncertainty estimated across training seeds and physical reset sessions?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the exact evaluated checkpoints, policy adapters, normalization, task assets/configurations and seed definitions, plus the documented RealEval geometry, camera mounts and layout replay. The PDF names software components but does not pin their complete environment or establish an inspected artifact license. interfacephysical-platformsimulation-stacktraining
Resolve the hidden-layout significance threshold and the discrepancy between averaging all three rater scores and filtering disagreements first. Also distinguish three training checkpoints from one checkpoint evaluated with three seeds; Appendix A.2 permits both, which measure different uncertainty sources. integrityscoring-gap
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate visual-shift effects from spatial-grounding gains
Reader-proposed check, not performed: fine-tune π0.5 and Spatial Forcing from the stated pi05_base initialization with the same demonstrations, batch 256, 60K steps and three recorded training seeds. Evaluate the 12 Generalization tasks on fixed standard layouts, then vary background, lighting, clutter and target objects separately before applying combined randomization. Keep horizons and execution settings fixed. Report paired per-task score changes, success and seed uncertainty. A consistently smaller loss for Spatial Forcing across isolated factors would support a robust grounding advantage; a benefit confined to one factor or seed would narrow that interpretation. Resolve the DLC specification before claiming exact reproduction. trainingsim-protocolrandomizationsource-variation
Check 2: Test whether layout replay stabilizes the noisy safe task
Reader-proposed check, not performed: hold an ARX X5 π0.5 checkpoint and deployment settings fixed and repeat store_in_safe across operators and days. Compare randomized trial blocks with reference-image overlay enabled against an ordinary reset without overlay, while keeping fixtures, lighting, task horizon and safety rules unchanged. Use at least three rounds of ten trials per condition; retain raw videos and three independent rater scores under a predeclared disagreement rule. Measure reset time, image-alignment error, task success and score variance. Lower placement error together with reduced between-session variance would support the reset mechanism; unchanged outcome variance would point toward contact or policy execution as the remaining bottleneck. safe-taskphysical-platformreal-protocolstabilityscoring-gap
8.3 Reading coverage
Visual audit: The title/author page, all numbered figures and tables, and every page supporting retained method, training, numerical, protocol and reproduction claims were visually inspected. Table 12 was checked against Appendix H; Figure 7 labels and assembly cues were checked against Section 4.2 and Appendix G. All six final crops were extracted from the supplied PDF and individually viewed, retaining the original table headers and necessary task-order footnote. The complete text, including all task specifications and references, was read, but uncited task-card images on other pages and external videos were outside this visual pass. The LDA-1B training entry on p. 45 visibly specifies batch 128 for 300K simulation steps, agreeing with the supplied extracted text.
PDF pages inspected for this edition: 1, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 28, 29, 30, 31, 32, 33, 34, 35, 37, 38, 41, 42, 44, 45, 46. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and contents, pp. 1–2
- Abstract and Sections 1–8, pp. 3–21
- References, pp. 22–27
- Appendices A–H: governance, comparisons, stability, datasets, platforms and interface, pp. 28–35
- Appendix I: every simulation task and DLC specification, pp. 35–41
- Appendix J: every physical task specification, pp. 41–44
- Appendix K: all supplied policy-training entries, pp. 44–46
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity/version: title and all 44 authors match. This review covers the supplied arXiv:2607.04434v3 dated 8 July 2026, whose title-page header says 5 July 2026. The catalog submission date is 5 July; earlier revisions were not supplied or compared. The printed leaderboards are frozen on 3 July 2026, not current results.
- Acquisition omission: text extraction does not reconstruct figure images. Original PDF pages and six final crops were inspected separately; visualAudit lists the exact visual coverage.
- Separate supplemental material availability has not been fully verified.
- External code, datasets, documentation, checkpoints, licenses and evaluation videos were not inspected; no experiments were reproduced.
- All 16 supplied text chunks were read individually. Unfilled instruction slots and other source ambiguities are preserved in limitations; missing values are not inferred.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author block, header date and arXiv margin
Exact title and all 44 named authors match the supplied catalog author string. The artifact is arXiv:2607.04434v3, 8 July 2026; the page header is dated 2026-7-5.
Go to primary source ↓problemPDF p. 3, Abstract and Section 1
The benchmark combines scalable simulation diagnosis with standardized physical testing because either setting alone gives incomplete evidence about manipulation.
Go to primary source ↓suitePDF pp. 5–8, Sections 3.1–3.2.1 and Figure 2
There are 42 ARX X5 simulation tasks: 12 Generalization, 6 Memory, 8 Precision, 8 Long-Horizon and 8 Open. Physical testing has six tasks per embodiment on ARX X5, Piper and Piper X. These are complementary suites without one-to-one sim/real task alignment.
Go to primary source ↓sim-dataPDF p. 7, Section 3.1.2; pp. 29–30, Appendix D.1 and Table 10
Simulation training comprises 34 task datasets plus auxiliary DLC, 100 trajectories each, totaling 3,500 trajectories, 1,859,602 frames and 20.66 hours at 25 Hz. Three head/wrist RGB-D views are 640×480; robot states and next-frame actions are recorded. Open has no task-specific demonstrations.
Go to primary source ↓real-dataPDF pp. 8–9, Section 3.2.2; pp. 30–31, Appendix E.1 and Table 11
Physical demonstrations use homogeneous leader-follower teleoperation, four operators and 100 demonstrations per task. Total: 1,800 trajectories, 1,611,841 frames and 17.91 hours at 25 Hz, with three 640×480 RGB views, language and robot states. Demonstrations are manually filtered.
Go to primary source ↓sim-protocolPDF p. 7, Section 3.1.3; p. 12, Section 5.1; p. 29, Appendix D.2
Each simulation task uses 50 evaluation episodes per seed; Generalization splits them into 25 standard and 25 random episodes. Overall metrics average five dimension metrics equally. Most policies use three training seeds. Human experts share success criteria and horizons and helped construct the dataset. The leaderboard is frozen on 3 July 2026. Simulation horizons use 1.2 times the 90th-percentile demonstration length, or 1.5 for short automatically generated tasks.
Go to primary source ↓real-protocolPDF p. 9, Section 3.2.3; p. 13, Section 5.2; p. 31, Appendix E.2
Ten policies are evaluated using one training seed per embodiment and ten trials per task, totaling 180 trials per policy. Reference layouts are replayed. Three evaluators independently score videos under the stated double-blind protocol. Physical horizons are 1.5 times the 90th-percentile demonstration length; unsafe trials may be stopped. Pink converts end-effector targets into executable motions. Expert teleoperators contributed to data collection.
Go to primary source ↓simulation-stackPDF pp. 10–11, Sections 4.1.1–4.1.4 and Figure 6; pp. 31–33, Appendices F.1–F.5
The simulation stack combines MagicSim infrastructure, Isaac Sim and Isaac Lab. YAML specifications define scenes and success checks. Heterogeneous parallelism disables strict physics replication, keeps independent task states/seeds and shares batched stepping; seeds are sharded across GPU processes. Annotation-grounded skills and cuRobo v2 generate demonstrations; VR teleoperation handles other tasks.
Go to primary source ↓physical-platformPDF pp. 11–12, Figure 7 and Section 4.2; pp. 33–34, Appendix G and Figure 10
RealEval fixes robot/camera geometry, uses controlled LED lighting and curtains, and replays layouts by overlaying reference images on live observations. The table is 1.2×1.2 m within a 1.5×1.5×2.1 m frame. Figure 7 labels one Gemini 335L head camera and two Gemini 305 wrist cameras. A touchscreen supports execution, reset, emergency stop and video collection.
Go to primary source ↓interfacePDF pp. 34–35, Appendix H and Table 12
XPolicyLab adapts datasets and observations while preserving policy internals. A WebSocket/MessagePack client-server protocol supports reset, observation update and action-chunk prediction. The environment executes actions until termination or chunk exhaustion. Batched queries use active environment indices and remove finished environments. Pose order is position followed by scalar-first quaternion.
Go to primary source ↓trainingPDF pp. 44–46, Appendix K
Hy-Embodied-0.5-VLA is fine-tuned with batch 128 for 200K simulation steps and six historical images sampled every 20 steps. Spatial Forcing and π0.5 start from pi05_base and use batch 256 for 60K simulation steps; π0.5 uses 30K physical-training steps. The LDA-1B entry on p. 45 specifies LDA-pretrain initialization, batch 128 and 300K simulation steps. Training budgets and foundations vary across policies. Appendix K does not specify a common loss, frozen-module policy, optimizer or full software environment.
Go to primary source ↓simulation-resultsPDF p. 13, Table 1, Average and dimension columns
Hy-Embodied-0.5-VLA: average score 13.07 / success 8.80%; Spatial Forcing: 12.38 / 8.04%; π0.5: 11.41 / 6.91%; human reference: 80.42 / 76.03%. X-VLA leads Precision at 18.32 / 12.00%. π0.5 leads Open at 1.98 / 1.67%. Values are score followed by success rate; the printed table contains no standard-deviation entries.
Go to primary source ↓physical-resultsPDF p. 14, Table 2, Overall Avg., π0.5 embodiment rows and task-order footnote
π0.5 has score 22.9 / success 12.8%; InternVLA-A1 12.0 / 7.2%; GalaxeaVLA 9.0 / 4.4%; human reference 100.0 / 100.00%. π0.5 embodiment scores/success rates are ARX X5 27.7 / 13.3, Piper 32.0 / 21.7 and Piper X 9.1 / 3.3. Task numbers refer to different task sets by embodiment.
Go to primary source ↓randomizationPDF p. 15, Figure 8, Table 3 and Section 6.1 Finding 2
Table 3 reports Standard Score / Random Score and relative drop (Standard−Random)/Standard. Hy-Embodied-0.5-VLA: 21.98 / 1.57, 92.9%; Spatial Forcing: 21.25 / 6.98, 67.2%; π0.5: 20.92 / 5.82, 72.2%. Randomization changes several visual and scene factors together.
Go to primary source ↓efficiencyPDF p. 18, Section 6.3.1 and Table 4
On eight RTX 4090 GPUs, RoboDojo runs one simulation process and ten parallel environments per GPU. Heterogeneous/non-heterogeneous throughput is 77.4/40.0 interactions per second for zero actions and 64.0/39.2 with π0.5 inference on a LAN A800 server; all four use 1,640,000 frames. RoboTwin uses different process counts, resolution and total frames.
Go to primary source ↓physical-timePDF p. 19, Section 6.3.2 and Table 5
π0.5 on a LAN RTX 4090 server completes 180 physical trials in a reported 202.0 minutes, including reset, inference and execution. The efficiency experiment runs full trials without manual early stopping.
Go to primary source ↓stabilityPDF pp. 19–20, Section 6.4 and Tables 6–7; p. 28, Appendix C; p. 30, Table 9
Cross-GPU simulation stability uses layout 0, three RTX 4090 GPUs and three seeds per GPU for π0.5, Xiaomi-Robotics-0 and GalaxeaVLA; overall success SD is at most 0.5 percentage points and score SD 0.49. Physical stability uses three repeated rounds for π0.5, GalaxeaVLA and InternVLA-A1; overall success SD is at most 1.3 percentage points and score SD 1.2. π0.5 on store_in_safe has success SD 23.1 percentage points and score SD 12.9.
Go to primary source ↓diagnosisPDF pp. 16–18, Section 6.1 Findings 3–6 and Section 6.2
Authors interpret failures as poor skill composition, correction after misalignment, memory-conditioned execution and semantic-to-action grounding. Physical rollouts reportedly show jitter and unsafe behavior. These are rollout-based diagnoses and cross-policy comparisons, not controlled interventions isolating each cause.
Go to primary source ↓integrityPDF p. 9, Section 3.3; p. 28, Appendix A.2
Public layouts supply primary leaderboard scores; hidden randomized layouts provide verification. A significant public/hidden success discrepancy invalidates a submission, but the threshold/test is not specified. Verified publication requires evaluated artifacts and videos. Appendix A.2 permits either three training checkpoints or one checkpoint evaluated with three evaluation seeds.
Go to primary source ↓scoring-gapPDF p. 31, Appendix E.2; p. 33, Appendix G, Safety control and scoring
Appendix E.2 averages three evaluator scores, whereas Appendix G says samples with large scoring discrepancies are filtered before averaging. The filtering threshold and reconciliation are not provided.
Go to primary source ↓documentation-gapsPDF p. 12, Section 5.1; p. 13, Table 1; pp. 37–38, Appendix I.2; pp. 44–46, Appendix K
Section 5.1 promises standard deviations and seed-exception markings, but Table 1 lacks deviations and Appendix K does not identify such exceptions. Appendix K omits policy-specific entries for ABot-M0, Dexora-1B, A1, TinyVLA and OpenVLA-OFT. Several Open task instruction strings visibly leave target/color/category slots blank.
Go to primary source ↓source-variationPDF p. 5, Section 3.1.1; p. 7, Section 3.1.2; p. 29, Table 8; p. 41, Appendix I.6
The main text describes DLC as auxiliary randomized pick-and-place data, while Appendix I.6 specifies arranging letters to spell RoboDojo and labels Data Source null (train-only). Table 8 lists 35+ integrated/evaluated policies, whereas the frozen simulation leaderboard lists 30. These differences are not reconciled in the supplied PDF.
Go to primary source ↓futurePDF p. 20, Section 7 and Figure 9
Dexterous, humanoid whole-body, tactile and mobile manipulation are proposed future extensions rather than completed benchmark evaluations.
Go to primary source ↓safe-taskPDF p. 42, Appendix J.1, store_in_safe
The ARX X5 task requires placing all objects in the safe and closing it; demonstrations are teleoperated and the task is used for training and evaluation.
Go to primary source ↓8.5 Primary sources
RoboDojo: A Unified Sim-and-Real Benchmark for Comprehensive Evaluation of Generalist Robot Manipulation Policies ↗
PDF · 26,420 extracted words
Source fingerprint
e07c215f30169dde6a792c34b046b1f54818d09331763988732dcf9ead89a9d4