RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins
1. Paper overview
In one sentence: RoboTwin uses annotated generative digital twins and language-generated robot programs to expand demonstration data, improving physical transfer while leaving coordinated dual-arm manipulation difficult. e-probleme-assetse-annotationse-codee-sim-protocole-real-protocole-supplement-gaps
| At a glance | What to know |
|---|---|
| Research problem | Source description Dual-arm learning needs varied demonstrations that capture handovers, shared workspace constraints and interference. Teleoperation is costly, while fixed simulation assets and task-specific generators restrict variation. RoboTwin seeks matched physical and simulated scenarios with scalable expert-data generation. e-problem |
| Core mechanism | Source description Image-to-3D generation supplies varied object instances; reusable spatial annotations connect their geometry to task-specific programs. e-assetse-annotationse-code |
| A key reported result | Real-world dual-arm transfer: 62% with 300 simulation + 20 real demonstrations. Average task success (%). Five Table 3 tasks; 50 randomized physical trials per task. 20% with 20 real demonstrations only. Calculated gain: 42 percentage points. Hard bottle picking remains 11/50; Sweep Ball rises from 25/50 to 43/50. No confidence intervals are supplied. e-real-protocole-dual |
| Reading caution | Reader analysis Component causality remains untested: no controlled removal of generated asset diversity, transferred annotations or iterative code repair appears in the supplied paper. Generated-code success in Figure 5 is separate from learned-policy or physical success. e-boundariese-benchmark |
Core contributions
- Source description
Image-to-3D generation supplies varied object instances; reusable spatial annotations connect their geometry to task-specific programs. e-assetse-annotationse-code
- Source description
The benchmark combines recorded multimodal demonstrations, generated-code evaluation, imitation-policy comparisons and physical transfer tests. These are distinct evaluation layers. e-benchmarke-sim-protocole-real-protocol
Figure 2. Object generation and task geometry meet at executable robot behavior. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three framed regions from left to right. A photograph supplies both a segmented object and a description; the generated geometry is textured into a simulation asset. The middle region attaches contact points, a function point and named axes. Those annotations and the task description feed the right region, where a hammering task is decomposed across the two arms before constraints and key poses are calculated. Section 3.3 completes the diagram with API-based code, motion planning and execution feedback. The arrows describe a data-generation pipeline, not a single neural network that jointly predicts future observations and actions. e-assetse-codee-annotationse-optimizatione-axis-discrepancy
What it supports. The mechanism makes object geometry available to program generation. A hammer is represented by more than its appearance: the tool has places to grasp and act, and directions that constrain its pose. The planner must turn those constraints into feasible trajectories before the resulting behavior can become a demonstration.
Where the evidence stops. Figure 2 names Axis 2 as lateral, yet asks Task 1 to align with it; Section 3.3 uses the approach axis for grasp alignment. The end-effector-vector mapping is unspecified. Its blue/green axis labels also differ from Figure 3. The original graphic is retained without resolving these ambiguities.
2. Motivation
2.1 The problem and the proposed response
Dual-arm learning needs varied demonstrations that capture handovers, shared workspace constraints and interference. Teleoperation is costly, while fixed simulation assets and task-specific generators restrict variation. RoboTwin seeks matched physical and simulated scenarios with scalable expert-data generation. e-problem
2.2 What this reading follows
RoboTwin asks how a robot benchmark can gain diverse expert demonstrations without scaling teleoperation alone. Its answer starts with object photographs, turns them into simulation assets, and annotates the places and directions that matter for grasping and tool use. Language-generated programs then call geometric and motion-planning machinery to produce demonstrations. Separate imitation policies learn from those demonstrations and are evaluated in simulation and on physical robots. Read the figures as a chain from asset construction to executable behavior, then use the tables to distinguish representation effects from transfer gains. The supplied CVPR paper supports that chain, while leaving implementation details in absent appendices. e-probleme-assetse-annotationse-codee-sim-protocole-real-protocole-supplement-gaps
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Benchmarks & simulators |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded benchmark/simulator and synthetic-data categories fit the contribution. RoboTwin combines asset generation, language-generated programs, a simulator and separate imitation baselines. It does not specify a unified future/action model or inverse-dynamics action extraction. Architecture, prediction paradigm and quadrant are therefore not applicable to this resource. e-assetse-codee-benchmarke-sim-protocol
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Give the task a geometric vocabulary
A photograph can indicate what a hammer looks like, but grasping and striking require different locations and orientations. RoboTwin first creates a textured 3D asset and then adds a contact point, a function point and directional axes. The contact annotation supports grasp-pose construction; the function annotation supports alignment of the operating part with a target. Section 3.2 transfers points between instances using top-view diffusion features, reducing the need to annotate every generated object again. This transfer is a correspondence procedure, not a learned dynamics model. Read the axis definitions from the text and local labels because Figures 2 and 3 use inconsistent colors. Their additional Part Axis and Figure 2’s Axis 2 grasp-direction binding remain incompletely specified. e-assetse-annotationse-axis-discrepancye-code
Figure 3. A reusable spatial vocabulary connects different shapes to manipulation constraints. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the labeled hammer. Its function point identifies the operating part, while the contact point marks an interaction or grasp location. Section 3.2 defines the function axis by the tool operation, the approach axis by its approach to a target, and the lateral axis as perpendicular to both. Compare the mug and shoe to see the same annotation vocabulary placed on different geometries. The text describes transferring points between same-class models by matching Stable Diffusion features in top-view images. This figure illustrates annotation examples; it does not visualize that correspondence algorithm or report its accuracy. e-annotationse-codee-axis-discrepancy
What it supports. The useful distinction is between an object feature used to hold the object and one used to perform the task. That separation allows grasp constraints and manipulation constraints to reference different geometry. Reusing annotations across instances is intended to reduce repeated labeling while retaining the functional meaning needed by generated programs.
Where the evidence stops. Read the printed axis names locally: green is approach and blue is lateral here, the reverse of Figure 2. The extra gray Part Axis is drawn but not separately defined among the text’s three principal axes. Transfer reliability remains unquantified.
5.2 Separate program generation from policy learning
The LLM turns a task into subtasks and spatial constraints, then writes code that calls predefined robot APIs. Pose calculation and MPlib planning connect those constraints to joint trajectories. The displayed optimization requires forward-kinematic consistency, position and orientation alignment, and collision-free configurations over time. Its cost function is described only generically, so it does not specify a complete numerical optimizer configuration. After execution, errors and final-state deviations can trigger another code-generation attempt. That loop belongs to expert-data production. The benchmark subsequently trains DP or DP3 imitation policies to predict actions from their respective observations. The paper does not specify an inference-time future-state rollout or a shared neural model containing both the digital twin and the action policy. e-codee-optimizatione-sim-protocol
5.3 Ask which comparison supports which conclusion
Reader analysis: the tables answer three different questions. Table 1 compares imitation methods and observation representations under demonstration budgets, revealing task-dependent effects of color and severe coordination failures. Tables 2 and 3 ask whether simulation pretraining helps when real demonstrations are scarce; they support large gains over the twenty-real-only baseline. Figure 7 adds a stronger real-data reference, but its two endpoints do not establish identical performance or a universal replacement ratio between simulation and teleoperation. None of these comparisons separately removes generated object diversity, spatial annotations or iterative code repair. A positive transfer result therefore supports the combined training pipeline, while attribution to an individual mechanism remains open. Physical-policy architecture and training schedules would also need clarification before a faithful reproduction. e-sim-protocole-sim-resultse-singlee-duale-real-protocole-scalinge-boundaries
5.4 Training and inference
During training
The benchmark description specifies 100 simulation and 20 real data sets per task. Policy experiments use 20, 50 or 100 expert demonstrations; physical transfer separately uses 300 simulation demonstrations for pretraining and 20 real demonstrations for fine-tuning. These budgets should not be conflated. e-benchmarke-sim-protocole-real-protocol
DP predicts actions from images; DP3 uses point clouds with or without color. The paper gives no new policy-loss formula, frozen-module specification, optimizer schedule or training compute budget in the supplied body. Detailed physical-policy identification is also missing. e-sim-protocole-real-protocole-supplement-gaps
During inference
Expert generation executes generated programs through geometric pose calculations and a planner. Evaluated imitation policies instead predict actions from observations and are scored on executed trajectories. Success requires final target poses and collision-free execution; no learned future-state rollout is specified for policy-time planning. e-codee-sim-protocol
The simulator varies object pose, shape, appearance and size while screening initial feasibility. Data include RGB/depth views, point clouds, and joint/end-effector poses. The body does not establish a held-out object-identity split or unseen-task evaluation. e-benchmarke-sim-protocol
5.5 Implementation flow
- Generate and screen assets
GPT-4V describes photographed objects; a language model varies descriptions, SDXL-Turbo generates image variants, and Rodin produces geometry and textures. UCLIP-I and GPT-4V checks trigger regeneration below unspecified thresholds. Material-based physics parameters receive ±5% variation. e-assets
- Attach functional geometry
Semi-automatic annotation distinguishes where a tool acts from where it is grasped. Function, approach and lateral axes specify orientation. Top-view Stable Diffusion feature matching transfers keypoints to same-class instances using highest feature similarity. Transfer accuracy is not quantified. e-annotations
- Convert tasks into constrained behavior
The LLM decomposes a task, relates contact/function points and axes, and writes API calls. Computed world-coordinate poses feed MPlib and screw-motion planning. Coordinated gripper control, independent arm motion and safe intermediate positions handle different collaboration requirements. e-code
- Validate and refine demonstrations
Execution checks task completion. Runtime exceptions, planning failures and final-state deviations join the original task, annotations and previous code as regeneration inputs. This feedback improves demonstration programs; it is not a specified online update rule for the learned policy. e-code
6. Experiments & results
RoboTwin turns object photographs into annotated simulation assets and uses language-generated programs plus motion planning to collect expert demonstrations. It benchmarks imitation policies and tests simulation pretraining on physical robots. The strongest evidence is improved performance with limited real demonstrations; difficult dual-arm coordination remains unresolved.
The supplied paper has no controlled component-removal ablation of generated assets, spatial-annotation transfer or code refinement. Figure 7 therefore serves as a data-scaling diagnostic, alongside the representation comparisons in Table 1. Appendix Table 4, Figure 9, task descriptions and prompt examples are referenced but absent from this PDF; this edition cannot illustrate or verify those additional materials. e-boundariese-scalinge-supplement-gaps
6.1 Read the original evidence
Table 1. Observation choice and demonstration count interact strongly with the task. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each side of the table contains task blocks with three method rows. Read across a row to compare 20, 50 and 100 demonstrations; compare rows within the same task and column to contrast DP, DP3 with XYZ, and DP3 with XYZ+RGB. The caption specifies the D435 setting and three seeds. Gray shading occurs within method rows and should not be interpreted as one winner across all methods. Focus on Pick Apple Messy and Container Place for opposite effects of color, then compare Dual Bottles Pick (Easy) with Dual Shoes Place for the difference between easier parallel manipulation and difficult coordination. e-sim-protocole-sim-resultse-boundariese-benchmark
What it supports. At 100 demonstrations, XYZ+RGB reaches 54.0 ± 12.8% on Pick Apple Messy versus 9.7 ± 2.1% for XYZ, but trails XYZ on Container Place, 62.3 ± 6.8% versus 85.3 ± 3.2%. Dual Shoes Place remains at 12.0 ± 1.7% or lower across methods. These results identify uneven policy capabilities rather than a uniformly solved benchmark.
Where the evidence stops. The supplied caption does not define the ± statistic or simulation test-trial count. These are simulated imitation-policy results, distinct from generated-code success and real-robot transfer. The comparisons do not isolate the asset-generation pipeline’s causal contribution.
Table 2. Simulation pretraining substantially improves the five tested single-arm tasks. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read every body entry as a count of successful physical trials out of 50. The left column trains only on 20 real demonstrations; the right column first uses 300 simulation demonstrations and then fine-tunes on 20 real demonstrations. Compare the final Average row with the individual counts because they are not internally consistent. Hammer Beat gives a concrete example: 2 successes become 37. Keep easy and hard bottle picking separate because their outcomes differ substantially. Section 5.3 says initial object configurations and arm placements were randomized within predetermined boundaries; their numeric bounds are not supplied. e-singlee-real-protocole-boundaries
What it supports. The printed average rises from 1.2% to 72%, a 70.8-percentage-point difference. However, the five combined-data counts sum to 183/250, or 73.2%, implying a 72-point gain. The improvement is large under either reading, but its exact aggregate is inconsistent. Container Place reaches 49/50 successes while Bottle Pick (Hard) reaches 16/50.
Where the evidence stops. The source does not reconcile the printed average with the counts; both are preserved here. Its prose also calls the gain 72%. The main paper leaves the policy architecture unidentified and supplies no confidence intervals for these physical tests.
Table 3. Dual-arm transfer improves, while difficult bottle manipulation remains weak. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the same column interpretation as Table 2, but read this as a separate group of dual-arm tasks. Dual bottle Pick, Container Place, Cup Place and Sweep Ball cover different coordination demands. The container and sweep baselines already succeed in 25 of 50 trials, whereas both bottle baselines and Cup Place are zero. Compare the absolute counts row by row before looking at the average. The hard bottle task reaches 11 successes after pretraining; the easy variant reaches 31. Those two rows prevent the 62% average from obscuring a persistent failure regime. e-duale-real-protocole-boundaries
What it supports. The reported average rises from 20% to 62%, a calculated gain of 42 percentage points. Sweep Ball improves from 25/50 to 43/50, and Cup Place from 0/50 to 26/50. Synthetic demonstrations help the tested dual-arm policies, but success varies widely across the task set.
Where the evidence stops. The five tasks are not identical to the single-arm group, so comparing their averages does not isolate the cost of adding a second arm. No confidence intervals, training-seed variation or per-failure breakdown accompany the physical counts.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Real-world dual-arm transfer Five Table 3 tasks; 50 randomized physical trials per task. | 62% with 300 simulation + 20 real demonstrations. Average task success (%) | 20% with 20 real demonstrations only. Calculated gain: 42 percentage points. Hard bottle picking remains 11/50; Sweep Ball rises from 25/50 to 43/50. No confidence intervals are supplied. e-real-protocole-dual |
| Real-world single-arm transfer Five Table 2 tasks; 50 randomized physical trials per task. | 72% printed average with 300 simulation + 20 real demonstrations; the five task counts imply 73.2%. Average task success (%) | 1.2% with 20 real demonstrations only. Unresolved source inconsistency: printed averages imply a 70.8-percentage-point gain; summed counts imply 72 points. Hammer Beat rises from 2/50 to 37/50. The prose also says 72% improvement, without reconciling the table. e-real-protocole-single |
| Dual Bottles Pick (Easy) Simulation, D435 setting, three seeds; 100 demonstrations unless indicated. | DP: 85.7 ± 6.7. Success rate (%) with reported ± quantity | DP3 XYZ: 55.3 ± 11.5; XYZ+RGB: 75.7 ± 17. DP at 20 demonstrations: 1.7 ± 0.6. DP benefits strongly from more demonstrations on this task; this does not establish a universal scaling law. The ± statistic is unspecified. e-sim-protocole-sim-results |
| Pick Apple Messy / Container Place representation comparison Simulation, D435 setting, 100 demonstrations, three seeds. | XYZ+RGB: 54.0 ± 12.8 / 62.3 ± 6.8. Success rate (%) with reported ± quantity | XYZ: 9.7 ± 2.1 / 85.3 ± 3.2, respectively. Color helps the cluttered apple task but hurts container placement in these comparisons. This diagnoses representation sensitivity, not the benefit of RoboTwin asset generation. e-sim-protocole-sim-results |
| Dual Shoes Place Simulation, D435 setting, 100 demonstrations, three seeds. | DP3 XYZ: 12.0 ± 1.7. Success rate (%) with reported ± quantity | XYZ+RGB: 6.0 ± 1.0; DP: 3.0 ± 1.0. All compared policies struggle with coordinated placement in a shared workspace. e-sim-protocole-sim-resultse-boundaries |
| Real-data scaling diagnostic Figure 7: single-arm bottle pick and dual-arm cup place; 300 real-only versus 300 simulation + 20 real demonstrations. | Combined: approximately 84% bottle / 52% cup. Success rate (%), approximate visual readings | 300 real-only: approximately 80% bottle / 64% cup. The authors describe comparable performance. The cup reference is below its real-only endpoint; without uncertainty or an equivalence test this supports a data-efficiency trend, not equal performance. e-scaling |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 7. A data-budget diagnostic compares more teleoperation with simulation pretraining. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the solid curves along the five labeled real-data budgets. Blue denotes Bottle Pick and red denotes Cup Place; the dashed line of each color is a fixed 300Sim+20Real reference, not a curve that acquires additional real data at each tick. Section 5.3 identifies these as single-arm bottle picking and dual-arm cup placement. The x-axis budgets are categorical, unevenly spaced quantities, so visual line slope is not a per-demonstration rate. Compare endpoints and reference levels within each task. The experiment motivates the selected simulation budget; it is not a sweep over the number of simulation demonstrations. e-scalinge-boundaries
What it supports. Approximate readings at 300 real demonstrations are 80% for bottle picking and 64% for cup placement, against fixed combined-data references near 84% and 52%. The plot supports substantial real-data savings on these examples, while showing that the mixed-data policy does not match both real-only endpoints exactly.
Where the evidence stops. The paper calls the results comparable, but supplies no uncertainty or equivalence test. These percentages are visual estimates from unlabeled points. This is a data-scaling diagnostic, not an ablation that independently removes spatial annotations, generated diversity or code refinement.
7. Analysis & limitations
7.1 What the evidence leaves open
Component causality remains untested: no controlled removal of generated asset diversity, transferred annotations or iterative code repair appears in the supplied paper. Generated-code success in Figure 5 is separate from learned-policy or physical success. e-boundariese-benchmark
Figure 2 reverses the approach/lateral colors used in Figure 3. Its Task 1 alignment references Axis 2 (lateral), while the grasping formulation uses the approach axis. The end-effector-vector mapping remains unresolved; color alone cannot define an implementation. e-axis-discrepancy
The benchmark declares 15 tasks, but the main policy table evaluates 14. Simulation test counts, the ± definition, task-specific success tolerances and exact randomization bounds are absent. Physical gains compare different five-task groups and do not isolate arm count alone. e-benchmarke-sim-protocole-real-protocole-boundaries
Table 2’s combined-data counts total 183 successes in 250 trials (73.2%), while its Average row prints 72%. Both are retained; neither is silently substituted for the other. Table 3’s counts agree with its 62% average. e-singlee-dual
7.2 Questions for discussion
- Would axis-aware constraints still improve program success after equalizing regeneration attempts and human intervention?
- How much physical-transfer benefit remains when comparing against equally diverse simulation assets without image generation?
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction requires the Cobot Magic master/puppet setup, D435 calibration, ManiSkill3/SAPIEN assets, MPlib APIs and the generation/annotation services. Three active camera streams run at 30 Hz; a fourth lower camera is optional. Hardware matching is part of the transfer protocol. e-benchmarke-codee-assets
Obtain the missing task definitions and prompt/API examples before attempting faithful replication. Record model versions, asset-quality thresholds, physics parameters, trajectory costs, training schedules and dataset splits; the supplied paper leaves these partly or wholly unspecified. Proposed checks should separately measure program validity and learned-policy transfer. e-assetse-optimizatione-supplement-gapse-real-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does axis information improve generated behavior?
Reader-proposed check, not performed: use the same hammer assets, initial scenes, contact/function points, LLM configuration and MPlib planner in three conditions: correctly named axes, points only, and deliberately shuffled axis directions. Define and document the end-effector-vector mapping explicitly because Figure 2 leaves it ambiguous. Give each condition the same code-regeneration budget and human-intervention allowance. Compare successful collision-free programs, orientation-related failures and repair attempts on held-out initial poses. If correct axes do not reduce orientation failures or improve success over both controls, the claimed practical value of axis-aware constraints is weakened. This would test the annotation mechanism separately from imitation-policy learning. e-annotationse-axis-discrepancye-codee-optimizatione-benchmark
Check 2: Separate asset diversity from the benefit of extra simulation data
Reader-proposed check, not performed: for the bottle-pick and cup-place tasks, compare 300 simulation demonstrations using repeated canonical assets with 300 using generated variants, then fine-tune both on the identical 20 real demonstrations. Hold the policy architecture, optimization budget between simulation conditions, scene randomization and evaluation configurations fixed. Include 20-real-only and 300-real-only reference policies, following the paper’s data-efficiency comparison. Use 50 matched physical trials per task, report binomial uncertainty and retain the individual task counts. If both simulation conditions perform similarly, additional practice rather than generated asset diversity may explain the transfer gain; a reproducible advantage for the diverse condition would support that component. e-assetse-benchmarke-real-protocole-scalinge-boundaries
8.3 Reading coverage
Visual audit: Visually inspected every page of the supplied PDF: title/author block and version watermark; Figures 1–8; Tables 1–3; method definitions, planning equation, platform and evaluation text; acknowledgements and references. All six final original crops were separately viewed and checked for complete labels, axes, legends and table rows. Figures 2 and 3 were cross-checked against Sections 3.2–3.3; their axis-label and grasp-direction ambiguities are disclosed. All pages supporting retained quantitative, methodological and reproduction claims are included. Referenced supplementary appendices and code remain outside the supplied material.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work (2.1–2.4)
- 3. Bridging Physical and Digital Worlds for Diverse Robot Behavior Generation (3.1–3.3, all six generation steps and optimization)
- 4. Benchmark
- 5. Experiment on RoboTwin Benchmark (5.1–5.3)
- 6. Conclusion
- Acknowledgements
- References [1]–[76]
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The complete supplied 12-page CVPR 2025 open-access paper was read. Its title and all 14 authors match the catalog. The watermark states equivalence to the accepted version except for the watermark; the IEEE proceedings and the separately cited 2024 early version were not supplied or compared.
- Text extraction does not reconstruct figure images; this gap was addressed by visually inspecting all 12 PDF pages, Figures 1–8, Tables 1–3 and every final crop.
- Separate supplemental material availability has not been fully verified. Referenced Appendices A, B and D, Appendix Table 4 and Appendix Figure 9 are absent from the supplied artifact and were not read.
- Code, external model services, datasets and licenses were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1 (printed p. 27649), title, author block and CVF watermark; p. 11, reference [57]
The title and all 14 authors match the catalog. The CVF watermark identifies the open-access version as identical to the accepted version except for the watermark, with final proceedings on IEEE Xplore. Reference [57] separately labels the 2024 arXiv work an early version; it is not the artifact reviewed here.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract and Section 1
RoboTwin addresses scarce dual-arm demonstrations and limited real-world alignment through image-derived assets, spatially aware code generation and matched simulation/physical evaluation.
Go to primary source ↓e-assetsPDF p. 3, Figure 2 and Section 3.1; p. 4, Section 3.1 continuation
GPT-4V descriptions are varied by a language model; SDXL-Turbo creates image variants and Rodin generates 3D assets. UCLIP-I and GPT-4V validation trigger regeneration below unspecified thresholds. GPT-4V assigns material-dependent physics parameters with ±5% random variation.
Go to primary source ↓e-annotationsPDF p. 4, Figure 3 and Section 3.2
Function/contact points and function/approach/lateral axes support manipulation. Lateral is perpendicular to function and approach. Top-view Stable Diffusion feature correspondence transfers annotated points between same-class instances by highest similarity; initial annotation is semi-automatic.
Go to primary source ↓e-axis-discrepancyPDF p. 3, Figure 2, axis labels and Task 1 constraint; p. 4, Figure 3 hammer labels and Section 3.2; p. 5, Section 3.3 constraint inference and displayed optimization
Figure 2 labels blue as approach (Axis 1) and green as lateral (Axis 2), whereas Figure 3 labels green as approach and blue as lateral. Figure 2 requests alignment with Axis 2; the grasping text and equation instead reference the approach axis. The figure does not define the end-effector direction well enough to resolve that mapping. Both figures show a Part Axis beyond the three principal axes defined in the text.
Go to primary source ↓e-codePDF pp. 4–5, Section 3.3, capabilities and steps 1–6; p. 3, Figure 2
The LLM decomposes tasks, infers spatial constraints and generates API-based code. World-coordinate contact/function points determine poses; MPlib and screw-motion planning produce trajectories with arm coordination and collision avoidance. Runtime errors, planning failures and final-state errors feed code regeneration, with limited human oversight for complex cases.
Go to primary source ↓e-optimizationPDF p. 5, Section 3.3, step 4, unnumbered displayed optimization and following definitions
Joint trajectory θ(t) minimizes J subject to forward kinematics, position/orientation alignment with object contact point and approach axis, and membership in collision-free configuration space over the trajectory interval. J has possible cost factors but no fixed weights. The scalar d has no explicit definition or value; separate definitions of P_ee and n_ee are also absent.
Go to primary source ↓e-benchmarkPDF pp. 5–6, Section 4; p. 6, Figures 4–5
The benchmark declares 15 tasks and uses ManiSkill3/SAPIEN with a Cobot Magic platform. Four arms comprise master/puppet pairs; four D435 cameras include an optional lower camera. Front and wrist streams run at 30 Hz. Each task has 100 pre-collected simulation and 20 real data sets; RGB, depth, point clouds and arm poses are recorded. Initial poses and object sizes vary, with feasibility screening. Figure 5 measures generated-code success.
Go to primary source ↓e-sim-protocolPDF p. 6, Section 5.1; p. 7, Table 1 caption
DP, DP3 (XYZ) and DP3 (XYZ+RGB) are evaluated on 14 tasks under the D435 setting with 20, 50 and 100 demonstrations and three seeds. Success requires final target-pose constraints and collision-free execution. The caption does not identify the ± statistic or simulation test-trial count.
Go to primary source ↓e-sim-resultsPDF p. 7, Table 1, Dual Bottles Pick (Easy), Pick Apple Messy, Container Place and Dual Shoes Place blocks
At 100 demonstrations, Dual Bottles Pick (Easy) reports DP 85.7±6.7, XYZ 55.3±11.5 and XYZ+RGB 75.7±17; DP at 20 demonstrations is 1.7±0.6. Pick Apple Messy reports XYZ+RGB 54.0±12.8 versus XYZ 9.7±2.1; Container Place reports XYZ+RGB 62.3±6.8 versus XYZ 85.3±3.2. Dual Shoes Place reports XYZ 12.0±1.7, XYZ+RGB 6.0±1.0 and DP 3.0±1.0. Values are success rates in percent.
Go to primary source ↓e-scalingPDF p. 7, Figure 7 and caption; p. 8, Section 5.3, paragraph selecting 300 simulation data sets
The plot compares real-only data scaling with fixed 300-simulation-plus-20-real references. Approximate endpoints are bottle 80% real-only versus 84% combined, and cup 64% real-only versus 52% combined. The text identifies single-arm bottle and dual-arm cup tasks and describes performance as comparable. The plot has no uncertainty bars or explicit endpoint labels.
Go to primary source ↓e-real-protocolPDF p. 8, Section 5.3, training comparison and randomized-trial paragraphs
Policies trained on 20 real demonstrations are compared with pretraining on 300 simulation demonstrations followed by fine-tuning on 20 real demonstrations. Each task has 50 physical trials with randomized object poses and robot-arm placements within predetermined, unspecified boundaries. The main text does not name the policy architecture for Tables 2–3 or give optimizer and fine-tuning schedules.
Go to primary source ↓e-singlePDF p. 8, Table 2, both success columns and Average row
Single-arm counts, real-only versus combined: Bottle Pick (Easy) 0/50 versus 42/50; Bottle Pick (Hard) 0/50 versus 16/50; Container Place 0/50 versus 49/50; Cup Place 1/50 versus 39/50; Hammer Beat 2/50 versus 37/50. Printed average rates are 1.2% versus 72%, a calculated 70.8-percentage-point difference. However, the combined counts sum to 183/250 = 73.2%, implying a 72-point gain over 1.2%. The printed average and task counts are inconsistent; the paper does not reconcile them.
Go to primary source ↓e-dualPDF p. 8, Table 3, both success columns and Average row
Dual-arm counts, real-only versus combined: Dual bottle Pick (Easy) 0/50 versus 31/50; Dual bottle Pick (Hard) 0/50 versus 11/50; Container Place 25/50 versus 44/50; Cup Place 0/50 versus 26/50; Sweep Ball 25/50 versus 43/50. Average rates are 20% versus 62%, a calculated 42-percentage-point difference.
Go to primary source ↓e-boundariesPDF p. 7, Section 5.2 and Table 1; p. 8, Sections 5.2–6 and Tables 2–3
The authors emphasize poor complex dual-arm coordination and inconsistent RGB benefits. Main-paper experiments compare observation representations, demonstration counts and simulation pretraining; they do not isolate individual asset-generation, annotation-transfer or code-refinement components. Physical averages concern different five-task groups and provide no uncertainty intervals.
Go to primary source ↓e-supplement-gapsPDF p. 5, Section 3.3, Appendix D pointer; p. 6, Section 4, Appendix A pointer; p. 7, Section 5.2, Appendix Table 4 pointer; p. 8, Section 5.3 and Figure 8, Appendix B/Figure 9 pointers; pp. 9–12, acknowledgements and references
Prompts, task details, additional results and scene views are referred to appendices, but this 12-page artifact ends with references and contains no appendix. These pointers cannot supply the omitted implementation or evaluation details.
Go to primary source ↓8.5 Primary sources
RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins ↗
PDF · 8,425 extracted words
Source fingerprint
ed4fbd32f89e40d3610dc1e0da42d7e22222e0eedf50bb81965cd4b5c5ba80dc