ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model
1. Paper overview
In one sentence: A task-labeled Gaussian world model predicts stabilizing-then-acting effects to improve a separate bimanual policy, trading stronger reported multi-task success for calibrated multi-view supervision and unresolved implementation details. problempipelinegaussiansdynamicsobjectivessim-resultsreal-resultslimits
| At a glance | What to know |
|---|---|
| Research problem | Source description Coordinated tasks require one arm to secure an object while the other acts on it. A scene representation must capture the consequences of both movements under limited demonstrations. The paper extends single-arm ManiGaussian to learn these interactions through future-scene supervision. problempipeline |
| Core mechanism | Source description Task-oriented Gaussian primitives attach instance logits to geometry and appearance, using instruction-prompted visual labels to distinguish task-relevant bodies. gaussians |
| A key reported result | RLBench2 ten-task bimanual manipulation: 35.6% Mean task success rate (%). Multi-task training with 100 demonstrations/task, six RGB-D cameras; Table I states 25 evaluation episodes. Completion budget: 25 steps. ManiGaussian: 18.8%; PerAct2: 15.4%. Using the author-printed averages, calculated gains are 16.8 and 20.2 percentage points, respectively; relative gains are 89.36% and 131.17%. The abstract calls the improvement 20.2%; the printed averages imply a 20.2-percentage-point difference. However, the ten displayed ManiGaussian++ task values have an equal-task mean of 35.2%, rather than the printed 35.6%; the aggregation or rounding procedure needed to reconcile this difference is unspecified. The reported 35.6% is retained, and 35.2% is only a diagnostic calculation. The table does not show superiority on every task: handover-easy is 40% versus PerAct2’s 41%. sim-protocolsim-resultsproblem |
| Reading caution | Author claim Calibrated multi-view cameras for supervision increase real-robot deployment cost. limits |
Core contributions
- Source description
Task-oriented Gaussian primitives attach instance logits to geometry and appearance, using instruction-prompted visual labels to distinguish task-relevant bodies. gaussians
Figure 2. A shared visual representation supports direct bimanual action prediction and supervised Gaussian scene prediction. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the lower path: voxelized observations pass through the sparse convolutional representation network, then aggregated volumetric features feed the multimodal policy and its action outputs. The upper path explains how training enriches those features. A Gaussian regressor produces scene primitives carrying appearance and instance logits; image reconstruction and detector-derived masks supervise them. Follow the black arrow from the leader deformation field to the stabilizing-motion intermediate and then to the follower. The green feature arrows feed both deformation fields. Equation (5) clarifies that the follower receives both actions, even though only its acting-action input is explicitly labeled here. pipelinegaussiansdynamicsobjectives
What it supports. The architecture places world prediction alongside the action policy, sharing the volumetric representation. Its proposed benefit is that learning the consequences of stabilizing and acting movements improves the features used for imitation. The figure supplies no candidate-action search or imagined-trajectory selection procedure; future-scene supervision should therefore be read as the documented mechanism.
Where the evidence stops. Figure 2 labels intermediate states with s/a, while Equation (5) uses r/l and an unsubscripted renderer input. The follower’s additional stabilizing-action input is explicit in the equation. These inconsistencies leave exact tensor naming unresolved; the labels do not establish fixed right/left roles.
2. Motivation
2.1 The problem and the proposed response
Coordinated tasks require one arm to secure an object while the other acts on it. A scene representation must capture the consequences of both movements under limited demonstrations. The paper extends single-arm ManiGaussian to learn these interactions through future-scene supervision. problempipeline
2.2 What this reading follows
When one robot arm holds a target and the other manipulates it, the next scene depends on their interaction. ManiGaussian++ uses this structure to train a visual representation: reconstruct the current scene as labeled Gaussian primitives, predict the effects of the stabilizing arm, then condition the acting-arm prediction on that intermediate result. The policy learns from the same representation but still outputs actions directly. The reading below connects that architecture to the simulation table, the component ablation, selected image predictions and physical-robot results. It also preserves source inconsistencies, especially the difference between the headline 60% and detailed 62.22% real-world average. problempipelinegaussiansdynamicsobjectivessim-resultsreal-resultslimits
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system / Other mechanisms / Outside quadrants assessment fits the separate policy and action-conditioned Gaussian prediction branch. Shared features and joint losses do not establish a single joint future/action generator. The mechanism is auxiliary forward prediction, not inverse dynamics or demonstrated inference-time planning. Multi-view geometry supports the 3D label; generalization evidence is limited to the reported multi-task settings. pipelinedynamicsobjectivessim-protocol
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Give the scene task-relevant identities before predicting motion
The first representation problem is to associate visible geometry with the bodies that matter for an instruction. The source begins with calibrated RGB-D observations, voxelizes them and learns volumetric features with sparse 3D convolutions. A feed-forward regressor turns those features into Gaussian primitives. Beyond ordinary geometry and appearance parameters, each primitive carries a three-dimensional instance logit. The renderer projects those logits to image pixels, and softmax plus cross-entropy compares them with instruction-prompted GroundedSAM labels. This gives the representation an additional supervision signal about task-relevant instances. The source does not fully document how arbitrary object identities and arm roles map into those three components. That missing mapping matters for reproduction: reconstructing appearance alone would omit the task-oriented supervision that distinguishes row 3 from row 2 in the ablation. pipelinegaussiansobjectivesablation
5.2 Read the leader–follower order as a prediction dependency
The leader uses current Gaussians, the stabilizing-arm action and volumetric features to predict an intermediate deformation. Equation (5) then gives the follower that intermediate result together with both arm actions and the same features. Equation (4) represents the resulting position and orientation changes as the sum of stabilizing and acting contributions, while other Gaussian properties remain fixed. Rendering makes the predicted future comparable with observed future images, avoiding direct supervision in Gaussian parameter space. Reader interpretation: the benefit of the hierarchy is a structured conditional prediction problem. It does not establish that the physical arms must execute in that temporal order. The policy still predicts both arms’ targets from features and language. The inconsistent s/a versus r/l state labels should therefore remain an implementation question, not be turned into an asserted left-to-right execution rule. pipelinedynamicsobjectives
5.3 Connect representation learning to control without skipping the evidence gap
The overall loss links action imitation to three auxiliary objectives: current reconstruction, task masks and future prediction. The intended causal chain is that these auxiliary constraints improve the shared visual features and thereby improve the policy. The simulation table measures the end of that chain: task completion within the evaluation budget. The progressive ablation tests adding components and shows a substantial final gain, but does not compare parameter-matched alternative dependency structures. Figure 3 probes an earlier link by displaying selected current and future renderings; its disabled behavior-cloning loss prevents treating it as a direct diagnostic of the deployed policy. Reader analysis: the evidence supports the usefulness of the combined training system more strongly than it isolates the precise cause of the gain. A controlled hierarchy comparison and a task-label intervention would test that narrower claim. objectivessim-protocolsim-resultsablationdiagnostic
5.4 Training and inference
During training
Optimize behavior cloning jointly with current-image reconstruction, task-mask classification and future-image prediction. Image losses compare renderings with observed views; mask loss uses softmax-normalized instance logits. The detector supplies pretrained supervision, but its freezing/fine-tuning configuration is not specified. gaussiansobjectives
Simulation uses 100 scripted demonstrations for each of ten tasks and six 256 × 256 RGB-D cameras. Real policies are described as trained from scratch without simulation pretraining. Numerical loss weights and the optimization schedule are absent from the supplied paper. sim-protocolreal-resultslimits
During inference
At each observation, the policy predicts the next bimanual targets conditioned on language. The real-robot setup uses one camera at inference despite multi-view training. pipelinereal-setup
The verified control path is direct policy prediction. The paper presents future reconstruction as representation supervision; it does not specify candidate-action search, imagined rollout scoring or inference-time optimization with the world model. pipelinedynamicsobjectives
5.5 Implementation flow
- Encode observed geometry
Project RGB-D observations into a calibrated voxel space. A sparse 3D convolutional representation network produces volumetric features shared by the Gaussian regressor and policy. Position classification uses 100³ bins; orientation uses 5-degree bins for each of three axes. pipeline
- Label the Gaussian scene
A feed-forward regressor predicts each primitive’s position, color, orientation, scale, opacity and three-dimensional instance logit. Rendered logits distinguish task-relevant instances; GroundedSAM is prompted with instruction keywords to provide labels. The exact mapping of those logits to arbitrary task objects is not fully specified. gaussians
- Predict interacting movements
The leader consumes current Gaussians, stabilizing action and volumetric features. Equation (5) gives the follower the leader output, both arm actions and the same features. Only primitive positions and orientations change; color, scale, opacity and logits stay fixed. Rendering the predicted scene enables supervision without ground-truth future Gaussians. dynamicsobjectives
- Decode executable action targets
PerceiverIO selects discrete action candidates from the volumetric representation and instruction. Actions specify both end-effectors and a collision-avoidance option; predicted images are auxiliary outputs, not physical task completion. pipelineobjectives
6. Experiments & results
ManiGaussian++ improves language-conditioned bimanual imitation learning by training visual features to reconstruct task-labeled Gaussian scenes and predict their action-conditioned deformation. A stabilizing-arm leader precedes an acting-arm follower; a separate policy maps the enriched features to robot actions. The strongest evidence is higher multi-task success, with unresolved protocol details and a discrepancy between headline and detailed real-world averages.
6.1 Read the original evidence
Table I. Ten-task mean success rises to 35.6%, with gains distributed unevenly across tasks. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two horizontal blocks as one ten-task evaluation: the lower block continues the task list and adds average success and average rank. Higher success is better; lower rank is better. The source caption specifies multi-task agents trained with 100 demonstrations per task and evaluated over 25 episodes. Section IV-A supplies the six-camera observation setup and the 25-step completion budget. Compare the shaded ManiGaussian++ row with both baselines within each column before moving to the average. Sweep-to-dustpan shows the largest conspicuous gain, while handover-easy illustrates why the average should not be interpreted as a win on every task. sim-protocolsim-results
What it supports. ManiGaussian++ reports 35.6% mean success, versus 18.8% for ManiGaussian and 15.4% for PerAct2. Calculated absolute gains are 16.8 and 20.2 percentage points. Sweep-to-dustpan reaches 92%, but lift-tray remains at 8%; the representation improves the benchmark average without solving all of its tasks.
Where the evidence stops. The ten displayed ManiGaussian++ task values have an equal-task mean of 35.2%, whereas the table prints 35.6%. The reported average and gains calculated from it are retained; the aggregation or rounding procedure needed to reconcile the difference is unspecified, and the diagnostic 35.2% does not replace the author’s result. The table supplies no uncertainty or seed count. Several percentages cannot come from one 25-episode tally, so averaging across runs remains unclear. The paper specifies no held-out-task split; these multi-task results do not establish generalization to unseen task families.
Figure 4. The real-robot comparison reports a 62.22% detailed average and substantial variation between tasks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the legend to identify PerAct2 in light gray, ManiGaussian in darker gray and ManiGaussian++ in gold. The first nine groups are individual tasks; the last is their average. Section IV-A defines the evaluation metric as task success percentage, although the plot itself lacks a printed y-axis title. Compare within a task before reading the average, because the gold bars range widely. The source describes a dual-UR5e setup, multi-view training and single-camera inference. It states 30 demonstrations and 10 evaluation episodes without explicitly defining their per-task allocation. Section IV-E says these policies were trained without simulation pretraining. real-setupreal-resultsproblemlimits
What it supports. Section IV-E gives a 62.22% mean for ManiGaussian++, against 45.56% for ManiGaussian and 31.11% for PerAct2. The plot shows 100% on Lift Box and Ping Pong, but only 20% on Handover Bowl, Press Handsan and Relocate Brush. Physical execution is demonstrated, with uneven reliability across the evaluated tasks.
Where the evidence stops. The detailed mean conflicts with the abstract and conclusion’s 60%; both values are preserved. No error bars appear, and the demonstration allocation is unclear. The physical-task comparison does not establish robustness to arbitrary new objects, tasks or lighting conditions.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RLBench2 ten-task bimanual manipulation Multi-task training with 100 demonstrations/task, six RGB-D cameras; Table I states 25 evaluation episodes. Completion budget: 25 steps. | 35.6% Mean task success rate (%) | ManiGaussian: 18.8%; PerAct2: 15.4%. Using the author-printed averages, calculated gains are 16.8 and 20.2 percentage points, respectively; relative gains are 89.36% and 131.17%. The abstract calls the improvement 20.2%; the printed averages imply a 20.2-percentage-point difference. However, the ten displayed ManiGaussian++ task values have an equal-task mean of 35.2%, rather than the printed 35.6%; the aggregation or rounding procedure needed to reconcile this difference is unspecified. The reported 35.6% is retained, and 35.2% is only a diagnostic calculation. The table does not show superiority on every task: handover-easy is 40% versus PerAct2’s 41%. sim-protocolsim-resultsproblem |
| RLBench2 sweep to dustpan Same multi-task protocol as Table I. | 92% Task success rate (%) | ManiGaussian: 24%; PerAct2: 0%. A large coordination-task gain accompanies the average improvement. It does not establish transfer to unseen task families. sim-protocolsim-results |
| Three-task component ablation Table II: sweep-to-dustpan, handover-item and push-box; progressive addition of components. | 60.00% with all components Mean task success rate (%) | No components: 5.67%; Gaussian Splatting: 20.00%; plus task-oriented GS: 26.67%. Using the author-printed Average cells, the hierarchical model adds 33.33 percentage points over row 3 (60.00% versus 26.67%). However, the last row’s displayed task values of 92%, 20% and 48% have an equal-task mean of 53.33%, not the printed 60.00%; the source does not specify an aggregation procedure that reconciles them. The printed values and their difference are retained, while 53.33% is only a diagnostic calculation. This is a three-task subset, not the ten-task result, and no matched-capacity alternative ordering is tested. ablation |
| Nine-task real-world bimanual manipulation UR5e dual-arm setup; the text states 30 training demonstrations and 10 evaluation episodes without explicitly defining their per-task allocation; single-camera inference. | 62.22% in Section IV-E Mean task success rate (%) | ManiGaussian: 45.56%; PerAct2: 31.11%. Figure 4 agrees with the detailed average, but the abstract and conclusion report 60%. This discrepancy is unresolved. The result concerns trained real-world tasks; no simulation pretraining is reported. real-setupreal-resultsproblemlimits |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table II. The author-printed three-task Average rises from 26.67% to 60.00%; the final row’s task cells do not reproduce that printed mean. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the checkmarks before reading the scores. Row 1 has none of the three components; row 2 introduces Gaussian Splatting, row 3 adds task-oriented GS, and row 4 adds the hierarchical world model. The rightmost mean summarizes only sweep-to-dustpan, handover-item and push-box. The most relevant comparison for the hierarchy is row 3 against row 4, since the earlier two components remain checked in both rows. The individual columns show that the final increment is uneven: sweep-to-dustpan moves from 32 to 92, handover-item from 16 to 20, and push-box from 32 to 48. ablationsim-protocol
What it supports. The author-printed Average column progresses through 5.67%, 20.00%, 26.67% and 60.00%. Subtracting the final two printed cells gives 33.33 percentage points, but this gain is conditional on the unreconciled printed averages, as explained below. The displayed task values improve in all three final-row comparisons, with the largest increase in sweeping; they support improvement on the selected tasks without resolving the aggregate discrepancy.
Where the evidence stops. The final row’s displayed task values of 92%, 20% and 48% have an equal-task mean of 53.33%, rather than the printed 60.00%. Their equal-task gain over row 3 is 26.67 percentage points, not the 33.33 points obtained from the printed Average cells. These are diagnostic calculations, not replacements for the author’s results; the source does not specify aggregation that reconciles the difference. The caption also refers to 12 tasks grouped into three categories, whereas the main experiment describes ten; supplementary grouping details are absent. Progressive additions do not isolate ordering from model capacity, and the table includes neither a reversed hierarchy nor a matched parallel predictor.
Figure 3. Selected novel-view predictions make the intended representation benefit visible, under a separate visualization setting. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row starts with a front-view observation. The middle triplet shows a ground-truth novel view at the current time beside ManiGaussian and ManiGaussian++ reconstructions. The right triplet repeats the comparison for a future time step. Section IV-D identifies the upper case as press-buttons and the lower case as sweep-to-dustpan. Follow the red boxes to compare gripper and object details, then inspect the PSNR annotations for the corresponding image rather than mixing current and future columns. Crucially, the caption says behavior-cloning loss is turned off for this illustration, so these images have a different training setting from the main policy results. diagnosticobjectives
What it supports. In the future-view columns, the upper example reports PSNR 21.90 for ManiGaussian++ versus 21.36 for ManiGaussian; the lower reports 21.95 versus 20.73. These selected examples are consistent with improved image prediction and motivate the proposed interaction model, but they are not a dataset-wide prediction benchmark.
Where the evidence stops. Rendering quality is distinct from executed manipulation success. These examples omit aggregate uncertainty and use behavior cloning disabled. They cannot by themselves demonstrate that better PSNR causes the success-rate gains of the jointly trained policy.
7. Analysis & limitations
7.1 What the evidence leaves open
Calibrated multi-view cameras for supervision increase real-robot deployment cost. limits
Table I gives means without uncertainty or seed counts; several percentages are incompatible with a single 25-episode tally, leaving the averaging procedure unclear. Table II’s caption says 12 tasks although the main protocol lists ten. No held-out-task split is specified. sim-resultssim-protocolablation
Figure 3 contains selected rendering examples with behavior cloning disabled. Its PSNR values cannot establish the dynamics accuracy of the deployed policy or prove that rendering quality causes control gains. diagnostic
Figure 2 and nearby prose use s/a intermediate-state subscripts, whereas Equation (5) switches to r/l. The equation explicitly gives the follower both actions; the diagram labels only its acting-action input. Arm roles should not be assumed to equal fixed left/right identities. dynamicspipeline
7.2 Questions for discussion
- Would a parameter-matched parallel deformation model match the leader–follower hierarchy?
- How are acting/stabilizing roles assigned when both arms move symmetrically or exchange roles?
8. Reproducibility audit
8.1 Requirements and known gaps
Recreating the simulation requires RLBench2 demonstrations, calibrated six-camera observations, instruction-derived masks, voxel encoding, Gaussian regression and deformation, and PerceiverIO action classification. Loss weights, optimizer, training duration, seeds, role-assignment rules and deformation rotation details need clarification. sim-protocolgaussiansdynamicsobjectives
The real setup specifies two UR5e arms, Robotiq 2F-85 grippers, two Xbox controllers, two RealSense cameras at 640 × 480 and 30 Hz, and an RTX 4080 for evaluation. That GPU is not a documented training-compute budget. Per-task data allocation and supplementary task definitions remain unresolved. real-setupreal-resultslimits
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does stabilizing-first conditioning beat matched alternatives?
Reader-proposed experiment, not performed: on the three Table II tasks, train the published hierarchy, a reversed acting-first hierarchy and a parallel two-arm predictor with matched parameter count and optimization budget. Keep demonstrations, cameras, role labels, Gaussian representation and all loss weights fixed. Use identical held-out episodes and five training seeds as a proposed new evaluation design; report per-task success under the paper’s 25-step budget and future-image squared error with behavior cloning enabled. A reproducible advantage for stabilizing-first conditioning over both matched controls would support the ordering mechanism. If parallel prediction matches success, the original ablation would support extra predictive capacity more clearly than the specific dependency order. dynamicsobjectivesablationsim-protocoldiagnostic
Check 2: Do consistent task labels improve coordination?
Reader-proposed experiment, not performed: first document the exact instance-channel and arm-role mapping omitted from the source. Then compare normal task-label supervision with labels randomly reassigned between task-relevant instances for each training frame, and with the task loss disabled. Include a consistent renaming of all label channels as a control; a mere channel-name change should preserve the supervision problem. Keep the hierarchy, demonstrations, cameras, other losses and training budget fixed. Evaluate held-out mask agreement, future-image error and per-task success, especially sweep-to-dustpan and handover-item. If inconsistent labels damage segmentation but leave success unchanged, their claimed role in coordination would be weakened; consistent renaming should not itself cause a systematic loss. gaussiansobjectivesdynamicsablation
8.3 Reading coverage
Visual audit: All eight pages of the supplied arXiv v1 PDF were rendered and visually inspected: title/authors/affiliations and Figure 1 on p. 1; introductory claims on p. 2; Figure 2 and formulation on p. 3; Gaussian definitions and Eqs. (1)–(5) on p. 4; objectives, protocol and Table I on p. 5; Table II, Figure 3, real hardware and evaluation details on p. 6; Figures 4–5, real results, limitations and start of references on p. 7; remaining references on p. 8. Every final crop was separately viewed, including full table headers, plot legend/axes and diagnostic annotations. The figure–equation follower-input and state-subscript discrepancies are disclosed. No appendix appears in this PDF; separate supplements and videos were not supplied or inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, authors, affiliations, Abstract and Section I
- PDF p. 2: Section I continuation and Section II Related Work
- PDF pp. 3–5: Section III-A–E, formulation, pipeline, Gaussian representation, dynamics and objectives
- PDF pp. 5–7: Section IV-A–E, simulation, real setup, comparisons, ablations and qualitative analysis
- PDF p. 7: Section V Conclusion and limitations
- PDF pp. 7–8: complete References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only image limitation was resolved by inspecting all eight PDF pages and every final crop. No appendix is present in this PDF.
- The supplied source is arXiv v1 dated 24 June 2025. Title and all ten authors match the catalog. The catalog cites IROS 2025 proceedings; that separate edition and any revision differences were not supplied or verified.
- Separate supplements and videos were not supplied. Code was not inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author block, affiliation footnotes and arXiv margin
The exact catalog title and ten authors appear on arXiv:2506.19842v1 [cs.RO], 24 June 2025. Four numbered affiliations identify Tsinghua Shenzhen International Graduate School, Hainan University, Nanyang Technological University, and Tsinghua University Department of Automation.
Go to primary source ↓problemPDF pp. 1–2, Abstract and Section I; p. 3, Section III-A
The method addresses language-conditioned dual-arm manipulation and multi-body dynamics; the abstract reports 20.2% improvement in simulation and 60% real-world success. The introduction also reports a 131.17% relative simulation improvement.
Go to primary source ↓pipelinePDF p. 3, Figure 2 and Sections III-A–B
Calibrated RGB-D observations become voxels and sparse-convolutional volumetric features. A Gaussian prediction branch and PerceiverIO policy share these features. Each arm action includes discretized position, orientation, gripper openness and a collision-avoidance flag.
Go to primary source ↓gaussiansPDF p. 4, Section III-C, Eqs. (1)–(3); p. 5, Eq. (7)
Gaussian primitives carry position, color, orientation, scale, opacity and a three-dimensional instance logit. Alpha compositing renders color and logits. Instruction keywords prompt GroundedSAM for instance supervision; rendered logits are softmax-normalized for cross-entropy.
Go to primary source ↓dynamicsPDF p. 4, Section III-D, Eqs. (4)–(5); p. 3, Figure 2
The leader conditions on the stabilizing action and representation; Eq. (5) gives the follower the leader output, both actions and representation. Only position and orientation change. Figure/prose use intermediate s/a subscripts, but Eq. (5) uses r/l and leaves the renderer input without either subscript.
Go to primary source ↓objectivesPDF p. 5, Section III-E, Eqs. (6)–(10)
Current reconstruction and future prediction use image squared error, task masks use cross-entropy, and behavior cloning uses left/right action cross-entropy. Their weighted sum trains the agent; the three auxiliary weights are named but numerical settings are not provided.
Go to primary source ↓sim-protocolPDF p. 5, Section IV-A Simulation and Table I caption; p. 6, Section IV-A Baselines
RLBench2 evaluation covers ten tasks, with 100 scripted demonstrations per task and six 256 × 256 RGB-D cameras. Table I states evaluation over 25 episodes. Success requires task completion within 25 steps. ManiGaussian is adapted to bimanual action dimensions.
Go to primary source ↓sim-resultsPDF p. 5, Table I, all task columns and Average Success/Rank
Table I prints mean success of 35.6% for ManiGaussian++, 18.8% for ManiGaussian and 15.4% for PerAct2; ranks are 1.1, 2.2 and 2.5. The ten displayed ManiGaussian++ task percentages sum to 352, giving a diagnostic equal-task mean of 35.2%, so their reconciliation with the printed 35.6% is unresolved. Sweep-to-dustpan success is 92%, 24%, and 0%. Handover-easy gives 40%, 36%, and 41%, respectively.
Go to primary source ↓ablationPDF p. 6, Table II, rows 1–4 and caption; Section IV-C
For sweep-to-dustpan, handover-item and push-box, Table II prints Average values of 5.67%, 20.00%, 26.67% and 60.00% for the four progressive configurations. The final step changes the displayed task rates from 32/16/32 to 92/20/48. Those last-row values have an equal-task mean of 53.33%, so the printed 60.00% and the source’s 33.33-percentage-point increase are retained as author-reported quantities with unresolved aggregation. The caption refers to 12 tasks categorized into three groups and defers details to a supplement.
Go to primary source ↓diagnosticPDF p. 6, Figure 3 and caption; p. 7, Section IV-D continuation
The two example rows depict press-buttons and sweep-to-dustpan, comparing current and future novel views. Future PSNR is 21.90 versus 21.36 in the top row and 21.95 versus 20.73 below, for ManiGaussian++ versus ManiGaussian. The caption explicitly disables behavior-cloning loss for this illustration.
Go to primary source ↓real-setupPDF p. 6, Section IV-A Real Robot; p. 7, Figure 5
Hardware comprises two UR5e arms, Robotiq 2F-85 grippers, two Xbox controllers and two RealSense RGB-D cameras at 640 × 480 and 30 Hz. The text states 30 demonstrations, 10 evaluation episodes and an RTX 4080 GPU; it specifies multi-view training and single-camera inference without saying whether demonstrations are counted per task.
Go to primary source ↓real-resultsPDF p. 7, Figure 4 and Section IV-E
The nine-task real-world average in Section IV-E is 62.22%, versus 45.56% for ManiGaussian and 31.11% for PerAct2, consistent with the plotted averages. ManiGaussian++ bars reach 100% on Lift Box and Ping Pong and 20% on Handover Bowl, Press Handsan and Relocate Brush. Training is described as from scratch without simulation pretraining or sim-to-real transfer.
Go to primary source ↓limitsPDF p. 7, Section V; pp. 4–6, Sections III-D–E and IV-A; p. 7, Section IV-E
The conclusion identifies calibrated multi-view supervision cost as a limitation and repeats 60% real-world success. The supplied methods give no numerical auxiliary-loss weights, optimizer schedule, training duration or seed count. Real-world qualitative task details are deferred to supplementary videos.
Go to primary source ↓8.5 Primary sources
ManiGaussian++: General Robotic Bimanual Manipulation with Hierarchical Gaussian World Model ↗
PDF · 6,377 extracted words
Source fingerprint
fa3e3c8d42fec18ad0c4aede2ea8d2d86d7bf7c6d831aef79091207377078a3d