GWM: Towards Scalable Gaussian World Models for Robotic Manipulation
1. Paper overview
In one sentence: GWM predicts action-conditioned futures in a compact Gaussian scene space, improving average policy performance while leaving exact conditioning and several evaluation comparisons unresolved. e-probleme-encodinge-dynamicse-policye-videoe-ile-reale-ablation
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors seek a world model that preserves geometry and visual detail without slow per-scene reconstruction. Their premise is that image prediction can miss manipulation-relevant spatial details, while learned feed-forward Gaussian reconstruction could make explicit 3D prediction practical for policy training. e-probleme-encoding |
| Core mechanism | Source description A Gaussian scene autoencoder and conditional diffusion transformer turn variable-sized reconstructed scenes into compact predictive states. e-encodinge-vaee-dynamics |
| A key reported result | Franka-PnP real-robot cup placement: 13/20 (65%): cup 7/10, plate 3/5, table 3/5. Successful trials; success rate. GWM features versus RGB Diffusion Policy; 30 demonstrations; 20 distractor trials per method. Diffusion Policy 7/20 (35%): cup 6/10, plate 1/5, table 0/5. Six additional successes give a 30-percentage-point gain. The small, single-task evaluation does not establish broad deployment robustness. e-real |
| Reading caution | Reader analysis Figure 2's Q/KV labels conflict with Section 3.2. Appendix B.2 further describes action conditioning through AdaLN. These descriptions establish conditioning but leave exact attention routing uncertain. e-conditioninge-architecture-details |
Core contributions
- Source description
A Gaussian scene autoencoder and conditional diffusion transformer turn variable-sized reconstructed scenes into compact predictive states. e-encodinge-vaee-dynamics
Figure 2. Gaussian reconstruction and latent diffusion form the predictive scene model. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the optional second image on the left: Splatt3R lifts the available views into a Gaussian scene. The first 3D VAE block compresses that scene; compact features and random noise enter the diffusion transformer. Time controls scale and shift through AdaLN, and the action enters the attention block. The final VAE block decodes the prediction into future splats. Check the small Q and KV labels carefully: the graphic assigns Q to the action branch and KV to latent tokens, whereas Section 3.2 says actions provide keys and values. This guide follows the text for that detail and leaves the conflict explicit. e-encodinge-vaee-dynamicse-conditioninge-policye-architecture-details
What it supports. The model predicts a future scene representation conditioned on a supplied action. The figure therefore explains a forward dynamics model, while Section 3.3 supplies the separate route from learned features or imagined transitions to a policy. Those interfaces support the catalog's Dual-system assessment.
Where the evidence stops. Attention routing is unresolved within the source. Appendix B.2 also describes action-conditioned AdaLN. The graphic is preserved unchanged; neither the diagram alone nor the prose alone establishes the exact implementation.
2. Motivation
2.1 The problem and the proposed response
The authors seek a world model that preserves geometry and visual detail without slow per-scene reconstruction. Their premise is that image prediction can miss manipulation-relevant spatial details, while learned feed-forward Gaussian reconstruction could make explicit 3D prediction practical for policy training. e-probleme-encoding
2.2 What this reading follows
A robot can generate a plausible future image yet miss the gripper motion that decides whether a grasp succeeds. GWM tackles that problem by reconstructing the observed scene as 3D Gaussian splats and learning to predict their future representation. Its compact autoencoder makes diffusion operate on scene latents, while separate policies consume its features or train on its imagined transitions. Read the evidence in three stages: the representation and conditioning mechanism, the difference between prediction quality and executed task success, and the component ablation. The reported gains are promising, but task regressions, small physical evaluations and inconsistencies within the source deserve equal attention. e-probleme-encodinge-dynamicse-policye-videoe-ile-reale-ablation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded Dual-system / Other mechanisms / Outside quadrants classification fits a forward action-conditioned world model coupled to separate policies. GWM neither jointly generates future scenes and executable actions nor extracts actions through inverse dynamics. Its 3D scene representation and model-based policy training also support the recorded subcategories. e-dynamicse-policye-rl
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Compress a reconstructed scene without discarding its supervision
The first modeling decision occurs before diffusion. Splatt3R uses a point-map reconstruction backbone to map one or two unposed images into Gaussians. Those primitives describe position, opacity, covariance and view-dependent appearance, so the scene can be rendered back to an image. A transformer autoencoder then converts the scene into a fixed collection of latent embeddings. Sampled Gaussian locations serve as queries into the larger set, providing a tractable representation for prediction. The source trains this representation using both center Chamfer distance and rendered-image L1 error: geometric reconstruction and appearance remain separate supervision signals. Appendix B.2 instantiates compression from 2,048 points to 512 latent points. Its probabilistic variant adds reparameterization and an optional KL term, which is not present in the main displayed loss. e-encodinge-vaee-architecture-details
5.2 2. Treat robot time and denoising time as different axes
Environment time advances from the current scene to the next scene under an action. Diffusion time instead indexes how much noise has been added to that next scene's latent representation. Training starts from the known future latent, corrupts it and learns to recover it while using observation and action history as conditions. At generation time the process begins with noise and denoises a future latent before decoding it into Gaussians. EDM's input, output and skip scaling change with the noise level; this stabilizes targets across easy and difficult denoising conditions. This is still conditional forward prediction: it does not select the robot action. For implementation, Section 3.2's action-as-keys/values description conflicts with Figure 2's action-as-query label, so the exact attention routing needs clarification. e-dynamicse-edme-conditioninge-vae
5.3 3. Trace the two policy interfaces before interpreting success
Reader interpretation: GWM's value should be evaluated at the interface where it enters control. In imitation learning, a separate policy consumes features after the first denoising step, then emits an action chunk; it need not execute a planned future scene. In model-based RL, the learned dynamics and reward model create additional transitions for training a DrQ-v2 actor-critic, while real environment interaction continues to provide feedback. Appendix B.2 describes a recurrent reward predictor whose hidden state is initialized using conditioning frames. These pathways explain why Table 1's image metrics, Table 2's imitation results and Figure 5's RL curves answer different questions. The physical cup-placement experiment tests the first pathway. It offers no evidence that an RL policy trained inside GWM was deployed on the physical arm. e-policye-rle-rewarde-videoe-ile-real
5.4 Training and inference
During training
The autoencoder is supervised by Gaussian-center Chamfer distance and rendered-image L1 loss. Dynamics training perturbs true future latents and learns an EDM-preconditioned denoiser. Appendix B.2 describes probabilistic latent sampling and optional KL regularization; Equation (4) itself contains no KL term. e-vaee-dynamicse-architecture-details
RoboCasa uses H-50 human or G-3000 generated demonstrations per task. The real robot uses 30 demonstrations. Appendix A lists 64×64 MetaWorld RGB observations and four-dimensional continuous actions. The source does not fully specify training-stage boundaries or which reconstruction modules are frozen. e-ile-reale-confige-architecture-details
During inference
Starting from observation context and an action, iterative denoising produces the next latent, which is decoded into a renderable scene. Model-based RL predicts one step per inference and uses ten-step model rollouts. The reward LSTM initializes hidden state through a conditioning-frame burn-in before prediction. e-dynamicse-confige-reward
The physical deployment tests imitation learning with a GWM representation. It does not demonstrate physical execution of a policy trained through online GWM-based RL; that experiment is in MetaWorld simulation. e-policye-rle-real
5.5 Implementation flow
- Lift and compress
Splatt3R uses Mast3R point maps and a Gaussian prediction head. Each primitive stores center, opacity, covariance and spherical-harmonic coefficients. Farthest-point samples query the full Gaussian set through cross-attention. Appendix B.2 specifies 2,048 input points and 512 latent points; its N/M notation differs from the main text's generic N latent count. e-encodinge-architecture-details
- Predict the next scene
A DiT denoises the future latent while conditioning on observations and actions. The decoder reconstructs Gaussian parameters for rendering. Section 3.2 specifies RoPE, time-conditioned AdaLN, action keys/values and RMSNorm. Figure 2 instead labels the action branch Q and latent branch KV; the prose formulation is reported here, with the implementation unresolved. e-vaee-dynamicse-conditioning
- Connect predictions to control
Imitation learning supplies a downstream BC-transformer or Diffusion Policy with first-step denoising features and predicts action chunks. RL adds a reward model and uses imagined transitions to augment DrQ-v2's replay buffer. Algorithm 1 alternates environment collection, world-model fitting and policy optimization; it does not establish a new monotonic-improvement theorem. e-policye-rle-reward
6. Experiments & results
GWM lifts RGB observations into Gaussian splats, compresses them, and predicts action-conditioned future scene latents with diffusion. Separate policies use these features for imitation learning or learn from imagined transitions. The experiments support improved average manipulation performance, with task regressions and unresolved implementation discrepancies that qualify the scalability claims.
6.1 Read the original evidence
Table 1. Future-scene metrics improve modestly across both evaluated domains. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each dataset as a paired comparison between iVideoGPT and GWM. Down arrows mean lower FVD and LPIPS are preferred; up arrows mean higher PSNR and SSIM are preferred. The source caption specifies that SSIM and LPIPS are multiplied by 100, so the displayed 9.0 LPIPS represents 0.090 on its unscaled convention. MetaWorld's GWM row is 73.0, 20.6, 82.8 and 9.0; Franka-PnP's is 61.5, 28.0, 91.0 and 4.5. Section 4.1 evaluates predictions conditioned on unseen validation action trajectories after training on demonstrations. These rows concern generated scenes rather than completed manipulation trials. e-video
What it supports. The paired improvements are consistent across metrics and domains, although their magnitudes are small. For example, MetaWorld FVD decreases from 75.0 to 73.0 and Franka-PnP FVD from 63.2 to 61.5. The paper's qualitative gripper examples motivate why such aggregate metrics may understate control-relevant differences.
Where the evidence stops. Table 1 supplies no uncertainty or numerical contact-dynamics measurement. Better FVD or rendered appearance cannot by itself establish physically correct rollouts, successful robot execution or robustness outside the evaluated trajectories.
Table 2. Average imitation gains coexist with clear task-level regressions. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each task has H-50 (50 human demonstrations) and G-3000 (3,000 generated demonstrations) columns. Compute percentage-point differences from the BC-transformer and green GWM method rows: the printed delta rows contain an error. In the third block, TurnOn Stove / G-3000 shows 46% and 80% but prints delta +24; subtraction gives +34 percentage points. The faithful crop retains that source error. Evaluation uses 50 episodes with varied floor plans and styles. Read all four blocks before accepting universal improvement: CoffeePress Button rises from 48 to 76 in H-50, while CounterToCab falls from 6 to 4. CounterToCab also falls from 28 to 22 in G-3000. Use method rows for averages, not the printed deltas. e-ile-ablatione-il-delta
What it supports. Unweighted averages of the 24 method-row entries give H-50 success of 39.25% versus 28.75%, and G-3000 success of approximately 55.17% versus 47.58%. These reader calculations confirm mean gains of 10.5 and approximately 7.6 percentage points despite the erroneous printed TurnOn Stove delta; they do not imply that every task improves.
Where the evidence stops. The two demonstration budgets provide limited scaling evidence and do not equate data quality or compute. The full GWM CabToCounter H-50 entry is 18%, whereas Table 4's full model reports 24%; the source does not explain that difference.
Figure 5. Learned-model rollouts improve the aggregate RL learning curve, with protocol qualifications. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. First inspect the aggregate panel on the left, then use the six task panels to see whether its pattern is shared. The horizontal axis is environment steps in units of 10,000 and the vertical axis is success rate. Green denotes GWM; orange denotes GWM w/ PT. Keep these separate when comparing them with the two blue iVideoGPT variants. The caption defines shading as a 95% confidence interval across three seeds, with each point evaluated over 20 episodes. Section 4.3 describes GWM-generated transitions augmenting a DrQ-v2 replay buffer and states matched context length, horizon and a maximum of 100,000 steps. e-rl
What it supports. The aggregate plot supports a policy-learning advantage from using GWM. The authors characterize convergence as approximately twice as fast as iVideoGPT; this is their qualitative summary of the curves, not a tabulated timing result or a reader-measured ratio. Task panels show why aggregate performance should be checked against individual behaviors.
Where the evidence stops. The prose says neither method uses pretrained initialization, yet the legend includes w/ PT curves. The source does not reconcile these conditions or define convergence. Overlapping bands and only three seeds also limit task-specific conclusions.
Table 3. The physical cup-placement test records six additional successful trials with GWM features. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. These entries are successful trials divided by attempted trials, rather than image-generation scores. Compare methods within each distractor row, preserving the different denominators: ten cup trials and five each for plate and table distractors. The total therefore sums counts, giving 7/20 for RGB Diffusion Policy and 13/20 for its GWM-based counterpart. Section 4.4 describes 30 training demonstrations, an FR3 arm with Panda gripper, and one third-view D435i camera supplying RGB-only observations. The task is to pick a colored cup and place it on a plate. This is the imitation-learning use of GWM's representation. e-reale-policye-rl
What it supports. The reported success rate increases from 35% to 65%, which is a gain of 30 percentage points. Most additional successes occur in the plate and table distractor groups. This supports a useful representation benefit in the tested physical task, while keeping the result separate from simulated model-based RL.
Where the evidence stops. Twenty trials and one task cannot establish broad physical generalization. No uncertainty is reported. Figure 6's caption calls the arm Panda, while its image label and Section 4.4 identify an FR3 with Panda gripper.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Action-conditioned future scene prediction Unseen validation action trajectories; models trained on demonstrations. Table 1 reports MetaWorld and Franka-PnP. | GWM: MetaWorld 73.0 / 20.6 / 82.8 / 9.0; Franka-PnP 61.5 / 28.0 / 91.0 / 4.5. FVD ↓; PSNR ↑; SSIM ×100 ↑; LPIPS ×100 ↓ | iVideoGPT: 75.0 / 20.4 / 82.3 / 9.5 and 63.2 / 27.8 / 90.6 / 4.9, respectively. Every listed metric improves, but these are scene-prediction metrics; no uncertainty or explicit contact-accuracy measurement is supplied. e-video |
| RoboCasa multi-task imitation learning 24 tasks; H-50 or G-3000 demonstrations per task; evaluation over 50 episodes with different floor plans/styles. | Reader-computed unweighted means from the 24 method-row entries, not the printed delta rows: 39.25% H-50; 55.17% G-3000. Success rate (%) | BC-transformer: 28.75% and 47.58%; gains 10.5 and approximately 7.6 percentage points, matching the prose. Average improvement is supported, universal improvement is not: CounterToCab falls from 6% to 4% in H-50 and 28% to 22% in G-3000. TurnOn Stove / G-3000 prints delta +24 beside 46% and 80%; those method entries differ by +34 percentage points. This source inconsistency does not change the means computed from the method rows. e-ile-il-delta |
| MetaWorld model-based reinforcement learning Six tasks; DrQ-v2 with synthetic rollouts; stated maximum 100,000 steps and matched context/horizon. Figure 5 includes separate w/ PT variants. | Authors report approximately 2× faster convergence than iVideoGPT. Success-rate learning curves; convergence speed | Figure 5 includes iVideoGPT, DrQ-v2 and DreamerV3; shaded 95% confidence intervals cover three seeds, with 20 evaluation episodes per point. The aggregate curve supports a learning advantage. The paper gives no operational convergence threshold, and its no-pretraining prose does not explain the additional PT curves; the 2× figure is an author characterization. e-rl |
| Franka-PnP real-robot cup placement GWM features versus RGB Diffusion Policy; 30 demonstrations; 20 distractor trials per method. | 13/20 (65%): cup 7/10, plate 3/5, table 3/5. Successful trials; success rate | Diffusion Policy 7/20 (35%): cup 6/10, plate 1/5, table 0/5. Six additional successes give a 30-percentage-point gain. The small, single-task evaluation does not establish broad deployment robustness. e-real |
| Gaussian representation and autoencoder ablation RoboCasa PnP CabToCounter, Human-50; Table 4's own ablation protocol. | Success 4 → 18 → 24; FVD 67.8 → 65.3 → 62.4 for image model, GS, then GS plus 3D VAE. Imitation success rate (%) and FVD | Adding GS lowers PSNR from 27.2 to 26.9 despite improving success; the full system reaches PSNR 28.1. Visual fidelity and control need not track identically. Table 2 reports 18% for full GWM on this named H-50 task, versus 24% here; the source does not reconcile the settings. e-ablatione-il |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. The ablation separates the Gaussian representation change from adding its compact autoencoder. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read downward as components are added. Crosses in the first row indicate a diffusion-based image model without GS or the 3D VAE. The next row adds Gaussian splatting; the final row also adds the autoencoder. All rows concern PnP CabToCounter using the Human-50 data according to the caption. Compare the rightmost success-rate column with the reconstruction columns instead of assuming they move together. Adding GS raises success from 4% to 18% while PSNR falls from 27.2 to 26.9. Adding the VAE then raises success to 24%, reduces FVD to 62.4 and raises PSNR to 28.1. e-ablatione-il
What it supports. Within this ablation, both architectural additions help task success. The GS-only row is especially instructive: control improves despite lower PSNR. Reader interpretation: a useful representation may preserve task-relevant geometry that a pixel metric does not capture, although this table does not directly measure that geometry.
Where the evidence stops. This is one task, with no uncertainty or matched-compute breakdown. Table 2's 18% full-model result for the same named H-50 task differs from the 24% shown here; the protocols must remain separate until reconciled.
7. Analysis & limitations
7.1 What the evidence leaves open
Figure 2's Q/KV labels conflict with Section 3.2. Appendix B.2 further describes action conditioning through AdaLN. These descriptions establish conditioning but leave exact attention routing uncertain. e-conditioninge-architecture-details
The paper attributes gains to spatial/contact understanding, but provides image metrics and task success rather than direct physical-consistency tests. Data scaling is examined at two RoboCasa demonstration budgets, without a compute-scaling study. e-probleme-videoe-ile-rl
Figure 6's caption names a Panda arm, whereas its image label and Section 4.4 identify an FR3 arm with Panda gripper. The report follows the latter agreement and preserves the caption discrepancy. e-real
7.2 Questions for discussion
- Would geometry-aware prediction still improve control when parameter counts and training budgets are matched?
- How sensitive is the deployed imitation policy to the action condition used when extracting GWM features?
8. Reproducibility audit
8.1 Requirements and known gaps
Table A1 supplies rollout horizon 10, two context frames, sequence length 12, world-model batch 16 and AdamW at 1e-4. Table A2 gives a 512-dimensional, six-layer, eight-head imitation policy, context/frame stack 10, batch 16 and L2 regularization 0.01. Its training-data row specifies H-50 only. e-config
Required clarifications include the denoising sampler and step count, reconstruction freezing, KL choice/weight, the imitation feature extractor's action input at deployment, PT initialization, and the Table 2/Table 4 mismatch. Hardware counts, runtime, software versions and a complete training schedule are not supplied in the implementation appendix. e-policye-architecture-detailse-confige-rle-ile-ablation
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Reproduce the component ladder under one declared protocol
Reader-proposed check, not performed: on RoboCasa PnP CabToCounter H-50, compare the image DiT, GS without VAE, and GS plus VAE using identical demonstrations, evaluation episodes, policy architecture and seed sets. Report parameter counts, training budgets and runtime alongside FVD, PSNR and task success; repeat with a matched-compute budget if costs differ. Keep the reconstruction initialization constant across the two GS variants. The discriminating observation is whether GS improves success even when its PSNR does not improve, and whether the VAE's added gain persists across seeds. Record whether the reproduced full-model setting corresponds to Table 2's multi-task run or Table 4's ablation; failure to reconcile 18% and 24% must remain visible. e-ablatione-ile-confige-architecture-details
Check 2: Check the causality of first-step features at policy deployment
Reader-proposed check, not performed: freeze one trained GWM and compare separate imitation readouts trained under three feature-conditioning rules: available past actions, a fixed zero-action placeholder, and the ground-truth next action used only as an offline oracle diagnostic. Match observation history, noise draws, readout capacity and data; never provide the oracle action during robot evaluation. Specify how each feasible rule fills the action input and keep that rule identical in training and deployment. A robust encoder benefit should remain measurable for at least one feasible rule. If the gain appears only with the oracle or collapses when training/deployment conditioning is matched, the feature-extraction procedure needs clarification before its control benefit can be reproduced. e-policye-conditioninge-dynamicse-ile-config
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version block, Figures 1–6, Tables 1–4, Algorithm 1, all method/evaluation pages and the embedded supplementary implementation pages with Tables A1–A2. All six final original crops were separately viewed for legibility, completeness and correspondence to the PDF. Figure 2's Q/KV discrepancy, Figure 5's PT/prose mismatch, Figure 6's arm-name discrepancy and the Table 2/Table 4 success-rate mismatch are preserved. Table 2's TurnOn Stove / G-3000 prints 46, 80 and delta +24; its +34-percentage-point method-row difference is explicitly distinguished from the printed delta, and averages use the method rows. Reference-only pages 9–12 were read in the complete text pass but not rendered; external supplemental videos and code were outside the supplied source.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 13, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; 1. Introduction
- 2. Related Work
- 3. Gaussian World Model; 3.1. World State Encoding; 3.2. Diffusion-based Dynamics Modeling; 3.3. GWM for Policy Learning
- 4. Experiments; 4.1. Action-conditioned Scene Prediction; 4.2. GWM-based Imitation Learning; 4.3. GWM-based Reinforcement Learning; 4.4. Real-world Deployment; 4.5. Ablation Analysis
- 5. Conclusion; Acknowledgement; References
- Supplementary Material: A. Datasets and Benchmarks; B. Implementation Details; B.1. EDM Preconditioning; B.2. Architectural Design; B.3. Hyper-parameters
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Edition scope: the title page identifies arXiv:2508.17600v2 [cs.RO], 17 September 2025. Its title and all seven authors match the catalog. The catalog cites ICCV 2025; this reading covers the supplied 14-page arXiv revision, including its embedded supplement. Neither v1 nor the venue edition was supplied for comparison, so differences from those editions remain unverified.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition omission was addressed by inspecting PDF pages 1–8 and 13–14 and all six final crops.
- Separate supplemental material availability has not been fully verified.
- The embedded supplementary text was read completely; external rollout videos, code, checkpoints and datasets were not inspected. No experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block and arXiv margin stamp
Exact catalog title and seven authors appear; the artifact identifies arXiv:2508.17600v2, 17 September 2025. Affiliations are Tsinghua University; State Key Laboratory of General Artificial Intelligence, BIGAI; and NTU's School of Electrical and Electronic Engineering.
Go to primary source ↓e-problemPDF p. 2, Section 1, Necessity of 3D Representation and Efficiency and Scalability
The motivation combines explicit geometry, detailed rendering and feed-forward efficiency; the claimed benefits include imitation learning and neural simulation.
Go to primary source ↓e-encodingPDF p. 3, Section 3.1, Eq. (1)–(2)
Defines Gaussian parameters and rendering, Splatt3R/Mast3R reconstruction, FPS queries and cross-attention compression.
Go to primary source ↓e-vaePDF p. 4, Section 3.1, Eq. (3)–(4)
The decoder reconstructs Gaussians; center Chamfer and rendered L1 losses supervise the autoencoder.
Go to primary source ↓e-dynamicsPDF pp. 4–5, Section 3.2, Eq. (5)–(10) and EDM discussion
Defines conditional future-state diffusion, Gaussian perturbation, denoising loss and EDM input/output/skip/noise preconditioning.
Go to primary source ↓e-conditioningPDF p. 4, Figure 2, Q/KV labels; PDF p. 5, Section 3.2, Implementation
Figure 2 labels the action branch Q and latent inputs KV; prose specifies actions as keys and values. Prose also specifies RoPE, time AdaLN and RMSNorm.
Go to primary source ↓e-policyPDF p. 5, Section 3.3 and Algorithm 1
First-denoising-step features feed separate imitation policies; RL fits dynamics/rewards from environment transitions and optimizes a separate policy using predictions.
Go to primary source ↓e-videoPDF pp. 5–6, Section 4.1; PDF p. 6, Table 1 and Figures 3–4
Prediction uses unseen validation action trajectories. Table 1 lists both methods' FVD/PSNR/SSIM/LPIPS on MetaWorld and Franka-PnP; SSIM and LPIPS are multiplied by 100. Figures illustrate gripper details and rollout examples.
Go to primary source ↓e-ilPDF p. 6, Section 4.2; PDF p. 7, Table 2, all 24 task columns and caption
H-50/G-3000 comparison evaluates 50 episodes. Averaging table cells gives 39.25/28.75 and 55.1667/47.5833 for GWM/baseline. CounterToCab regresses in both settings; GWM CabToCounter H-50 is 18.
Go to primary source ↓e-rlPDF pp. 6–8, Section 4.3; PDF p. 7, Figure 5 and caption
GWM rollouts augment DrQ-v2. Six-task curves use three seeds, 95% confidence intervals and 20 evaluation episodes. Prose states matched contexts/horizons, 100,000 steps, no pretrained initialization and approximately 2× faster convergence; the figure additionally labels w/ PT curves.
Go to primary source ↓e-realPDF p. 8, Section 4.4, Figure 6 and Table 3
Thirty demonstrations, RGB-only D435i observation and FR3/Panda-gripper setup accompany 20 trials. Table 3 reports baseline/GWM successes of 6/10 vs 7/10, 1/5 vs 3/5 and 0/5 vs 3/5, totaling 7/20 vs 13/20. Figure caption's Panda-arm wording conflicts with the image label and prose.
Go to primary source ↓e-ablationPDF p. 8, Section 4.5 and Table 4, all three rows
CabToCounter H-50 ablation reports success 4/18/24, FVD 67.8/65.3/62.4 and PSNR 27.2/26.9/28.1 for no GS/no VAE, GS only and GS plus VAE.
Go to primary source ↓e-edmPDF p. 13 (Supplement A1), Section B.1, Eq. (A1)–(A5)
Preconditioners use sigma_data=0.5; log noise follows a normal distribution with P_mean=-0.4 and P_std=1.2.
Go to primary source ↓e-architecture-detailsPDF p. 13 (Supplement A1), Section B.2
Specifies 2,048 input points and 512 latent points; probabilistic reparameterization and KL are described as variant/optional. DiT inputs stack current and noisy next observations, time and action embeddings. Freezing and full optimization stages are not specified.
Go to primary source ↓e-configPDF p. 13 (Supplement A1), Section A and Tables A1–A2; PDF p. 14 (Supplement A2), Section B.3
Lists MetaWorld image/action dimensions and rollout/training settings, including one-step inference and ten-step rollouts. Table A2 describes the H-50 imitation configuration. No hardware, software versions or complete training duration are specified here.
Go to primary source ↓e-rewardPDF pp. 13–14 (Supplement A1–A2), Section B.2, reward model paragraph
Reward prediction uses observation pairs, embedded actions, ResBlocks, optional attention, an LSTM and an MLP; conditioning frames burn in the LSTM hidden state before inference.
Go to primary source ↓e-il-deltaPDF p. 7, Table 2, third task block, TurnOn Stove / G-3000, BC-transformer, GWM and delta rows
The source prints BC-transformer 46%, GWM 80% and delta +24. Reader subtraction of the two method entries gives 80 - 46 = 34 percentage points, so the printed delta disagrees with its method rows. The original crop retains +24; the reported task means are calculated from method rows, not printed deltas.
Go to primary source ↓8.5 Primary sources
GWM: Towards Scalable Gaussian World Models for Robotic Manipulation ↗
PDF · 9,017 extracted words
Source fingerprint
58b8b5cf74dab26bc44bfd8fa7842e7b6a1623b44a8b2bdef422a50c0fbf5da3