Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoning
1. Paper overview
In one sentence: Spatial Policy couples geometry-conditioned video planning to a separate action diffusion policy and feedback loop, improving simulated control while depending on accurate spatial inputs and several incompletely specified interfaces. e03e05e06e13e15e16
| At a glance | What to know |
|---|---|
| Research problem | |
| Core mechanism | |
| A key reported result | Meta-World manipulation across 11 tasks: SP-R 86.7%; SP 82.0% Reported average task success rate (%). Simulated evaluation; Section 4.1 specifies 25 seeds and three views across 11 tasks, while Table 1 inconsistently says 800 episodes per task. VA-OR 53.7%; AVDC-R 43.1%; reader-computed SP-R minus VA-OR: 33.0 percentage points. Large aggregate improvement, but SP-R assembly remains 32.0%; replanning is not uniformly beneficial on every task. No confidence intervals are supplied. e12e13 |
| Reading caution | Reader analysis Simulator coordinates provide privileged geometry. Dataset size, explicit train/test partitions, human-evaluation sample counts and uncertainty are not adequately specified. Masking evidence is qualitative; Figure 7 also shows substantially weaker assembly performance despite the prose's broad claim of consistently high success. e03e09e12e21 |
Core contributions
Figure 1. A structured spatial plan links visual prediction, action generation and replanning. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the table on the left: each line combines an atomic action, a direction vector and a distance. The example moves left, forward and down before pressing. Follow these three fields into their separate embedding blocks, then into the task-and-plan condition of the central video generator. Generated frames feed the right-hand action module, where visual, flow and robot-state features condition a second diffusion model with horizon four. The bottom arrows return environmental spatial information to the plan table. Section 3.3 clarifies the two feedback events: a rejected video is regenerated, whereas execution stagnation prompts a revised spatial plan. e03e05e06e08e10e23e24
What it supports. The architecture supports a modular visual-planning interpretation. Geometry guides what future images should depict, and a separate policy translates those images into actions. The figure establishes this information flow; whether it improves control must be judged from the executed-task comparisons and ablations.
Where the evidence stops. The legend calls distances positive but uses 0.00 for an in-place push. Action-coordinate inputs also differ between A.1 and Algorithm 1. Figure 10's rejection route differs from Section 3.3; the latter governs the explanation here.
2. Motivation
2.1 The problem and the proposed response
A plausible video plan can omit contact geometry or become unusable after execution drift. The authors address the gap between visual prediction and control by exposing spatial intent to both planning and feedback. e02e03e06
2.2 What this reading follows
A video of a robot reaching an object does not tell a controller how to recover when the real gripper misses. Spatial Policy makes that connection explicit: a table records what to do, in which direction, and over what distance; a video generator imagines the result; an action policy follows visual goals; and feedback can rebuild the plan. The six source excerpts below separate this mechanism from its evidence. The simulated gains are substantial, but raw geometry explains much of the ablation improvement, and the single physical task succeeds in only 30% of trials. Read the aggregate scores alongside their task-level exceptions. e03e05e06e13e15e16
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
Separate video generation and goal-conditioned action diffusion support Dual-system × IDM: imagined future images feed an inverse-dynamics-style policy rather than joint video/action prediction in one model. Spatial geometry and multiple viewpoints support the subcategories, but do not establish a persistent learned 3D world representation or multiview reconstruction model. e03e04e05e09e15
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn an offset into a visual intention
Consider the button-press example in Figure 1. The simulator provides robot and object coordinates, and their difference gives GPT-4o a spatial constraint that a task name alone cannot express. The resulting table splits the approach into directed movements followed by an in-place interaction. Learned embeddings make those action, direction and distance fields available to the video U-Net alongside the CLIP task embedding. My interpretation is that the table acts as an intermediate representation of intent: it constrains which future scene changes should be generated without directly commanding the motors. Table 3 tests the value of that representation. Raw offsets already deliver most of its reported improvement, so the evidence motivates a stronger comparison against a deterministic plan builder before attributing the remaining benefit specifically to VLM reasoning. e03e04e10e16
5.2 Make an imagined frame actionable
The video generator predicts an eight-frame trajectory including the initial observation. The action model then receives the current view and a visual goal, together with motion and coordinate features, and denoises a four-action chunk. During action training, goals are sampled within the next twenty frames of demonstrations; the image encoder and coordinate MLP are trainable. This is a different learning problem from video synthesis: the target is a sequence of end-effector and gripper controls, and the environment actually executes those controls. Appendix A.6 advances the target frame when a weighted geometry/position/SSIM/flow similarity reaches 0.8, with a forced switch after 28 steps. The precise implementation remains ambiguous because the appendix's image-input description omits flow and Algorithm 1 includes object coordinates that A.1 excludes. e04e05e08e11e24
5.3 Separate screening a plan from recovering an execution
An accepted video can still fail when the gripper interacts with the environment. Section 3.3 therefore describes two correction opportunities. First, GPT-4o checks generated video plausibility and task progression; rejection resamples the video under the same spatial condition. Second, rule-based execution diagnostics detect drift or stagnation, refresh robot–object geometry and ask for a revised plan. Table 6's larger gain after adding execution replanning is consistent with this distinction. As reader analysis, however, it cannot show whether semantic correction or simply additional attempts caused the improvement, because a matched retry-budget control is absent. The source also leaves IsStuck unspecified, and Figure 10 draws rejection toward spatial replanning rather than the unchanged-condition route in Section 3.3. A reproduction should declare that choice explicitly. e06e10e19e23e24
5.4 Training and inference
During training
Both generators learn noise-prediction MSE objectives. Video training uses 128×128 eight-frame clips, AdamW at 10^-4, batch 8, ordinarily 100,000 updates, and a 1,000-step linear noise schedule. Table 8 separately specifies 10,000 initial plus 2,000 finetuning updates for its finetuned variant. e04e07
Action training uses horizon 4, goals sampled within the next 20 frames, batch 512, AdamW at 10^-4 with weight decay 10^-6, and a 100-step cosine noise schedule. ResNet and coordinate encoding remain trainable; no GPT-4o finetuning is described. e05e08e10
Expert Meta-World trajectories supply RGB, depth, masks and positions. Close-contact intervals identified from gripper–target geometry are sampled five times as densely as coarse intervals; task/view-dependent crops are manually chosen. Recovery finetuning adds expert-corrected failed rollouts. e09e20
During inference
Both generators use DDIM with classifier-free guidance. GPT-4o screens generated videos; Section 3.3 regenerates rejected videos under the same spatial condition. Execution stagnation/drift diagnostics trigger a fresh offset, revised plan, new video and new actions. Actions are applied sequentially to the environment. e06e07e08
Goal-frame matching combines geometry, positional appearance, grayscale SSIM and inverse optical-flow magnitude with weights 0.35, 0.35, 0.2 and 0.1. Advance when similarity is at least 0.8; force a switch after 28 steps. This is Appendix A.6, despite the main text's A.4 cross-reference. e05e11
5.5 Implementation flow
- Convert geometry into subgoals
Compute the object-minus-end-effector offset and prompt pretrained GPT-4o. Embed action and discretized direction with learned tables, project distance with an MLP, and concatenate subgoal embeddings with CLIP task features. The main formulation injects this condition throughout the video U-Net. e03e10
- Predict a visual trajectory
An Imagen/AVDC-style conditional U-Net denoises seven future RGB frames given the initial image and composite task/plan condition. These frames are high-level visual goals, not executed actions. e04
- Infer actions from visual change
Farnebäck flow supplies motion features through a CNN; image features and MLP-encoded robot coordinates condition a separate action diffusion U-Net. The appendix concatenates current and goal RGB into a six-channel image with a trainable ImageNet-pretrained ResNet-50. e05e08
6. Experiments & results
Spatial Policy turns robot–object geometry into a structured plan, conditions video diffusion on that plan, and uses a separate goal-conditioned diffusion policy to execute the imagined trajectory. Video screening and execution feedback close the loop. The strongest evidence is improved simulated task success; the physical demonstration remains limited.
6.1 Read the original evidence
Table 1. The aggregate Meta-World gain coexists with task-specific failures and occasional replanning regressions. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper and lower panels as a continuation of the same eleven-task comparison. R denotes replanning; VA is VideoAgent; VA-OR also uses online reinforcement learning. The shaded Average column belongs to the complete set of task columns. Compare SP-R with VA-OR for the strongest reported system-level contrast, then compare SP with SP-R to see whether replanning helps each task. Shelf placement improves, but basketball falls from 77.3% to 73.3%. Assembly remains the weakest SP-R task at 32.0%. These differences prevent the aggregate from being read as uniform mastery of manipulation. e12e13
What it supports. The reported average is 86.7% for SP-R versus 53.7% for VA-OR, a reader-computed 33.0 percentage-point difference. SP without the execution-replanning extension already reaches 82.0%. The table therefore supports a substantial simulated-control improvement beyond the effect of adding replanning alone.
Where the evidence stops. Table 1 says 800 episodes per task, but Section 4.1 describes 25 seeds × 11 tasks × three views and over 800 total rollouts. Preserve the reported rates; their precise denominators and confidence intervals remain unresolved. The displayed eleven SP-R rates also average about 86.53%, rather than the printed 86.7%, under equal task weighting. The 33.0-point comparison is arithmetic on the printed averages; their exact aggregation remains unresolved.
Table 2. Room-level results show where spatial conditioning helps and where its advantage disappears. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each row pools tasks from one room type; the experiment spans twelve navigation tasks with twenty random seeds per task. Read down the SP column before looking at its average. Kitchen is much stronger than Bathroom, despite using the same named method. Then compare horizontally against VideoAgent: SP improves Kitchen, Living Room and Bedroom, while Bathroom is 36.7% against 40.0%. Appendix A.3 describes these tasks as locating or approaching objects under changing viewpoints. The result should therefore be interpreted as embodied visual navigation, rather than as another physical robot manipulation benchmark. e12e14
What it supports. SP reports 59.6% average success, compared with 34.2% for VideoAgent and 31.3% for AVDC. The reader-computed improvement over VideoAgent is 25.4 percentage points. The room breakdown supports an uneven benefit, with Kitchen reaching 88.3% and no advantage in Bathroom.
Where the evidence stops. The table labels the method SP and does not identify it as SP-R. Explicit held-out scene partitions and uncertainty are not supplied, so these aggregates alone cannot establish broad generalization to unseen household layouts.
Figure 5. The source includes an actual robot execution sequence from two camera viewpoints. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the High row from left to right to see the arm's approach, lift and transfer relative to the tabletop. The Wrist row provides the complementary close view of the rabbit and destination box. Section 4.3 identifies this as execution on an SO-100 arm, not a generated-video example. The physical setup also changes the inputs: DepthAnything supplies estimated depth, and action prediction uses top-down current/goal frames plus a wrist-camera frame. The two rows help explain those viewing conditions, but the selected sequence contains neither a baseline comparison nor the complete set of experimental trials. e15
What it supports. The associated text reports 30% success over twenty pick-doll trials. This establishes that the pipeline can sometimes complete a physical grasp-and-place task despite estimated depth and incomplete observations. The low rate is a major boundary on transferring the much higher simulated averages to practical reliability.
Where the evidence stops. This is one selected sequence from one task. The images cannot establish the twenty-trial success rate on their own; that number comes from Section 4.3. No matched physical baseline or reported uncertainty supports a broader deployment claim.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Meta-World manipulation across 11 tasks Simulated evaluation; Section 4.1 specifies 25 seeds and three views across 11 tasks, while Table 1 inconsistently says 800 episodes per task. | SP-R 86.7%; SP 82.0% Reported average task success rate (%) | VA-OR 53.7%; AVDC-R 43.1%; reader-computed SP-R minus VA-OR: 33.0 percentage points. Large aggregate improvement, but SP-R assembly remains 32.0%; replanning is not uniformly beneficial on every task. No confidence intervals are supplied. e12e13 |
| iTHOR navigation across 12 tasks Four room types; 20 random seeds per task; locating/approaching objects under varying viewpoints. | SP 59.6% Reported average success rate (%) | VideoAgent 34.2%; AVDC 31.3%; SP minus VideoAgent is 25.4 percentage points. Kitchen reaches 88.3%, but Bathroom is 36.7% versus VideoAgent's 40.0%. These are simulated navigation results. e12e14 |
| Physical SO-100 pick-doll manipulation Place a yellow rabbit toy in a box; 20 trials; DepthAnything depth estimates and top-down/wrist images. | 30% Task success rate (%) | No matched physical baseline reported. Demonstrates some physical execution feasibility on one task, with limited reliability and no evidence of broad deployment generalization. e15 |
| Meta-World video-conditioning ablation Table 3 varies video conditioning; Table 4 separately varies action prediction. | Text/frame baseline 63.4%; raw Distance 79.4%; Subplan 82.0% Reported average success rate (%) | Table 4: heuristic AVDC action prediction 19.6% versus learned policy 63.4%. Raw geometry accounts for much of the conditioning gain; structured plans add 2.6 points on average but reduce the Hard score from 25.3% to 21.3%. The 63.4% row is not the complete AVDC system from Table 1. e13e16e17 |
| Meta-World feedback and training ablations Separate comparisons in Tables 5–7; improvements should not be summed. | No replanning 82.0%; generation replanning 83.4%; generation plus execution replanning 86.7% Reported average success rate (%) | Local versus global conditioning: 84.3% versus 82.0%; expert-assisted versus original training: 85.6% versus 82.0%. Feedback, conditioning location and recovery data each help in their reported comparison; this is not a complete factorial isolation of all components. e18e19e20 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Most of this ablation's average gain comes from providing geometry; structured plans add a smaller increment. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Treat AVDC in this table as the video-conditioning baseline in the ablation, not the complete AVDC system whose Table 1 average is 19.6%. Table 4 separately explains the learned action-policy improvement that reaches 63.4%. Here, Distance adds raw spatial offsets and Subplan turns those offsets into structured action cues. Move across each difficulty column to check whether abstraction consistently helps. It does not: the Hard and Very Hard entries favor alternatives. Appendix A.3 assigns unequal numbers of tasks to the four groups, so an unweighted mean of those four columns is not an appropriate reconstruction of the Average. Unequal group size does not fully reconcile the source: even the stated 7/2/1/1 task weights fail to reproduce several averages in Tables 5–7. Treat the retained average differences as comparisons of printed aggregates, with unknown raw weighting, rather than independently reconstructed effect sizes. e12e13e16e17e18e19e20
What it supports. Raw Distance raises the reported average from 63.4% to 79.4%, and Subplan adds another 2.6 percentage points to reach 82.0%. The evidence supports useful spatial conditioning, while leaving a more modest and task-dependent benefit attributable to converting geometry into structured subgoals.
Where the evidence stops. Subplan's Medium score is 69.3% here, but Tables 5–7 print 63.3% in corresponding original/global rows. The discrepancy is unresolved. These comparisons also do not include a deterministic geometric-plan control or confidence intervals.
Table 6. Execution feedback adds recovery opportunities beyond filtering imagined videos. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the rows as increasingly extensive feedback: NR has no replanning, GR replans during generation, and GR&ER adds execution replanning. Section 3.3 supplies the operational distinction: video rejection triggers regeneration, while execution drift or stagnation causes the spatial plan itself to be updated. Follow the Average column to see the aggregate change, then inspect Very Hard, corresponding to shelf placement in Appendix A.3. Its rise from 78.7% to 92.0% suggests that acting on observed failures matters beyond screening predictions. This is a reader interpretation of the comparison, not an independently isolated measurement of the failure-recovery mechanism. e06e12e19e23
What it supports. Generation-only replanning raises average success from 82.0% to 83.4%; adding execution replanning reaches 86.7%, another 3.3 percentage points. The result supports checking the environment after acting, rather than assuming that an accepted imagined trajectory will remain executable throughout the task.
Where the evidence stops. Boldface is unreliable: NR's 63.3% Medium entry is bold even though GR&ER reaches 73.3%. Extra retries and computation are not controlled here, and the rejection route differs between Figure 10 and Section 3.3. Using A.3's 7/2/1/1 task weights gives about 80.93%, 83.40% and 86.52% for the displayed NR, GR and GR&ER rows, rather than 82.0%, 83.4% and 86.7%. The stated gains are differences between printed averages; raw outcomes and aggregation rules are needed to establish their exact magnitudes.
7. Analysis & limitations
7.1 What the evidence leaves open
Simulator coordinates provide privileged geometry. Dataset size, explicit train/test partitions, human-evaluation sample counts and uncertainty are not adequately specified. Masking evidence is qualitative; Figure 7 also shows substantially weaker assembly performance despite the prose's broad claim of consistently high success. e03e09e12e21
Method descriptions disagree: A.1 says the action coordinate input is solely the end effector, but Algorithm 1 encodes both robot and object. A.2.2 omits flow from its input list. Table 8 describes a nine-dimensional distance vector where Section 3.1 defines a three-dimensional offset. Exact implementation remains unresolved. e03e05e07e08e24
Figure 2's forward/backward verbal labels reverse the z-sign convention in Figure 1 and A.5. Figure 10 routes rejection through spatial replanning, whereas Section 3.3 says video rejection retains the condition. Table 3 reports Subplan Medium 69.3%, versus 63.3% for the corresponding global/no-replanning rows elsewhere; Table 6 boldface also misidentifies the best Medium entry. Several printed averages also remain unreconciled: the eleven SP-R task rates in Table 1 average about 86.53%, versus 86.7%; weighting the four Table 6 groups by the 7/2/1/1 task counts in A.3 gives about 80.93% for NR and 86.52% for GR&ER, versus 82.0% and 86.7%. Table 7 expert-assisted gives about 86.18% under those weights, versus 85.6%. These are conditional reader reconstructions, not corrected results. Raw counts and aggregation rules are unavailable; printed averages and differences derived from them are retained with unresolved effect magnitudes. e06e16e18e19e23e12e13e20
Runtime reporting mixes 15 seconds per task and 27.11 seconds end-to-end; the listed components are 2.43, 9.68 and 0.75 seconds. Appendix A.2 labels RTX 4090 configurations as 80GB and 40GB and describes video training as both FP16/AMP and FP32. These unreconciled descriptions prevent a dependable compute budget. e07e08e22
7.2 Questions for discussion
- How much benefit survives when simulator coordinates are replaced by noisy estimated geometry?
- Does structured VLM planning beat a deterministic plan builder with the same spatial inputs and sampling budget?
8. Reproducibility audit
8.1 Requirements and known gaps
Required assets include expert demonstrations with masks/depth/coordinates, calibrated spatial conventions, both diffusion models and GPT-4o planning/checking prompts. A.4 names sampling thresholds and manual crop windows without values; guidance scales, DDIM inference step counts, exact flow pairing, stagnation thresholds, retry limits and a pinned VLM version remain unspecified. e05e06e07e08e09e10e24
Proposed checks: compare correct, shuffled and deterministic geometric plans under identical data and compute; then compare generation-only and dual-stage feedback under matched execution perturbations. Measure executed success, geometry error, replans and latency. These distinguish semantic planning and corrective feedback from extra conditioning or repeated sampling. e16e19e23
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does structured reasoning add value beyond the same geometry?
Reader-proposed, not run: on shelf-place and assembly, compare raw Distance conditioning, GPT-4o Subplan, a deterministic geometric plan and shuffled Subplan fields. Keep expert trajectories, fine-contact sampling, train/test seeds, action policy, training updates and generation budgets fixed. First verify the coordinate signs against Figure 1/A.5, recording Figure 2's conflicting verbal z labels. Evaluate executed success and gripper–target error, not only video acceptability. A specific planning benefit predicts that correct structured plans beat the deterministic and shuffled controls across held-out offsets; similar performance would weaken attribution to VLM reasoning. e03e09e12e16e23
Check 2: Does execution replanning correct geometry or just buy more attempts?
Reader-proposed, not run: freeze the trained policies and introduce the same small target displacement during shelf-place execution for all conditions. Compare generation-only screening, full execution replanning, and a control allowed the same number of extra video samples without updated spatial plans. Predeclare a stagnation rule, retry cap and whether video rejection preserves the plan, following Section 3.3 while documenting Figure 10's alternative route. Log actual success, recovery time, final spatial error, VLM calls and video generations under equal budgets. Higher success and lower error only with updated plans would support corrective spatial feedback; parity with extra sampling would challenge that explanation. e06e12e19e22e23e24
8.3 Reading coverage
Visual audit: All six text chunks and every page of the supplied 14-page v2 PDF were read. Visually inspected the title/authors/version on p. 1; introduction on pp. 1–2; architecture and equations on pp. 3–4; Tables 1–2 and evaluation protocol on p. 5; Figures 2–5 and physical results on p. 6; Tables 3–6 on p. 7; Table 7 and Figures 6–7 on p. 8; training and hardware descriptions on p. 9; Algorithm 1, Table 8 and task groups on p. 10; Figures 8–9 and data processing on p. 11; prompts and frame matching on p. 12; Figure 10 on p. 13; and references through p. 14. All six final original crops were individually viewed; the physical crop was corrected to retain both complete camera rows. Captions and relevant body text were checked on full pages even when excluded from crops. No external supplements, code or project material were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1. Introduction
- 2. Related Work, including 2.1–2.2
- 3. Spatial Policy, including 3.1–3.3
- 4. Experiments, including 4.1–4.7
- 5. Conclusion
- A.1. Inference Pipeline and Algorithm 1
- A.2. Training Details, including A.2.1–A.2.2
- A.3. Tasks in Meta-World and iThor
- A.4. Dataset Construction
- A.5. VLM Prompt Design
- A.6. Frame Matching Mechanism
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity: PDF p. 1 identifies arXiv:2508.15874v2, dated 18 November 2025. The exact title and all 11 authors match the catalog. This report concerns that supplied revision; the catalog submission date is 21 August 2025. The original revision was not supplied, and changes between revisions were not verified.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by visually inspecting all 14 supplied PDF pages and the six final crops.
- Separate supplemental material availability has not been fully verified. The Supplementary Material embedded in PDF pp. 9–14 was read completely.
- Code, checkpoints, external project pages and cited works were not inspected. No experiments were reproduced. Source inconsistencies remain unresolved.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author/affiliation block and arXiv margin stamp
Exact catalog title; Yijun Liu, Yuwei Liu, Yuan Meng, Jieheng Zhang, Yuwei Zhou, Ye Li, Jiacheng Jiang, Kangye Ji, Shijia Ge, Zhi Wang and Wenwu Zhu. Affiliations: Tsinghua University, Beijing University of Technology and Guangzhou University. Stamp identifies 2508.15874v2, 18 November 2025.
Go to primary source ↓e02PDF pp. 1–2, Abstract and Section 1; p. 2, Section 3 opening
Motivates explicit spatial modeling to connect predicted visual trajectories to actions and organizes SP into video generation, action prediction and spatial feedback.
Go to primary source ↓e03PDF p. 3, Figure 1 and Section 3.1, Spatial Plan Table; Eqs. (1)–(2)
Simulator object-minus-end-effector offset prompts GPT-4o; learned action/direction embeddings and distance MLP form subplan embeddings; CLIP task embedding and subplans condition all video U-Net layers. Figure 1 shows separate video and action diffusion pathways.
Go to primary source ↓e04PDF p. 4, Section 3.1, Conditional Diffusion Formulation, Eqs. (3)–(4)
Initial image plus composite spatial/task condition predicts seven future frames, forming an eight-frame trajectory; Gaussian-noise prediction is trained by MSE in an Imagen/AVDC-style U-Net.
Go to primary source ↓e05PDF p. 4, Section 3.2, Eqs. (5)–(7)
Farnebäck consecutive-frame flow, current/goal images and robot coordinates condition action diffusion; actions include three-dimensional end-effector pose and gripper control. Encoders are CNN for flow, ResNet for images, MLP for coordinates. Main text points frame switching to A.4.
Go to primary source ↓e06PDF p. 4, Section 3.3; PDF p. 10 (supplement p. 2), Algorithm 1
GPT-4o validates videos; rejected videos regenerate under the same condition in Section 3.3. Rule-based execution diagnostics trigger updated geometry and plan/video/action regeneration. Algorithm 1 repeats until VLMCheck succeeds and names IsStuck without specifying thresholds or retry caps.
Go to primary source ↓e07PDF p. 9 (supplement p. 1), A.2.1; PDF p. 10, Table 8
Video settings include 128×128 eight-frame clips, AdamW 10^-4, 100,000 updates unless specified, batches 8/16, 1,000-step linear schedule, FP16/AMP, EMA and conditional dropout; inference uses DDIM/CFG. The same section later claims FP32 training within two days on RTX 4090 (80GB). Table 8 lists a nine-dimensional distance vector and a 10k+2k finetuned variant.
Go to primary source ↓e08PDF p. 9 (supplement p. 1), A.1 and A.2.2
A.1 specifies solely end-effector coordinates for action spatial state. A.2.2 lists six-channel current/goal images, trainable ImageNet ResNet-50, H=4, K=20, AdamW 10^-4, decay 10^-6, batch 512, cosine 100-step diffusion and DDIM/CFG inference. It omits flow from the input description and claims FP32 within three days on RTX 4090 (40GB).
Go to primary source ↓e09PDF pp. 11–12 (supplement pp. 3–4), A.4 Dataset Construction
Expert Meta-World RGB/depth/mask trajectories yield point-cloud-derived gripper/target positions. Fine-contact intervals receive 5× temporal sampling; manual task/view-specific crops are used. Sampling parameter names are supplied without numerical values; dataset size and explicit partitions are absent.
Go to primary source ↓e10PDF p. 12, A.5; PDF p. 13 (supplement p. 5), Figure 10
Structured VLM prompts request Accept/Reject video decisions and short plans with action, direction and distance. Figure 10 limits plans to five steps and gives x right negative/left positive, y up positive/down negative, z forward negative/backward positive. Examples permit zero distance for in-place actions.
Go to primary source ↓e11PDF p. 12 (supplement p. 4), A.6, Eqs. (8)–(9)
Frame matching weights geometry, position, SSIM and flow similarities by 0.35, 0.35, 0.2, 0.1; advance for similarity ≥0.8 and enforce a hard switch every 28 steps.
Go to primary source ↓e12PDF p. 5, Section 4.1 and Table 1 caption; PDF p. 10, A.3
Section 4.1 gives 25 seeds across 11 Meta-World tasks and three views, totaling over 800 rollouts; Table 1 instead states 800 episodes per task. iTHOR has 12 tasks in four room types with 20 seeds per task. A.3 groups Meta-World into seven Easy, two Medium, one Hard (Assembly), one Very Hard (Shelf Place) tasks and describes iTHOR object navigation.
Go to primary source ↓e13PDF p. 5, Table 1, all task columns and Average
Average success: AVDC 19.6%, AVDC-R 43.1%, VA 22.3%, VA-OR 53.7%, SP 82%, SP-R 86.7%. R denotes replanning, OR online reinforcement learning plus replanning. SP-R assembly is 32.0%; shelf-place 92.0%; basketball 73.3% versus SP 77.3%.
Go to primary source ↓e14PDF p. 5, Table 2; PDF p. 6, Section 4.2 iTHOR
SP success is Kitchen 88.3%, Living Room 50.0%, Bedroom 63.3%, Bathroom 36.7%, Average 59.6%. VideoAgent averages 34.2% and reaches 40.0% in Bathroom; AVDC averages 31.3%.
Go to primary source ↓e15PDF p. 6, Figure 5 and Section 4.3
SO-100 pick-doll task moves a yellow rabbit toy to a box; 20 trials yield 30% success. DepthAnything estimates depth; top-down current/goal and wrist-camera frames support action prediction. Figure 5 shows High and Wrist execution views. No physical baseline is reported.
Go to primary source ↓e16PDF pp. 6–7, Section 4.4 video-generation ablation; PDF p. 7, Table 3
Video-conditioning averages: AVDC-style text/frame 63.4%, Distance 79.4%, Subplan 82.0%. Subplan Medium is 69.3%; Hard scores are 26.7%, 25.3%, 21.3%, respectively; Very Hard is 48.0%, 81.3%, 78.7%.
Go to primary source ↓e17PDF p. 7, Table 4 and action-prediction ablation
Heuristic AVDC flow-to-target controller with gripper rules averages 19.6%, versus 63.4% for the learned flow-conditioned diffusion policy; the latter row matches Table 3's video baseline.
Go to primary source ↓e18PDF p. 7, Table 5 and spatial-condition injection ablation
Global conditioning averages 82.0%, local GLIGEN-style intermediate cross-attention 84.3%. Global Medium is printed as 63.3%, unlike Table 3 Subplan's 69.3%; Local Medium is 62.0%.
Go to primary source ↓e19PDF p. 7, Table 6 and replanning ablation
NR, GR, GR&ER averages are 82.0%, 83.4%, 86.7%; Very Hard scores 78.7%, 80.0%, 92.0%. NR Medium 63.3% is incorrectly bold despite GR 70.7% and GR&ER 73.3%. Caption defines bold as best.
Go to primary source ↓e20PDF pp. 7–8, expert-assisted recovery ablation; PDF p. 8, Table 7
Additional expert-corrected failed trajectories raise average success from 82.0% to 85.6%; Hard rises from 21.3% to 38.7%. Original and expert-assisted configurations are compared separately from replanning and local-conditioning ablations.
Go to primary source ↓e21PDF p. 6, human-evaluation paragraph; PDF p. 8, Sections 4.5–4.7 and Figures 6–7
Authors report 82% plan accuracy, 67% replanning-decision accuracy, 65% correct videos and 85.5% correct action predictions without evaluation sample counts. Figure 6 offers a qualitative masked-input example; Figure 7 shows six noise seeds with visibly weaker Assembly bars. Goal-window and noise-versus-action regression findings are described without a complete numerical table.
Go to primary source ↓e22PDF p. 5, Section 4.2, final paragraph
Runtime text reports VLM 2.43 s, video generation 9.68 s, action execution 0.75 s, average 15 s per task, and end-to-end 27.11 s versus VideoAgent 50 s. The timing boundaries are not reconciled.
Go to primary source ↓e23PDF p. 3, Figure 1; p. 6, Figure 2 embedded plans; p. 12, A.5; p. 13, Figure 10; p. 4, Section 3.3
Figure 2 describes [1,-1,-1] as left-down-backward and [-1,1,1] as right-up-forward, reversing the z labels in Figure 1/A.5. Figure 10 routes Reject toward renewed spatial-information processing and a plan table, while Section 3.3 specifies same-condition regeneration for rejected videos.
Go to primary source ↓e24PDF p. 10 (supplement p. 2), Algorithm 1; p. 9, A.1
Algorithm 1 calls EncodeCoord with both end-effector and object coordinates, unlike A.1's sole-end-effector description. Its compact pseudocode does not spell out the frame-matching loop, flow encoding or bounded retries.
Go to primary source ↓8.5 Primary sources
Spatial Policy: Guiding Visuomotor Robotic Manipulation with Spatial-Aware Modeling and Reasoning ↗
PDF · 8,747 extracted words
Source fingerprint
82fae8a67fd77573701e7a83c4252318399d3e66c60dc7fa73c9c678f41369b6