Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Models

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Yijie Zhu; Zitong Yu; Wei Li; Hui Ma; Wen Li; Rui Shao; Liqiang Nie

Affiliations: Harbin Institute of Technology (Shenzhen), China; Great Bay University, Dongguan, China; University of Electronic Science and Technology of China, Chengdu, China

Source: 2609.06578 ↗ · Catalog record

Reading: 24 / 558 · 6 original figures & tables · ~19 min ·

1. Paper overview

In one sentence: ProWAM uses recurrent execution progress to select useful imagined-future tokens for action generation, improving manipulation while relying on demonstration-derived progress and contact priors. motivation progress-encoder attention-modulation real-results components

At a glance	What to know
Research problem	Source description Imagined futures have different utility during approach, contact and placement. ProWAM asks how action generation should adapt their influence as execution advances. Its motivating diagnostic compares failure-stage distributions and horizon-group perturbations; the former alone cannot establish absolute stage failure risk. motivation joint-model
Core mechanism	Source description SS-DTPE learns latent progress slots using short-term feedback, Mamba memory, instruction-prefix alignment and temporal structure. progress-encoder progress-objectives
A key reported result	Real-world long-horizon manipulation: Galaxea R1 Lite: 72; AgileX Cobot Magic: 71. Average full-task success rate (%). Eight tasks per platform, 150–200 demonstrations per task and 50 evaluation rollouts; average final cumulative stage completion. Author-reproduced Fast-WAM: 56 and 57. Executed robot-task gains of 16 and 14 percentage points; these are not generated-video quality scores. real-results real-protocol metrics
Reading caution	Source description The authors identify structured tabletop tasks and limited cross-embodiment evaluation as boundaries; each robot retains its own training and action configuration. limitations

Core contributions

Source description
SS-DTPE learns latent progress slots using short-term feedback, Mamba memory, instruction-prefix alignment and temporal structure. progress-encoder progress-objectives
Source description
HPIM converts progress into a global gate and token-specific relevance inside action-to-future attention. attention-modulation
Author claim
The authors report gains across five simulation benchmarks and two real robots; ablations support both progress representation and modulation. libero robotwin vlabench roboeval mikasa real-results components

Two grouped bar charts show failure-stage distributions for action-only versus joint WAM policies and completion drops after perturbing near-, mid- and long-horizon future latents. — **Figure 2.** Future utility depends on both execution stage and prediction horizon. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

2. Motivation

2.1 The problem and the proposed response

Source description

Imagined futures have different utility during approach, contact and placement. ProWAM asks how action generation should adapt their influence as execution advances. Its motivating diagnostic compares failure-stage distributions and horizon-group perturbations; the former alone cannot establish absolute stage failure risk. motivation joint-model

2.2 What this reading follows

A robot approaching an object and a robot already gripping it need different information. ProWAM turns that observation into a control architecture: first estimate execution progress from the previous action, current image and accumulated memory, then use progress to shape attention to predicted visual futures. The paper combines simulation benchmarks, physical robot rollouts and component ablations. Read its evidence in that order: the initial diagnostics motivate nonuniform future use, the architecture specifies how information flows, and the ablations test which additions matter. One distinction is essential throughout: the global gate interpolates between uniform and selective future weighting; it is not a switch that removes imagination. motivation progress-encoder attention-modulation real-results components

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimension	Recorded classification
Major category	Not assigned
Architecture	Not assigned
Prediction paradigm	Not assigned
Quadrant	Not assigned

This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.

3.1 Evidence-based assessment

Insufficient evidence to decide

Reader analysis

The catalog is unassigned, so there is no substantive recorded quadrant to confirm. Architecture evidence supports joint future/action prediction through shared MoT attention, with modality-specific experts and a separately trained frozen progress encoder. The core is a joint generator, not an inverse-dynamics decoder; calling the whole pipeline One Model requires a definition accommodating the auxiliary encoder. joint-model attention-modulation training-inference

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

Inputs	Outputs
Current visual observation and language instruction Previously executed action and recurrent progress state	A low-level action chunk for receding-horizon execution Jointly modeled future visual latents and updated progress tokens

4.2 Equations and their role

Z_t^{\mathrm{prog}}=\mathrm{SS\text{-}DTPE}(O_t,a_{t-1},L,Z_{t-1}^{\mathrm{prog}})

Equation (6): progress tokens use observation O_t, preceding executed action a_{t-1}, instruction L and previous progress tokens. joint-model

\omega_{t,f}^{l}=F\,\operatorname{Softmax}_{f}(\rho_{t,f}^{l}),\qquad \beta_{t,f}^{l}=\eta_l[(1-d_t)+d_t\omega_{t,f}^{l}],\qquad \widetilde C_{a\to v}^{l}(i,f)=C_{a\to v}^{l}(i,f)+\log(\beta_{t,f}^{l}+\epsilon)

Equations (28), (30), (32): F counts future tokens; rho is a relevance logit; omega is mean-one relevance; d is the gate; eta is the layer coefficient; beta modulates attention logits C. Indices t,l,i,f denote time, layer, action token and future token; epsilon stabilizes the logarithm. attention-modulation

\mathcal L_{\mathrm{SS\text{-}DTPE}}=\mathcal L_{\mathrm{sem}}+\lambda_{\mathrm{ord}}\mathcal L_{\mathrm{ord}}+\lambda_{\mathrm{temp}}\mathcal L_{\mathrm{temp}},\qquad \mathcal L_{\mathrm{HPIM}}=\mathcal L_{\mathrm{act}}+\mathcal L_{\mathrm{vid}}+\lambda_{\mathrm{gate}}\mathcal L_{\mathrm{gate}}

Equations (23), (36): semantic alignment, order consistency and temporal contrast organize progress; action/video generation and gate supervision train stage 2. The lambdas weight the named losses. progress-objectives training-inference attention-modulation

5. Method in detail

5.1 Learn progress without assuming an annotated stage label

Source description

Consider the drawer task: the policy must remember whether opening and placement already happened before attempting closure. A current image can be ambiguous, so SS-DTPE combines the previous action's observed outcome with persistent memory. Its short-term encoder models both action-to-observation and observation-to-action interactions; the long-term path maintains progress tokens and selectively reads the resulting evidence. Training supplies structure without manually labeling every stage. Instruction prefixes form a semantic axis, normalized demonstration time centers a soft Gaussian target on that axis, and monotonicity discourages reversing expected prefix completion. Temporal contrastive learning adds local coherence. These are priors learned from successful demonstrations, not ground-truth semantic timestamps. Stage 2 freezes the encoder, so action learning must use the progress representation already learned. progress-encoder progress-objectives training-inference real-protocol

Three connected blocks show the dual-temporal progress encoder, global and local imagination modulation branches, and a joint video/action Mixture-of-Transformers backbone. — **Figure 3.** Recent feedback and recurrent history determine how future tokens influence action generation. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

5.2 Translate progress into attention, keeping the normalization explicit

Reader analysis

The gate reads both progress and its recent change, while the relevance branch asks which future tokens fit that progress state. Multiplying a softmax by the number of future tokens makes local relevance average one. Equation (30) then interpolates between uniform weights and those selective weights, scaled by a depth-dependent coefficient. The log-bias is applied only to future keys reached by action queries, and Equation (34) normalizes against every valid key. Algebraically, a gate of zero leaves the layer coefficient on every future token; a gate of one activates the full relevance pattern. Thus, my reading is that the gate controls selectivity directly. Its effect on total future-attention mass depends on how relevance aligns with the original logits, so more gate activation need not mean more total imagination use. attention-modulation hpim-ablations

5.3 Separate closed-loop success from evidence about the mechanism

Reader analysis

During deployment, each replan updates progress from actual execution feedback and conditions the joint generator before an action chunk is executed. Table 6 therefore evaluates robot behavior, not the plausibility of predicted images. Its final stage counts a rollout only if all preceding stages succeeded; averaging those final-stage rates yields 72% on Galaxea and 71% on AgileX. Table 8 then asks whether progress or modulation accounts for the benefit, while Table 11 separates gating from local relevance. These comparisons support the proposed components under the reported protocols. My interpretation remains narrower than broad transfer: each platform uses its own configuration, seed uncertainty is absent, and the qualitative relevance plots illustrate allocation rather than independently establishing why an action succeeded. training-inference metrics real-results components hpim-ablations qualitative-relevance limitations

5.4 Training and inference

During training

Source description

First train SS-DTPE on successful demonstrations: Gaussian instruction-prefix targets follow normalized time, an order penalty discourages backward semantic progress, and contrastive learning organizes temporal neighbors. Freeze SS-DTPE, then train WAM and HPIM with action/video rectified-flow losses and gate supervision. progress-objectives training-inference joint-model

Source description

The gate uses automatic weak labels. Gripper transitions and task-dependent height/motion cues identify contact windows, expanded by 20 frames on either side. Contact targets are zero, others one; unreliable labels are masked. Local relevance has no token-level labels. weak-labels attention-modulation

Source description

Wan2.2-5B components combine with a 1B action expert, 32-action chunks, nine video frames and eight progress tokens. Stage 1 runs five epochs on four H100 GPUs, batch four per GPU; stage 2 runs ten epochs on eight H100s, batch sixteen per GPU. Both use AdamW, learning rate 0.0001, weight decay 0.01, cosine scheduling and clipping at 1.0. implementation

During inference

Source description

Reset memory at episode start, using a start-action token initially. At each replan, update progress from actual observation/action feedback, condition HPIM, generate and execute an action chunk. Figure 3 includes training future observations; deployment does not receive ground-truth future frames. Exact denoising steps and the executed chunk prefix are unspecified. training-inference joint-model implementation

5.5 Implementation flow

Extract execution feedback
Encode the observation and previous action. Language-conditioned FiLM modulates encoded visual tokens; bidirectional cross-attention links the executed command with its observed outcome. Fuse these streams into current evidence. progress-encoder
Update progress memory
A selective Mamba recurrence filters previous progress before Progress Attention queries current evidence. The result is latent progress slots, not predefined phase labels or an externally supplied completion percentage. progress-encoder joint-model
Estimate future relevance
The gate uses observation, instruction, progress and its change. Future-token queries attend to progress keys/values, undergo self-attention and receive mean-one relevance weights. This query direction agrees with Figure 3 and Equation (26). attention-modulation joint-model
Modulate joint generation
Combine the gate and relevance with a middle-emphasized layer schedule. Add a log-bias only to action-query/future-key logits, normalizing across all valid keys. Video and action experts communicate through shared MoT attention; an action head predicts commands. attention-modulation joint-model

6. Experiments & results

ProWAM learns execution progress from recent action–observation feedback and recurrent memory, then uses that state to redistribute imagined-future attention inside a joint video–action policy. Its strongest evidence combines executed manipulation results with ablations separating progress estimation from future utilization.

6.1 Read the original evidence

LIBERO success-rate table groups general VLAs, predictive VLAs and world action models, with Spatial, Object, Goal, Long and average columns. — **Table 1.** ProWAM leads the reported LIBERO average, with a remaining gap on the Long suite. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

6.2 Results and evaluation conditions

Task & protocol	Reported result	Comparison & interpretation
LIBERO manipulation Four suites, 40 tasks, 50 demonstrations per task and 2,000 evaluation trials.	99.1 average; Spatial/Object/Goal/Long: 99.6/100.0/98.8/97.8. Success rate (%)	Fast-WAM 97.6 average; LingBot-VA 98.5 average. Best reported average, but LingBot-VA scores higher on Long (98.5%). No uncertainty intervals are supplied. libero simulation-protocol
RoboTwin2.0 robustness Clean/randomized evaluation, 100 trials per task; mixture of 2,500 clean and 25,000 randomized demonstrations.	93.9 clean, 92.8 randomized, 93.4 average. Success rate (%)	FlowWAM: 92.9/92.1/92.5. Reported average gain is 0.9 percentage points; embodied pretraining differs between these methods. robotwin simulation-protocol
VLABench progress-aware manipulation Ten evaluated tasks, 50 episodes each.	68.4 / 85.4 / 81.2. SR (%), IS and PS (reported percentage scale)	Author-reproduced Fast-WAM: 58.4/81.2/74.5; pi0.5: 65.4/80.4/77.8. Improves complete success and intermediate progress; Fast-WAM SR gain is 10.0 percentage points. vlabench simulation-protocol metrics
RoboEval structured manipulation Eight task families, 30 variations, 100 episodes per task–variation pair.	0.375 / 0.494. SR and TP (fractions)	Author-reproduced Fast-WAM: 0.313/0.404. Absolute gains are 0.062 SR and 0.090 TP; TP concerns partial advancement. roboeval simulation-protocol metrics
Mikasa-Robo temporal manipulation Five tasks; 250 demonstrations and 100 evaluations per task, 128×128 observations.	50.8. Average success rate (%)	MemoryVLA++: 44.4; InterceptMedium: 68 versus 40. A 6.4-point average gain, with substantial task variation rather than universal dominance. mikasa simulation-protocol
Real-world long-horizon manipulation Eight tasks per platform, 150–200 demonstrations per task and 50 evaluation rollouts; average final cumulative stage completion.	Galaxea R1 Lite: 72; AgileX Cobot Magic: 71. Average full-task success rate (%)	Author-reproduced Fast-WAM: 56 and 57. Executed robot-task gains of 16 and 14 percentage points; these are not generated-video quality scores. real-results real-protocol metrics
Real-world generalization and robustness Four tasks per platform with lighting, scene/object variation; 50 trials per task.	Galaxea: 69; AgileX: 68. Average full-task success rate (%)	Author-reproduced Fast-WAM: 58 and 57. Gains concern specified variations; they do not establish cross-robot transfer. real-results real-protocol limitations
Component and HPIM ablations Galaxea long-horizon tasks; controlled variants in Tables 8 and 11.	Full model: 72. Average full-task success rate (%)	Neither module: 56; SS-DTPE alone: 66; stateless HPIM: 62. Fixing the gate to one: 58; uniform local relevance: 62. Both modules and modulation levels contribute. Single-module variants change the progress interface, limiting a pure factorial interpretation. components hpim-ablations

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

A four-row component ablation crosses SS-DTPE and HPIM presence and reports four LIBERO suites plus real-world long-horizon and generalization success. — **Table 8.** Progress estimation and progress-conditioned modulation contribute complementary gains. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

HPIM ablation table compares removal of the inter-progress gate, removal of intra-progress relevance, the full mechanism and uniform versus middle-emphasized layer schedules. — **Table 11.** Stage-dependent selectivity and token-specific relevance both matter in the reported ablations. Original paper, p. 14 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

LIBERO-Spatial and LIBERO-Goal snapshots show stage-specific future-frame relevance splits and cumulative relevance curves over ranked future tokens. — **Figure 10.** Learned relevance concentrates on different future tokens as execution changes. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.

7. Analysis & limitations

7.1 What the evidence leaves open

Source description

The authors identify structured tabletop tasks and limited cross-embodiment evaluation as boundaries; each robot retains its own training and action configuration. limitations

Reader analysis

Algebraically, mean modulation equals the layer coefficient because relevance averages one. A low gate interpolates toward uniform weighting; it does not turn future attention off or guarantee reduced total future-attention mass. This is narrower than the prose description of overall reliance. attention-modulation

Reader analysis

Tables lack confidence intervals and training-seed dispersion. Time-based prefix targets and monotonicity also leave recovery, repeated subtasks and reordered instructions insufficiently tested. libero real-results gate-supervision progress-objectives

7.2 Questions for discussion

Does learned progress remain useful when a task requires undoing an earlier step or instruction order differs from execution order?
Does the gate improve control by selecting future tokens, changing total attention mass, or both?

8. Reproducibility audit

8.1 Requirements and known gaps

Source description

Reproduction needs pretrained video components, synchronized robot demonstrations with reliable proprioceptive traces, and benchmark success implementations. Appendix metrics defer IS/PS and TP semantics to official benchmarks. implementation weak-labels metrics

Reader analysis

Resolve omitted Gaussian width, temporal sampling, weak-label height/motion and segment/gap thresholds, denoising configuration and execution cadence. No latency or GPU-hour budget is supplied. Proposed comparisons should hold backbone, training budget and rollout protocol fixed. progress-objectives weak-labels implementation training-inference

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Measure what the gate changes before interpreting it as reliance

Reader-proposed check, not performed: on held-out rollouts, record one layer's action-to-key logits, local relevance and valid-key mask. Freeze these quantities and sweep the gate from zero to one under Equations (30)–(34). Compare learned relevance with all-one and randomly permuted relevance controls, recording total future-attention mass and its entropy. All-one relevance should make the sweep inert; nonmonotonic total mass with learned relevance would falsify a simple 'higher gate always means more future attention' interpretation. Follow with matched rollouts to test whether the learned allocation improves stage completion. attention-modulation hpim-ablations metrics

Check 2: Test feedback-based progress when execution time becomes misleading

Reader-proposed check, not performed: train full SS-DTPE and the paper's Time-Only control with the same backbone, data and optimization budget. Evaluate matched drawer-task initial states with nominal execution, a controlled pause, and a safely induced failed placement requiring retry. Keep the weak-label pipeline and its 20-frame windows fixed; report gate trajectories, cumulative stage completion and final success across repeated seeds. If SS-DTPE simply advances with elapsed time during the pause or cannot recover after retry, its interpretation as feedback-grounded progress is weakened despite the original Time-Only ablation. progress-objectives progress-encoder hpim-ablations real-protocol weak-labels metrics

8.3 Reading coverage

Visual audit: Visually inspected the title/byline/affiliations and version on page 1, every original figure (1–13), every table (1–15), and all method, training, protocol, metric, weak-label and limitation pages supporting retained claims. All six final crops were separately viewed. Cross-checked Figure 3 query/key directions and modulation inputs against Equations (12)–(17), (24)–(34), and Figure 10 normalization against Section 5.6. The architecture guide distinguishes schematic FiLM placement from Equation (12) and training future observations from inference. The report preserves the distinction between gate-controlled selectivity and total future-attention mass. Reference-only pages 17–18 were read as text, not visually inspected; no claims depend on their cited works being independently verified. External videos, code and separate supplements remain uninspected.

PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 19, 20, 21, 22, 23. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions

Abstract and Introduction
Section 2: Related Work
Section 3: Problem Formulation and diagnostics
Section 4: Method, all subsections and Equations (6)–(36)
Section 5: Experiments, all subsections and Tables 1–15
Section 6: Conclusion
References
Appendix 1: Detailed Experimental Setup, all subsections
Appendix 2: Qualitative Rollout Visualization
Appendix 3: Limitations and Future Work

Outside the original text pass

Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
Separate supplemental material availability has not been fully verified.
The extraction-only figure gap was addressed by inspecting all 13 original figures and all 15 tables in the PDF. All 11 supplied text chunks were read individually, including references and appendices.
Identity/version scope: the title and all seven authors match the supplied catalog. The inspected artifact is arXiv:2609.06578v1, dated 6 September 2026. No other revision was supplied or compared. The IEEE TPAMI running header does not establish publication acceptance.
Separate code, project-page contents, videos and other supplements were not inspected; no experiment was reproduced.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

identityPDF p. 1, title, byline, author footnote and arXiv marginInspect

The title and seven authors match the catalog. The artifact identifies arXiv:2609.06578v1 [cs.CV], 6 September 2026; three institutions are named.

1. Paper overview

Core contributions

2. Motivation

2.1 The problem and the proposed response

2.2 What this reading follows

3. Research context

3.1 Evidence-based assessment

4. Problem formulation

4.1 Inputs and outputs

4.2 Equations and their role

5. Method in detail

5.1 Learn progress without assuming an annotated stage label

5.2 Translate progress into attention, keeping the normalization explicit

5.3 Separate closed-loop success from evidence about the mechanism

5.4 Training and inference

During training

During inference

5.5 Implementation flow

6. Experiments & results

6.1 Read the original evidence

6.2 Results and evaluation conditions

6.3 Ablations and diagnostic examples

7. Analysis & limitations

7.1 What the evidence leaves open

7.2 Questions for discussion

8. Reproducibility audit

8.1 Requirements and known gaps

8.2 Proposed reproduction checks

Check 1: Measure what the gate changes before interpreting it as reliance

Check 2: Test feedback-based progress when execution time becomes misleading

8.3 Reading coverage

Outside the original text pass

8.4 Traceable evidence

8.5 Primary sources

Learning to Use Imagination: Progress-Conditioned Future Utilization for World Action Models ↗