WALL-WM: Carving World Action Modeling at the Event Joints
1. Paper overview
In one sentence: Event-aligned pretraining links language and visual futures to executable trajectories, improving reported instruction generalization while retaining expensive annotation and multi-module training. event-unitdual-towerstaged-traininggeneralization-resultsdexterity-resultsdiscussion-limits
| At a glance | What to know |
|---|---|
| Research problem | Author claim Fixed-duration chunks can cut across grasp, transfer and release boundaries while their instruction describes the whole task. The authors argue that this mismatch encourages short-horizon correlations and erodes pretrained video semantics. Their remedy makes the executable event itself the training target, giving language, vision and control a common temporal interval. event-unit |
| Core mechanism | Source description A staged event-learning recipe preserves a pretrained video tower while learning multi-view fusion and then trajectory recovery through layer-wise video-to-action attention. dual-towerstaged-training |
| A key reported result | Real-robot Generalization: 53.75 Mean Task Progress, 0–100; higher is better. Four tasks in a shared cluttered tabletop scene, randomized instructions; event-mode policy. DreamZero 28.50; pi0.5 24.00; U-Scratch 18.50; LingBot-VA unevaluated. A 25.25-point advantage over the strongest available external baseline, calculated from reported means. This is partial progress, not 53.75% successful trials; U-Scratch wins lid covering. robot-protocolgeneralization-resultsprogress-rubrics |
| Reading caution | Source description Platform-aligned pretraining advantages and unequal tuning remain despite a shared evaluation protocol. Dense event labels are costly; narrow, low-data tasks may favor fixed horizons. These are acknowledged source limitations, not demonstrated universal advantages for event mode. discussion-limits |
Core contributions
- Source description
A staged event-learning recipe preserves a pretrained video tower while learning multi-view fusion and then trajectory recovery through layer-wise video-to-action attention. dual-towerstaged-training
- Source description
Event captions, balanced vision-language/action sampling and recovery data accompany two deployment interfaces: variable event execution and fixed chunks with a continuous language-conditioning path. caption-samplingdata-mixtureinference-modes
Figure 2. The training unit ends at a behavior boundary; a fixed deployment window needs temporal context. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top band as a sequence of executable events: each local caption refers to the same interval as its video and action trace. The displayed 2.5, 7.5 and 5.0 seconds illustrate unequal event lengths. The middle band instead keeps one global instruction while earlier observations become history for the next chunk. Finally, compare the three lower arrows. The left event caption localizes what should happen; the middle global instruction does not identify the current phase; the right adds history to disambiguate it. This is the motivation for event-centric pretraining and observation-centered adaptation, not a requirement that every deployed action sequence have variable length. event-unittime-alignmentchunk-adaptationinference-modes
What it supports. Events supply a semantic boundary shared by language, visual dynamics and control. Fixed-length inference can still use the resulting pretrained backbone after adding history. The distinction is between how the learning target is localized and how the controller packages its next execution interval.
Where the evidence stops. The displayed durations are examples, not measured speed comparisons. The figure argues that targets are better specified; it does not demonstrate flawless event detection or describe a complete runtime stopping rule.
2. Motivation
2.1 The problem and the proposed response
Fixed-duration chunks can cut across grasp, transfer and release boundaries while their instruction describes the whole task. The authors argue that this mismatch encourages short-horizon correlations and erodes pretrained video semantics. Their remedy makes the executable event itself the training target, giving language, vision and control a common temporal interval. event-unit
2.2 What this reading follows
A robot asked to organize a table must identify its current stage before choosing the next movement. WALL-WM makes that stage an explicit learning unit: an event caption, a visual transition and an action trajectory cover the same interval. Its video tower first learns embodied futures; an action transformer then reads those features to produce control. This reading follows that information flow, explains how event execution differs from fixed chunks and Staircase reasoning, and checks the evidence against original tables. The strongest results concern partial-credit progress under changing instructions; precision contact, unequal platform familiarity and incompletely specified reproduction details limit the broader claim. event-unitdual-towerstaged-traininggeneralization-resultsdexterity-resultsdiscussion-limits
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Supports the recorded classification
The separate video/action DiTs and one-way visual-feature-to-action interface support Dual-system and an IDM-style reading. Here IDM means trajectory recovery from predicted-video features, not a detached network consuming fully decoded future images. The source calls the overall target coupled video-action denoising; its continuous language path does not make the distinct towers one shared transformer. Preserve this qualification alongside the recorded Q4 label. dual-toweraction-objectivestaged-trainingappendix-design
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 1. Localize the target before learning the motion
Consider the gap between organizing scattered objects and carrying one identified object to its destination. The global instruction can remain unchanged while the required motion switches from approaching to grasping, transporting and releasing. WALL-WM segments synchronized video-action streams at behavior boundaries before adding the caption hierarchy. Its sampler can then expose corrective movements that an episode-level label would hide. Video pretraining receives a caption and visual target for the same event; action pretraining uses the aligned trajectory. Reader interpretation: the data representation is doing part of the disambiguation work usually left to a history-conditioned policy. This explains why Figure 2 and the caption/sampling pipeline are central to the mechanism. It does not imply boundaries are learned without annotation: the source explicitly identifies dense temporal grounding as a remaining dependency. event-unitsynchronizationcaption-samplingstaged-trainingdiscussion-limits
Figure 3. Separate towers connect through visual features at every depth, with execution feeding the next observation. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow panel (a) from its two language sources into the shared conditioning interface. The T5 label depicts the inherited text path; the prose also aligns VLM features to that space. Panel (b) contains distinct video and action denoisers, plus the Execute/Rollout feedback loop. Panel (c) gives the operational order: S1 keeps within-view computation, S2 exchanges camera information, S3 gathers views along the token axis, and S4 supplies visual keys and values to action blocks. Equation (9) confirms that the action stream reads video features in one direction. Proprioception also has its own action-side attention path, so it is not recovered solely from generated pictures. dual-towerlanguage-bridgestaged-trainingaction-objectiveinference-modes
What it supports. The world-model representation is used while generating control. It is not just a training-time reconstruction target: action denoising consumes per-layer visual features. The diagram also supplies architectural grounds for a dual-system interpretation, despite the paper treating the overall stack as one WALL-WM model.
Where the evidence stops. The figure's joint-denoising description does not imply simultaneous main-stage training or symmetric coupling. Table 1 freezes video during action pretraining; the default inference schedule reuses a video anchor rather than fully refreshing both towers at each action step.
5.2 2. Preserve visual structure while teaching its use for control
The action learner receives more than a completed picture. At each depth it cross-attends to intermediate multi-view features from the matching video block, together with an independently accessible proprioceptive state. Cross-view attention first establishes communication among camera streams; ViewConcat then assembles the evidence for the action queries. The main recipe trains this visual machinery before freezing it for action learning. Its asymmetric schedule fixes a visual anchor while action noise varies, so the action transformer learns to use a consistent visual representation throughout denoising. Six action-noise draws reuse that forward pass during training only. Reader interpretation: this separation limits action gradients from rewriting the video prior, while still learning a control-specific decoder. The physical robot subsequently executes the predicted trajectory; a plausible visual future alone does not count as successful action. dual-towerview-maskstime-alignmentaction-objectivestaged-traininginference-modes
Figure 6(c). Staircase panel (c) shows shared grounding feeding reasoning at staggered depths. Original paper, p. 13 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read this isolated panel (c) from the bottom input through Layer 1. Dotted rightward arrows relay intermediate states at increasing depths; later positions continue through upper blocks instead of each restarting the lower computation. Follow the paths upward to Layer L and the shared projector. Section 3.4 formalizes shared grounding below a relay depth and reasoning above it, preserving causal cache updates. Training projects the reasoning states into prefixes for a frozen text reconstructor; unified rollout consumes continuous context directly. The full source figure prints both Traditional CoT and Latent CoT as 'a)', although its caption identifies Latent CoT as '(b)'. This crop contains only the correctly labeled Staircase panel. The omitted comparisons depict serial schedules without measured latency. staircasestaircase-panel-labelslanguage-bridgeinference-modesrobot-protocolreasoning-results
What it supports. The diagram makes computational reuse explicit: shared grounding supplies several latent reasoning positions. WALL-WM intends this schedule to reduce repeated language-model work and provide differentiable context to unified-mode action chunks. Its reconstruction objective supervises the meaning of those latents while the main language backbone stays frozen.
Where the evidence stops. No isolated latency or control-quality comparison establishes the size of Staircase's benefit here. The main robot scores use event-mode next-event descriptions, so they should not be presented as a direct ablation of this unified-mode decoder.
5.3 3. Separate the deployed interface from the claim being tested
Event mode receives a new event description and advances observation after executing its segment. Unified mode instead uses a fixed action horizon and history, with continuous global context, atomic instructions or Staircase latents as interchangeable text-side inputs. Staircase's text reconstructor supervises training; it is not a requirement to emit that text during unified rollout. The main physical evaluation uses explicit event descriptions, while U-Scratch removes both event pretraining and the event reasoning path. Table 4 then supplies a different pretrained baseline that removes view interaction and changes execution format together. Read those comparisons as tests of complete configurations. A further boundary comes from the score itself: Task Progress credits intermediate actions. Higher progress supports better advancement through the prescribed rubric, but without binary completion counts it cannot establish how often the entire task succeeds. inference-modesstaircaserobot-protocolreasoning-resultscombined-ablationprogress-rubrics
5.4 Training and inference
During training
First train video with flow matching, frozen T5/VAE, caption dropout, EMA and shape bucketing. Then freeze video and train the randomly initialized action tower. Anchor video at step 45 of a 50-step schedule; reuse its features for six independently noised action draws per optimizer step. Main runs use velocity prediction, without x-prediction or DCT loss. staged-trainingaction-objective
Train only VLM projection/auxiliary heads for T5 alignment, next-event text and remaining time. Separately train Staircase branches and their prefix projector by reconstructing text through frozen Qwen3.5-0.8B. Optional next-chunk adaptation updates both DiTs with frozen T5 and reclustered history windows; a differentiable interface does not imply every module was jointly optimized. language-bridgestaircasechunk-adaptation
DMuon, fused kernels and isolated event packing support scale. Distribution-matching distillation retains action supervision; FP8 reduces per-step cost. The reported 10 Hz lacks a hardware-specific protocol, and the reported 53% MAE degradation without action supervision lacks absolute errors, limiting independent comparison. systems
During inference
A human or VLM provides the next event; the denoiser predicts its trajectory, the robot executes, and a new observation starts the next event. The formal target includes future video, but the default control path reuses anchored intermediate video features across action steps. This is inference-time visual conditioning, not merely an auxiliary video loss. dual-towerinference-modes
Unified mode slides fixed action horizons over observation/history windows after adaptation. Its text context can be a reused continuous instruction, a fresh atomic instruction, or Staircase latents. The paper does not give a complete event-stop controller or the mapping from remaining-time predictions to execution duration. inference-modeslanguage-bridge
5.5 Implementation flow
- Construct executable event targets
Synchronize visual motion with end-effector logs before segmentation. Assign nested Task/Subtask/Action/Segment captions, preserve short corrections, and balance offline scene-language and trajectory clusters. Internet/egocentric video, UMI-style recordings and robot data have different action availability; they are not all robot demonstrations. synchronizationcaption-samplingdata-mixture
- Adapt the video prior across cameras
Keep Wan within-view computation, add frame-wise cross-view attention through a zero-initialized projector, and identify cameras with Camera RoPE. Training calibration restricts attention to compatible sight cones; these restrictions disappear at inference. Tube masking is an optional correspondence exercise, disabled in the main recipe. dual-towerview-masks
- Read visual futures into actions
Each action block reads the matched video block after concatenating views, while a separate state-attention path preserves absolute proprioception. Information flows from video to action only. Shared temporal embeddings align action groups with video frames; unified windows add history and chunk identity, with actions expressed relative to the current pose. dual-towertime-alignment
- Ground task progress in language
Qwen3.5-9B features are projected toward the existing T5 conditioning space. Event mode uses explicit next-event descriptions. Staircase decoding supplies continuous reasoning states for unified chunks by sharing lower-layer grounding and relaying information through staggered upper-layer computation. language-bridgestaircaseinference-modes
6. Experiments & results
WALL-WM trains on semantic events that align language, future video and robot trajectories. A Wan-derived multi-view video tower supplies intermediate features to a separate action denoiser; event-language execution and history-conditioned fixed chunks share this backbone. Its clearest reported advantage is Task Progress under cluttered, changing instructions, with much smaller gains on precision insertion. The evidence supports this combined system, with substantial data and evaluation qualifications.
6.1 Read the original evidence
Table 2. Embodied adaptation improves interaction-oriented video scores while giving up some visual-quality score. Original paper, p. 25 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare rows within a column, not numbers across different metrics. The first two columns concern image and aesthetic quality; the remaining groups address motion, semantic consistency and physical plausibility. For a useful baseline, follow Wan2.2-5B across to WALL-WM, which normally inherits that video backbone. Interaction Quality, Instruction Following and Trajectory Accuracy improve, while Image Quality and Aesthetic Quality decline. Section 7.1 describes a benchmark containing 200 held-out ID and 50 OOD tasks, but this table provides a single score per model and metric. The comparison therefore shows aggregate embodied adaptation, without revealing an OOD-specific effect or controlling for equal embodied training. video-protocolvideo-table
What it supports. Interaction quality rises from 0.226 to 0.434 and motion smoothness from 0.683 to 0.771 relative to Wan2.2-5B. Image quality instead falls from 0.527 to 0.503. The strongest supported conclusion is improved embodied-video behavior across these reported metrics, not uniformly better image generation.
Where the evidence stops. The Wan baselines lack WALL-WM's event-centric embodied training. These dimensionless video scores neither isolate event segmentation from data adaptation nor measure physical task completion. Separate ID/OOD scores and uncertainty are absent.
Table 5. Event mode leads each suite average, with uneven task-level gains and a small dexterity margin. Original paper, p. 36 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with each bold Average row, then inspect individual tasks that qualify the average. WALL-WM-E is event mode; WALL-WM-U-Scratch is a fixed-length model without event pretraining and reasoning, not the adapted unified mode of Section 5.5. The retained table note defines the dashes as unevaluated entries. All scores are Task Progress: Section 7.2 and Table 6 assign partial credit to intermediate steps and normalize to 0–100. Compare the large generalization difference with the near tie on dexterity. Then examine exceptions such as lid covering, relational pairing and basket picking. Those rows prevent the suite averages from being mistaken for universal task dominance. robot-protocolgeneralization-resultsdiverse-resultsreasoning-resultsdexterity-resultsprogress-rubricsdiscussion-limits
What it supports. Generalization reaches 53.75 against DreamZero's 28.50 and U-Scratch's 18.50. Dexterity reaches 32.00 against U-Scratch's 31.25, a much smaller margin. Thus the system's advantage is more visible in instruction-conditioned, cluttered-scene behavior than in the two scored precision-contact tasks. This conclusion is specific to the reported task sets.
Where the evidence stops. These are partial-progress scores, not success percentages. Trial counts and confidence intervals are missing. The authors acknowledge platform-aligned data advantages and unequal tuning; LingBot-VA's absent generalization scores must remain unavailable. The stationery rubric totals nine points despite the stated ten-point scale; its normalization remains unclear.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Real-robot Generalization Four tasks in a shared cluttered tabletop scene, randomized instructions; event-mode policy. | 53.75 Mean Task Progress, 0–100; higher is better | DreamZero 28.50; pi0.5 24.00; U-Scratch 18.50; LingBot-VA unevaluated. A 25.25-point advantage over the strongest available external baseline, calculated from reported means. This is partial progress, not 53.75% successful trials; U-Scratch wins lid covering. robot-protocolgeneralization-resultsprogress-rubrics |
| Real-robot Diverse Manipulation Seven direct tabletop tasks under the common physical protocol. | 75.86 Mean Task Progress, 0–100 | U-Scratch 63.00; pi0.5 55.64; DreamZero 39.97; LingBot-VA 29.71. Strongest suite average, with task-level exceptions for basket picking and glasses placement. robot-protocoldiverse-resultsprogress-rubrics |
| Real-robot Reasoning Manipulation Five classification, ordering, matching and selection tasks; explicit VLM event descriptions. | 71.60 Mean Task Progress, 0–100 | U-Scratch 59.50; pi0.5 56.40; DreamZero 32.70; LingBot-VA 31.60. The gain combines pretraining and event reasoning; relational pairing remains weaker than pi0.5. It does not isolate Staircase decoding. robot-protocolreasoning-resultsprogress-rubrics |
| Real-robot Dexterous Manipulation Two scored tasks: Insert Wireline and Put Stationery in Case. | 32.00 Mean Task Progress, 0–100 | U-Scratch 31.25; DreamZero 25.00; LingBot-VA 24.00; pi0.5 15.00. Only 0.75 points over U-Scratch; precision contact remains difficult. Illustrated fan unboxing is not included in this average. robot-protocoldexterity-resultsprogress-rubrics |
| Embodied future-video generation Internal held-out benchmark: 200 ID and 50 OOD tasks; WorldArena protocol, without separate split scores. | 0.434 / 0.391 / 0.234 Interaction quality / instruction following / trajectory accuracy | Wan2.2-5B: 0.226 / 0.298 / 0.223. Embodied-video improvements coexist with lower image/aesthetic scores, 0.503/0.393 versus 0.527/0.409. Baselines lack the same embodied adaptation; video metrics do not establish executed control. video-protocolvideo-table |
| CO3Dv2 spatial representation probing Representation evaluation following the cited VidFM-3D protocol. | 0.271 / 0.132; 0.210 / 0.727 Point/depth error (lower); AUC@5/AUC@30 (higher) | WAN2.1-14B: 0.284 / 0.151; 0.200 / 0.736. Competitive spatial features, with worse AUC@30. Probe scores are distinct from manipulation performance. spatial-probes |
| Combined event-execution and view-attention ablation Pretrained event mode versus pretrained fixed-length Base without VI-SA; not U-Scratch. | 71.6 / 53.75 Mean Task Progress: reasoning / generalization | Base: 32.6 / 22.0. The 39.0/31.75-point differences combine two architectural/execution changes and cannot identify their separate causal effects. combined-ablationprogress-rubrics |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. The ablation measures the combined value of view interaction and event execution. Original paper, p. 30 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The Base and Event columns form paired comparisons within each task. Unlike U-Scratch in Table 5, Base already starts from a pretrained backbone. Its two changes are removal of View-Interaction Self-Attention and use of instruction-conditioned fixed-length decoding. Event restores the view module and event-mode execution together. Read the final row for suite means, then the ordered-button task for a concrete transition from zero progress to substantial progress. Section 7.2.5 explicitly says that the comparison measures both changes jointly. It is therefore evidence against this particular reduced pretrained configuration, not an estimate of the standalone causal effect of either event boundaries or cross-view attention. combined-ablationrobot-protocolprogress-rubrics
What it supports. Reasoning improves from 32.6 to 71.6 and generalization from 22.0 to 53.75. Press Button in Order rises from 0 to 64. Pretraining alone is insufficient for the reduced Base configuration to match the full event-mode system under the reported evaluation.
Where the evidence stops. Two factors change at once, and event mode also changes the conditioning/execution interface. A factorial comparison with matched training and reasoning resources is needed before assigning these gains to one component.
7. Analysis & limitations
7.1 What the evidence leaves open
Platform-aligned pretraining advantages and unequal tuning remain despite a shared evaluation protocol. Dense event labels are costly; narrow, low-data tasks may favor fixed horizons. These are acknowledged source limitations, not demonstrated universal advantages for event mode. discussion-limits
No robot trial counts, confidence intervals or seed-level variation are reported. Video scores merge the stated ID/OOD benchmark rather than exposing OOD-only performance. Consequently, neither statistical significance nor broad transfer beyond the evaluated platform can be established here. robot-protocolvideo-protocoldiscussion-limits
Appendix diagnostics and generated/executed frame galleries motivate visual priors but do not constitute a controlled capacity-sharing sweep or aggregate execution validation. The text-to-video versus latent-prior discussion is a design argument, not an experimentally isolated comparison. appendix-designqualitative-appendix
7.2 Questions for discussion
- Would event execution still help with view attention, pretraining, captions and replanning budget held constant?
- Do improvements in partial-credit Task Progress predict full completion on unfamiliar robot geometry?
8. Reproducibility audit
8.1 Requirements and known gaps
A reproduction needs the exact Wan/Qwen checkpoints, action interface, camera calibration for training masks, synchronized event data and caption/cluster assignments. Missing details include total embodied-data size, mixture weights, clustering configuration, training budgets, optimizer schedules, loss weights, hardware counts/models and software versions; named techniques alone do not specify the production run. data-mixturecaption-samplingview-masksstaged-traininglanguage-bridgesystems
Resolve frame conventions first: Section 3.3 gives a leading-one/trailing-four VAE rule, but Section 5.1 calls 65 latent frames equivalent to 129 raw frames under stride-2 sampling. Neither convention should be silently corrected. Event termination, anchor jitter and distilled step count also require explicit implementation choices. Table 6 also gives stationery-task component weights summing to nine despite its stated ten-point scale; the normalization cannot be inferred safely. frame-count-gapstaged-traininginference-modessystemsprogress-rubrics
Reader-proposed checks: hold pretraining constant in a factorial VI-SA/execution-mode comparison; separately sweep the video anchor with fixed action schedules and matched physical resets. Score both component-level progress and binary completion, publish trial counts and uncertainty, and distinguish reduced compute from improved control. combined-ablationaction-objectiveprogress-rubrics
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate view interaction from event execution
Reader-proposed experiment, not performed: train four matched configurations crossing VI-SA present/absent with event/fixed-length execution, all starting from the same pretrained initialization and using the same data, update budget, camera inputs and action interface. Record how the two execution modes receive language, and control the upstream reasoner's resources rather than silently granting one richer guidance. Use the reasoning and generalization tasks from Table 4, paired physical resets, a prespecified trial count, randomized policy order and blinded rubric scoring. Report Task Progress, full completion and uncertainty for every cell. Event execution helping only when VI-SA is present would support an interaction; an effect of VI-SA under both modes would support an independent view contribution. Failure to recover the published corner-to-corner difference would challenge reproduction of the combined result. combined-ablationrobot-protocollanguage-bridgeprogress-rubrics
Check 2: Test whether the selected visual anchor guides executable motion
Reader-proposed experiment, not performed: first resolve the raw-frame/VAE convention, then keep the video checkpoint, action architecture, event data, text conditions and action schedule fixed while training matched action heads against the published step-45 anchor and several prespecified earlier/later anchors. Add a matched-observation control with video features exchanged between different events, preserving tensor shapes. Keep the number of noise draws and optimizer updates equal. Evaluate held-out trajectory error and paired robot trials for insertion and screwdriver placement, reporting component scores and full completion. Log video-forward count and latency separately. A reproducible control optimum near the selected anchor, lost under event-mismatched features, would support informative future-feature conditioning; unchanged physical performance under the mismatch would weaken that account even if offline action loss improved. frame-count-gapaction-objectivestaged-trainingdual-towerdexterity-resultsgeneralization-resultsprogress-rubrics
8.3 Reading coverage
Visual audit: Original PDF pages 1–40 were rendered and visually inspected, including the title and contributor pages, all 24 figures, all six tables, method equations, training-stage matrix, systems discussion and appendix rubrics. All 15 text chunks, including references on pages 41–45, were read individually. Every one of the six final crops was separately viewed. Figure 3 coupling direction was checked against Eq. (9); Figure 6 relay arrows against Section 3.4; Table 4 definitions against Section 7.2.5. The final Figure 6 crop contains only the correctly labeled panel (c); its guide discloses the duplicated a) label on the full source figure's other panels. No source labels or pixels were altered. The figures are schematic where noted: joint-denoising language does not override staged freezing or anchored inference. The frame-count inconsistency on pages 10 and 20 remains explicit. Exact robot values come from Table 5 and the accompanying results prose, not estimates from overview bars. The Table 5 crop retains its model-variant and unavailable-entry note. Table 6's stationery-rubric total remains unresolved rather than silently renormalized. No external videos, repositories, datasets or separate supplements were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title and abstract; Sections 1–2: introduction and related work (pp. 1–5)
- Sections 3.1–3.4: architecture, geometry, action dynamics and reasoning (pp. 6–13)
- Sections 4.1–4.5: data composition, synchronization, captions, sampling and recovery (pp. 13–19)
- Sections 5.1–5.5: training stages and both inference modes (pp. 19–22)
- Sections 6–8: infrastructure, complete experiments and discussion (pp. 22–32)
- Contributors (p. 32); Appendix 9.1–9.5, all tables and galleries (pp. 33–40)
- References (pp. 41–45); all 15 supplied text chunks read individually
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation above was addressed by inspecting original PDF pages 1–40 and every final crop. References on pages 41–45 were read as text.
- Only the supplied arXiv:2606.01955v1 artifact dated 1 June 2026 was reviewed; no later revision or separate supplement was supplied or compared.
- The title-page byline is X Square Robot Team; page 32 separately lists individual contributors and role markers. Metadata follows the collective byline. No individual institutional-affiliation mapping is supplied.
- The title-page code URL ends in wall-x, whereas the catalog code URL ends in wall-wm. Neither repository was inspected; availability and equivalence remain unverified.
- No code, external datasets or linked references were independently inspected, and no training, robot experiment or proposed reproduction check was run.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block and arXiv margin
The exact title is WALL-WM: Carving World Action Modeling at the Event Joints; the byline is X Square Robot Team. The margin identifies arXiv:2606.01955v1, 1 June 2026. The printed code link ends in X-Square-Robot/wall-x.
Go to primary source ↓contributorsPDF p. 32, Contributors
The page identifies WALL-WM as a collaborative effort of the X Square Robot team and lists individual contributors. Shalfun Li bears the project-lead marker and Hao Wang the corresponding-author marker; J.W. Gao is printed without spaces between initials. No individual institutional-affiliation mapping is supplied.
Go to primary source ↓event-unitPDF pp. 3–4, Section 1 and Figure 2
Action-grounded semantic events align captions, video and executable behavior. Figure 2 contrasts event-localized targets with ambiguous global-instruction chunks and history-conditioned chunks; its durations are illustrative.
Go to primary source ↓dual-towerPDF pp. 6–7, Section 3.1 and Figure 3; p. 9, Section 3.3, Eq. (9)
A Wan-derived video DiT and an independently initialized, equal-depth action DiT model event futures. Each action layer reads the corresponding video layer after ViewConcat; video receives no reverse action branch. State has dedicated cross-attention.
Go to primary source ↓view-masksPDF pp. 6–9, Section 3.2, Eqs. (1)–(8), Figure 4
Cross-view attention is added through a zero-initialized projector; Camera RoPE identifies views. Calibration defines training-only sight-cone masks. Optional tube masking removes a co-visible spatial region across time. Border exclusion and sight-cone supervision remain in the main recipe, but tube sampling/up-weighting are disabled.
Go to primary source ↓time-alignmentPDF pp. 9–10, Section 3.3, Eqs. (9)–(11), Figure 5
Shared within-window frame embeddings align video features and action groups. Unified windows additionally use an absolute chunk index, history and observation-relative pose targets, with absolute proprioception kept separately. The codec maps 1+4M+4N input frames to 1+M+N latents.
Go to primary source ↓action-objectivePDF pp. 10–11, Section 3.3, Eqs. (12)–(14); p. 20, Action Pretraining
The main action loss is flow-matching velocity MSE against noise minus clean actions. A frozen video anchor supplies all action denoising steps. Training uses six action-noise draws per video forward; x-prediction and the DCT auxiliary are disabled in main runs.
Go to primary source ↓language-bridgePDF pp. 11–12, Section 3.4, Eqs. (15)–(19); pp. 20–21, Section 5.2, Eq. (21)
Qwen3.5-9B hidden states are projected to the DiT text-conditioning space. Project-out and auxiliary heads learn T5 feature alignment, next-event text and remaining-time prediction while the backbone and denoisers are frozen.
Go to primary source ↓staircasePDF pp. 12–13, Section 3.4, Eqs. (16)–(19), Figure 6; p. 21, Section 5.3
Staircase decoding shares lower-layer grounding and relays hidden states at staggered depths to continuous reasoning positions. A prefix projector feeds a frozen Qwen3.5-0.8B text reconstructor; only the reasoning branch and prefix projector train under reconstruction cross-entropy.
Go to primary source ↓staircase-panel-labelsPDF p. 13, Figure 6 panel labels and caption
The graphic prints both Traditional CoT and Latent CoT with an a) label, whereas the caption identifies Traditional CoT as (a), Latent CoT as (b), and Staircase reasoning as (c). The rightmost Staircase panel is consistently labeled c) in the graphic and (c) in the caption. This is a source labeling defect, not a difference in reasoning mechanisms.
Go to primary source ↓data-mixturePDF pp. 13–15, Section 4.1, Figures 7–10; p. 19, Section 4.5
Data span internet and egocentric video, robot-free UMI-style trajectories, public and internal teleoperation, and recovery/takeover episodes. The specified OpenVID slice contains 1.2 million clips. Contact-pose perturbations enrich recovery behavior; a full embodied-data count and mixture weights are not provided.
Go to primary source ↓synchronizationPDF pp. 15–17, Section 4.2, Figure 11
Visual-motion and end-effector-motion signals are correlated over lags before captioning. Actions are shifted, boundaries trimmed, and fields subsequently resampled; weak estimates are quarantined or down-weighted. Missing streams and invalid kinematics are filtered.
Go to primary source ↓caption-samplingPDF pp. 17–18, Sections 4.3–4.4, Figures 12–13
Action-boundary segmentation precedes Task/Subtask/Action/Segment captions, with an optional human layer. Offline vision-language and action clustering supplies a balanced sampler; production uses a small selected subset, whose absolute size is unspecified.
Go to primary source ↓staged-trainingPDF pp. 19–20, Table 1 and Section 5.1
Train the video DiT first with frozen T5 and causal VAE, then freeze video and train action. The selected video anchor is step 45 on a 50-step schedule, with small unspecified jitter. Video training uses uniform timestep sampling, length-dependent caption dropout, EMA and shape bucketing.
Go to primary source ↓chunk-adaptationPDF p. 21, Section 5.4; p. 19, Table 1
Optional next-chunk adaptation updates both DiTs on observation-centered history windows with frozen T5 global instructions; clustering is rerun on these windows. This adaptation starts from the event-pretrained backbone.
Go to primary source ↓inference-modesPDF pp. 21–22, Sections 5.5.1–5.5.2; p. 10, Video-Action Timestep Mapping
Event mode receives next-event language, predicts an event segment, executes and advances observation. Unified mode predicts fixed action horizons with history and interchangeable continuous, atomic-instruction or Staircase context. Both use asymmetric video-anchor conditioning rather than a new video forward for every action step.
Go to primary source ↓systemsPDF pp. 22–24, Sections 6.1–6.2
The systems recipe includes DMuon, fused kernels, communication overlap, isolated multi-event packing, distribution-matching distillation retaining action supervision, and blockwise FP8 quantization. The authors report 10 Hz end-to-end inference and 53% worse action MAE when action supervision is omitted during distillation, without absolute MAE or a hardware-specific latency protocol.
Go to primary source ↓video-protocolPDF p. 24, Section 7.1, Evaluation Protocol and Benchmark construction
The internally constructed video benchmark contains 200 held-out in-distribution and 50 OOD tasks and follows WorldArena metrics. WALL-WM normally starts from Wan2.2-5B; Wan baselines do not receive its event-centric embodied training. Table 2 does not separate the ID and OOD scores.
Go to primary source ↓video-tablePDF p. 25, Table 2, WALL-WM and Wan2.2-5B rows
WALL-WM versus Wan2.2-5B: interaction quality 0.434/0.226, instruction following 0.391/0.298, trajectory accuracy 0.234/0.223, motion smoothness 0.771/0.683, semantic alignment 0.886/0.805, image quality 0.503/0.527 and aesthetic quality 0.393/0.409.
Go to primary source ↓spatial-probesPDF pp. 25–26, Table 3 and Section 7.1 analysis, Figure 15
On CO3Dv2, WALL-WM has point/depth errors 0.271/0.132 and AUC@5/AUC@30 0.210/0.727; WAN2.1-14B reports 0.284/0.151 and 0.200/0.736. The accompanying discussion uses VidFM-3D probing and cross-view correspondence evaluation.
Go to primary source ↓robot-protocolPDF pp. 26–27, Section 7.2, Experimental setup
The main policy uses event mode on internal tabletop bimanual arms. Policies share physical task definitions, instructions, multi-view observations, scene randomization and rubrics. U-Scratch lacks event pretraining and event reasoning; it is distinct from adapted unified mode. Trial counts, seeds and confidence intervals are not reported.
Go to primary source ↓diverse-resultsPDF p. 27, Section 7.2.1; p. 36, Table 5, Diverse Manipulation
Across seven tasks, average Task Progress is 75.86 for event-mode WALL-WM, 63.00 U-Scratch, 55.64 pi0.5, 39.97 DreamZero and 29.71 LingBot-VA. DreamZero leads basket picking (97.8 versus 88) and pi0.5 leads glasses placement (67 versus 64).
Go to primary source ↓reasoning-resultsPDF p. 28, Section 7.2.2; p. 36, Table 5, Reasoning Manipulation
Five-task means are WALL-WM-E 71.60, U-Scratch 59.50, pi0.5 56.40, DreamZero 32.70 and LingBot-VA 31.60. pi0.5 leads Pair Up Items with 77 versus event mode 36. The deployed reasoner generates explicit next-event descriptions.
Go to primary source ↓dexterity-resultsPDF pp. 28–29, Section 7.2.3 and Figure 18; p. 36, Table 5, Dexterous Manipulation
The scored tasks are Insert Wireline and Put Stationery in Case. Means are WALL-WM-E 32.00, U-Scratch 31.25, DreamZero 25.00, LingBot-VA 24.00 and pi0.5 15.00. Fan unboxing is illustrated but is not a scored row in Table 5.
Go to primary source ↓generalization-resultsPDF pp. 29–30, Section 7.2.4 and Figure 19; p. 36, Table 5, Generalization
Four tasks share a cluttered scene and randomized instructions. Means are WALL-WM-E 53.75, DreamZero 28.50, pi0.5 24.00 and U-Scratch 18.50. LingBot-VA is unevaluated. Cover Pot with Lid favors U-Scratch, 32 versus 26.
Go to primary source ↓combined-ablationPDF p. 30, Table 4 and Section 7.2.5
A pretrained fixed-length baseline without View-Interaction Self-Attention scores 32.6 on reasoning and 22.0 on generalization, versus 71.6 and 53.75 for event mode. Press Button in Order is 0/64. The paper explicitly attributes the comparison to the combined change in view modeling and execution format.
Go to primary source ↓discussion-limitsPDF pp. 31–32, Section 8
The authors acknowledge platform-aligned pretraining advantages and unequal baseline-specific tuning. Narrow, low-data tasks may favor fixed-horizon optimization; dense temporal annotations remain a dependency. Model scaling benefits are asserted without a controlled scaling table.
Go to primary source ↓appendix-designPDF pp. 33–35, Sections 9.1–9.4, Figures 20–21
The appendix motivates native text-to-video initialization and a dual-tower design; Figure 20 is explicitly conceptual, not an architectural sweep. VAE temporal robustness and Wan cross-view PCA are diagnostics. KV-cache history is argued not to resolve fixed semantic endpoints.
Go to primary source ↓qualitative-appendixPDF pp. 35–39, Section 9.5, Figures 22–24
Appendix galleries show pretraining events, evaluation scene variations, and generated versus executed frames. Figure 24 labels G as generated and E as execution; these examples do not provide an aggregate execution-success measure.
Go to primary source ↓progress-rubricsPDF p. 40, Table 6, caption and task rows
The table describes task-specific partial-credit rubrics on a 10-point scale, normalized to 0–100. For Insert Screwdriver into Cup the component scores are identify 2, grasp 3, insert 4 and retract 1. The Put Stationery in Case row assigns 3.5 for opening, one per item for three items, 1.5 for zipping and one for retracting; these sum to nine by reader arithmetic, despite the caption's ten-point convention.
Go to primary source ↓frame-count-gapPDF p. 10, VAE-aligned video stream and Figure 5; p. 20, Video Pretraining
Section 3.3 defines a leading-one/trailing-four VAE compression rule; Section 5.1 instead calls 65 latent frames equivalent to 129 raw frames under stride-2 subsampling. The text does not reconcile the two frame-count conventions.
Go to primary source ↓8.5 Primary sources
WALL-WM: Carving World Action Modeling at the Event Joints ↗
PDF · 24,298 extracted words
Source fingerprint
81cbdda89cfc88e109e8226db7b0606734a7c624dd1b9653a1782c02a6a68c4f