MemoryWAM: Efficient World Action Modeling with Persistent Memory
1. Paper overview
In one sentence: MemoryWAM keeps detailed initial and recent observations alongside per-frame gist memory, improving memory-dependent control while reducing the cost of a still-growing historical cache. E02E03E04E09E11E13
| At a glance | What to know |
|---|---|
| Research problem | Source description A robot may need an earlier cue after it disappears or becomes occluded. Recent-frame policies forget that cue; full-history attention preserves it with growing cost. MemoryWAM seeks persistent context with a smaller storage and retrieval burden. E02 |
| Core mechanism | Source description The hybrid cache assigns different fidelity to task-onset observations, recent interaction, and older history, with an attention mask that permits old visual KVs to be evicted while retaining their gist KVs. E04 |
| A key reported result | RMBench nine-task average: 83.0% Average task success rate. 50 expert demonstrations per task; success over 100 rollouts per task, with task-specific appendix additions. LingBot-VA 78.2%; π0.5 10.4%; FastWAM 5.9%. The reported margin over LingBot-VA is 4.8 percentage points. MemoryWAM ties or leads each row, but Observe and Pick Up remains 27% and Battery Try 41%. The separate no-pretraining Observe and Pick Up comparison is only 5% versus 3%. E08E09E13 |
| Reading caution | Author claim The authors identify weak semantic understanding and reasoning inherited from video diffusion models. E12 |
Core contributions
- Source description
The hybrid cache assigns different fidelity to task-onset observations, recent interaction, and older history, with an attention mask that permits old visual KVs to be evicted while retaining their gist KVs. E04
Figure 2. Video prediction teaches the representation; the action branch uses its cached features at deployment. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the bottom: observations pass through the VAE encoder, while noisy actions enter their own encoder. Yellow anchor tokens, orange recent-frame tokens, and green gist tokens form the hybrid history available to attention. Follow the text-conditioning arrows to both branches and the proprioception arrow to the action branch. The upper decoders show the model's two training outputs. Figure 2's caption and Section 3.2 qualify that picture: deployment processes a clean observation latent once, caches the video representations, and repeatedly denoises only the action tokens. Appendix A.1 then connects this prediction to execution and the next observation update. E03E04E14
What it supports. The efficiency design has two distinct parts: avoid repeated future-video denoising at inference, and reduce the historical visual tokens retained for attention. The video expert still processes observations and maintains memory, so disabling video generation does not remove the learned visual representation from the control loop.
Where the evidence stops. The visible VAE decoder and noisy-frame pathway describe training capability. They are not evidence of inference-time imagined rollouts or planning over generated futures; the stated deployment protocol disables video generation.
2. Motivation
2.1 The problem and the proposed response
A robot may need an earlier cue after it disappears or becomes occluded. Recent-frame policies forget that cue; full-history attention preserves it with growing cost. MemoryWAM seeks persistent context with a smaller storage and retrieval burden. E02
2.2 What this reading follows
A robot can see the correct object now and still need an earlier observation to act correctly. MemoryWAM addresses this gap by deciding which parts of visual history deserve full detail and which can survive as compact learned summaries. Its video/action transformer learns from both prediction targets, but deployment uses observed-frame features to generate actions without synthesizing future video. Read the architecture and mask first, then the efficiency curves and executed-task results. The central tradeoff is supported by strong benchmark performance; the interpretation needs the appendix's training exceptions and an ablation table whose individual rows are more nuanced than the surrounding prose. E02E03E04E09E11E13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Other mechanisms |
| Quadrant | Outside quadrants |
3.1 Evidence-based assessment
Insufficient evidence to decide
WAM, persistent memory, and efficient inference are supported. The architecture is specifically a coupled video/action MoT; two experts alone do not establish a hierarchical Dual-system, which the conclusion lists as future work. Prediction is joint video/action learning during training but action-only denoising from cached observations at inference, without generated-future inverse dynamics. 'Other mechanisms' and 'Outside quadrants' are plausible under an inference-based taxonomy, but the entire recorded classification is not unambiguously verified by this architecture. E03E04E12E14
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Follow one observation-to-action cycle
An observation first becomes a latent through the causal video VAE. The video expert processes that clean latent once and updates the layerwise visual cache; the action expert then attends to that cache while denoising its next chunk. During training, video and action prediction both contribute flow-matching losses, teaching the representation through a dense visual target as well as actions. At deployment, the future-video generation branch is disabled. The policy executes all 16 predicted actions before the next chunk. It samples observation mosaics at sub-step indices {3,7,11,15}, appends them to its buffer, re-encodes with the VAE, and prefills the updated memory. This sequence distinguishes learned dynamics supervision from actual control: feedback comes from newly observed images after execution, and the paper does not describe planning by evaluating generated candidate futures. E03E05E06E14
Figure 3. The mask preserves access to an old frame's gist after its full visual tokens leave memory. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read queries down the Q axis and keys across K. The paper labels clean visual frames f, gist tokens g, and noisy action tokens a. This schematic has three frames: frame zero is an anchor and frame two is recent. In the final-frame rows, colored entries remain under the anchor's visual tokens and the middle frame's gist g₁, while the middle frame's full visual-token columns f₁ are gray. That is the crucial compression boundary. Earlier rows cannot access future frames. The caption omits noisy video frames because they share the actions' historical context; it does not omit their training objective. E04E05E14
What it supports. The memory is a persistent collection of per-frame summaries, with selected full frames retained alongside it. Section 3.3 and Appendix A.1 agree with the mask: an old non-anchor frame's visual KVs can be evicted while its gist KVs continue to support subsequent video and action queries.
Where the evidence stops. The drawing uses one anchor and one recent frame. The implementation retains two initial and four recent frames, with eight gist tokens per frame. It should not be read as a single fixed-size summary of the whole episode.
5.2 Understand what survives eviction
The method allocates high fidelity to two initial frames and four recent frames, while preserving eight gist tokens for every frame. A gist token is not a stored text description: it is a learned token whose contextual representation is built by attention to visual tokens and historical context. Once a frame becomes old and is not an anchor, subsequent queries lose direct access to its full visual KVs but retain access to its gist KVs. With 120 visual tokens and eight gist tokens per frame, the paper's long-term ratio is 15. Reader analysis: this changes the coefficient of growth rather than making the history constant-size. It also introduces a representational bottleneck, so success requires task-relevant information to survive compression. The full anchor and recent-frame paths offer detail that this bottleneck may discard. E04E05E14
5.3 Use task-level evidence to test the memory story
The benchmark average favors MemoryWAM, but the ablations provide the more discriminating evidence about its mechanism. Press Button collapses when gist memory is removed, whereas Cover Blocks is more sensitive to removing full anchor frames. That pattern is consistent with different temporal information needs, although the paper does not directly measure what each token remembers. Reader analysis: the source's broad claim that all components improve performance needs qualification because removing anchors raises Press Button success. Likewise, full attention ties hybrid on that task, and the two-task average differs by only one percentage point. The real-world Shell Game advantage is larger in trial counts, but the authors connect baseline failures to missed swaps during inference. A fair mechanism test should therefore control both which history is available and when the robot observes it. E09E10E11
5.4 Training and inference
During training
Both branches use continuous flow matching with 1000 training timesteps. Video/action logit-normal shifts are 5.0/1.0; their scheduler-reweighted MSE losses have equal weights. Interleaved clean/noisy sequences train under the hybrid mask. Every clean video conditioning latent is mixed with Gaussian noise using a uniformly sampled ratio in [0,1]. E06
AdamW uses learning rate 2×10^-4, weight decay 0.01, β=(0.9,0.95), eight GPUs, and batch size one per GPU. Training uses bfloat16, FSDP, blockwise activation checkpointing, and clipping at 1.0. Appendix exceptions matter: Swap T includes action history, and Observe and Pick Up adds RoboTwin pretraining. Other tasks omit those additions; the video backbone remains pretrained. E05E06E13
During inference
Process the current clean latent through the video DiT once, then iteratively denoise the action chunk against cached visual features: 50 steps in simulation, 10 in the real world. Video generation is disabled. Execute all 16 actions, sample new mosaics at sub-steps {3,7,11,15}, append them to the observation buffer, re-encode, and update memory before the next chunk. E03E14
5.5 Implementation flow
- Encode and couple modalities
A causal video VAE converts observations into latents for the pretrained Wan2.2-TI2V-5B video DiT. A separate approximately 1B action DiT yields an approximately 6B MoT system. Both experts have 30 blocks; action weights are initialized by interpolating pretrained video weights. Proprioception is appended to the action expert's text context. E03E05
- Retain detail selectively
Keep full visual tokens for two initial anchor frames and four recent frames. The anchors are fixed task-onset observations, not outputs of a learned event detector. Add eight learnable gist tokens per frame; gist tokens attend to their frame and historical context. Their cached representations persist after older full-frame KVs are evicted. E04E05E14
- Align representation and visibility
RMBench's three views form a 384×320 mosaic producing 120 latent visual tokens per frame after patchification. Gist positions share the video's 3D RoPE coordinates with fixed spatial markers, and action queries use the same positional basis. The training mask exposes the hybrid history that inference will retain. E05E06
6. Experiments & results
MemoryWAM compresses older visual history into per-frame gist tokens while retaining full initial and recent observations. A video/action mixture of transformers learns with both prediction targets, then executes action chunks without generating future video. Its strongest evidence concerns memory-dependent manipulation and the efficiency of its cache design.
6.1 Read the original evidence
Figure 4. Hybrid memory trades a slowly growing cache for lower measured cost and full-attention-level Press Button success. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the panels' units separate. The first plots latency in milliseconds against sequence length in frames; the second plots GPU memory in megabytes against the same length. The caption limits both measurements to one layer and one pass. Both legends identify the red square-marked curves as Ours, meaning hybrid memory; the green Full Attention curves rise more rapidly over the reported range. Orange TTT and blue RNN augment sliding-window attention with additional memory modules, whose outputs are added to the attention output. The third panel measures a different quantity: robot-policy success on RMBench's Press Button task, rather than a property of one transformer layer. E07E04
What it supports. Hybrid and full attention both reach 87% Press Button success, compared with 67% for TTT and 78% for RNN. Through the reported 1,600-frame range, hybrid memory has lower plotted latency and memory cost than the recurrent alternatives, supporting the paper's measured efficiency–performance tradeoff.
Where the evidence stops. These layer-level timings cannot be substituted for end-to-end action-chunk latency or a full-model GPU-memory ratio. The graph also does not prove that the growing gist cache remains cheaper than fixed-state methods at arbitrarily long horizons.
Table 1. Persistent memory accompanies a large gain over bounded-context baselines, with a smaller gain over LingBot-VA. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each task row reports success over 100 rollouts; the stated training budget is 50 demonstrations per task. Compare π0.5's direct policy, FastWAM's bounded observation context, and LingBot-VA's full-history memory before reading the final MemoryWAM column. The average summarizes nine different tasks, so inspect individual rows as well: several are saturated, while Observe and Pick Up and Battery Try remain difficult. Appendix A.1 modifies the training picture: Observe and Pick Up uses RoboTwin pretraining and Swap T adds action history. Those conditions belong with the table, even though they are documented on a later page. E08E09E13
What it supports. MemoryWAM reports 83.0% average success versus LingBot-VA's 78.2%, a 4.8-percentage-point margin. It ties or leads every listed task. Nevertheless, 27% on Observe and Pick Up and 41% on Battery Try show that the strong average does not imply uniformly reliable memory-dependent manipulation.
Where the evidence stops. The comparison is not uniformly controlled for pretraining or action-history access. The appendix's separate no-pretraining Observe and Pick Up comparison is only 5% versus 3%. No confidence intervals or repeated-seed variability accompany Table 1.
Table 2. The physical tests show an appreciable Shell Game margin and a one-trial Look and Press margin. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each fraction as successes over trials, not a percentage printed without a denominator. Shell Game asks the robot to select the cup hiding a cube after a human swaps cups. Look and Press asks it to remember two numbers, press the corresponding left and right buttons that many times, and finish with a rear-button press. Appendix A.2 specifies 50 and 100 demonstrations for these tasks, respectively. The tests use ARX dual arms and a D455 RGB camera. Compare each baseline within the same row; the two tasks impose different memory demands and training-data budgets. E10E15
What it supports. MemoryWAM succeeds in 18/20 Shell Game trials versus LingBot-VA's 13/20, and in 15/20 Look and Press trials versus 14/20. These are executed robot outcomes. The stronger margin concerns tracking through cup swaps, whereas the button-counting advantage over the full-history model is only one observed success.
Where the evidence stops. Twenty trials provide limited precision. The authors say LingBot-VA misses some swaps during slow inference; the table therefore combines representation quality with observation-timing effects and does not isolate a causal benefit of gist compression alone.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RMBench nine-task average 50 expert demonstrations per task; success over 100 rollouts per task, with task-specific appendix additions. | 83.0% Average task success rate | LingBot-VA 78.2%; π0.5 10.4%; FastWAM 5.9%. The reported margin over LingBot-VA is 4.8 percentage points. MemoryWAM ties or leads each row, but Observe and Pick Up remains 27% and Battery Try 41%. The separate no-pretraining Observe and Pick Up comparison is only 5% versus 3%. E08E09E13 |
| Press Button memory-mechanism comparison WAM memory variants on RMBench; accompanying efficiency curves measure a single layer and single pass. | 87% Task success rate | Full attention 87%, TTT 67%, RNN 78%. Hybrid matches full-attention success with lower plotted latency and memory over long histories. These curves do not measure complete action-chunk latency. E07 |
| Shell Game Physical ARX dual-arm setup; 50 demonstrations and 20 evaluation trials. | 18/20 Successful trials | LingBot-VA 13/20; π0.5 5/20. This measures executed cup selection after swaps. The authors attribute some baseline failures to slow observation updates; memory fidelity and deployment latency are not isolated. E10E15 |
| Look and Press Physical button-counting task; 100 demonstrations and 20 evaluation trials. | 15/20 Successful trials | LingBot-VA 14/20; π0.5 0/20. The advantage over the full-history baseline is one trial; the source supplies no uncertainty estimate. E10E15 |
| Hybrid-memory component ablation Cover Blocks and Press Button on RMBench; Table 3 reports task rates without separately restating the trial count. | 98% / 87%; 92.5% average Success rate: Cover Blocks / Press Button; two-task average | No anchors: 58%/90%; no gist: 75%/5%; no sliding window: 96%/69%; full attention: 96%/87%. Gist removal has the largest average loss, but component effects depend on the task. Removing anchors improves Press Button by 3 percentage points. Hybrid's average gain over full attention is only 1 point. E11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. The components have task-dependent effects; average improvements should not conceal the exceptions. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read across a task before reading the average. Removing anchors replaces their full visual latents with gist tokens; removing the sliding window similarly substitutes gist for recent full-frame latents. Removing gist instead deletes the long-term gist memory. Full Attention retains all historical visual latents. For Cover Blocks, anchor removal is particularly damaging, whereas Press Button is highly sensitive to gist removal. The final average combines these distinct responses. The text above the table broadly says component removal degrades performance, but the Press Button row explicitly gives 90% without anchors versus 87% for the complete hybrid design. E11
What it supports. Removing gist reduces Press Button from 87% to 5% and produces the lowest two-task average, 40%. Hybrid's 92.5% average exceeds full attention's 91.5% by one percentage point; full attention ties it on Press Button. The evidence supports complementary, task-dependent memory roles rather than universal benefits from every component.
Where the evidence stops. The paper's prose overstates the uniformity of the ablation effect: anchor removal improves one reported task. No repeated-seed variation is supplied, and the small hybrid/full-attention difference does not establish that compression reliably improves retrieval accuracy.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify weak semantic understanding and reasoning inherited from video diffusion models. E12
Persistent gist storage still grows with episode length. The 15× token ratio is neither constant-memory inference nor a measured 15× reduction in total GPU memory. E04E07
The ablation prose overgeneralizes component benefits: anchors hurt the reported Press Button rate, and full attention ties hybrid there. Small gains have no reported seed variation or confidence intervals; two real-world tasks do not establish broad robotic generalization. E10E11
7.2 Questions for discussion
- Would cue placement later in an episode reduce the benefit of fixed initial anchors?
- Does hybrid memory still outperform full attention when observation timing and inference latency are matched?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires matching the pretrained backbone, camera mosaics, flow schedules, positional conventions, visibility mask, cache eviction, and task-specific training additions. The source gives no total training steps, training GPU model, software versions, or explicit frozen-module policy. E05E06E13E14
Physical deployment additionally requires the ARX/gripper and D455 setup, 256×352 inputs, and the reported demonstrations. One RTX 4090 runs 10-step action denoising; control is 10 Hz within chunks with approximately 0.3 seconds between chunks. Timing comparisons should preserve this feedback schedule. E14E15
Proposed checks should separate cache retention from information availability and end-to-end timing: repeat task-specific ablations with matched seeds, then test cache cost over increasing history length. Neither check was performed for this report. E07E11E14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Move the remembered cue across the cache boundary
Reader-proposed, not performed: repeat Cover Blocks and Press Button with hybrid, no-anchor, and no-gist variants using identical demonstration sets, training budgets, and paired evaluation seeds. Then introduce a controlled task-relevant cue either in the initial anchor interval or later, followed by delays that move it outside the four-frame recent window. Keep task semantics and action opportunities matched, and report each task separately with uncertainty across repeated training runs. Measure success as a function of cue position and delay. If initial anchors specifically preserve onset information, removing them should be more damaging for onset cues than otherwise matched later cues. If gist carries intermediate history, removing it should be especially harmful after a later cue leaves recent memory. Failure to observe those interactions would weaken this interpretation of the components. E04E05E08E11E14
Check 2: Separate memory cost from missed observations
Reader-proposed, not performed: first profile hybrid and full-attention variants on the same GPU, precision, weights, recorded observation stream, and action-denoising schedule over increasing histories through the source's 1,600-frame range. Record KV bytes, visual-prefill time, action-denoising time, and complete chunk latency separately. Next evaluate Shell Game with matched observation timestamps and a controlled swap schedule, then with each policy's natural deployment timing. Keep the source's 16-action chunks and distinguish its 10 real-world denoising steps from the 50-step simulation protocol. Slower KV growth would support the cache-efficiency mechanism. If the success margin largely disappears when observation timing is matched, the original advantage would be better explained by fewer missed swaps than by a superior compressed representation alone. E04E07E10E14E15
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version on p. 1; introduction and overview on pp. 2–3; architecture, conditioning arrows, attention-mask query/key orientation, and Eqs. (2)–(7) on pp. 4–5; training/model settings on p. 6; all Figure 4 panels and simulation protocol on p. 7; Tables 1–2 and real-world task images on p. 8; Table 3 and limitations on p. 9; and all appendix training, feedback, hardware, demonstration, and timing details on pp. 14–15. Figures 1–6 and Tables 1–3 were viewed. All six final crops were separately inspected. The Figure 4 crop preserves readable Full Attention, TTT, RNN, and Ours legends in both efficiency panels, all axis labels, and all four success values. Figure 3's retention pattern agrees with the method text; Table 3's exception to the prose is explicitly preserved. Reference pages 10–13 were read in the supplied text chunks but not rendered. No separate supplemental images or code were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title page and abstract (p. 1)
- 1 Introduction (p. 2)
- 2 Related Work (p. 3)
- 3 Method, including 3.1–3.3 (pp. 3–5)
- 4 Experiments, including 4.1–4.5 (pp. 6–9)
- 5 Conclusion and limitations (p. 9)
- References (pp. 10–13)
- A.1 Implementation Details (p. 14)
- A.2 Real-World Experiment Details (pp. 14–15)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction-only figure omission above was addressed by inspecting the retained PDF's Figures 1–6 and Tables 1–3; reference-list pages were read as text, not rendered.
- Identity verified against the title page: arXiv:2606.20562v1, 18 June 2026. The title and all eleven authors agree with the catalog; no different revision was supplied. The catalog affiliation string contains a stray '1mm'; metadata retains only the three institutions visible on the title page.
- No code, project website, or separate supplement was inspected, and no experiments were reproduced. All five supplied text chunks were read, including the full appendix.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
E01PDF p. 1, title/author block and arXiv margin
The title is MemoryWAM: Efficient World Action Modeling with Persistent Memory; the artifact is arXiv:2606.20562v1 [cs.RO], 18 June 2026. Eleven authors match the catalog, with Sizhe Yang and Juncheng Mu marked equal contributors. Affiliations are The Chinese University of Hong Kong, Tsinghua University, and Zhejiang University.
Go to primary source ↓E02PDF pp. 2–3, Section 1 and Section 3.1, Eq. (1)
The problem is memory-dependent manipulation: a bounded recent-observation policy cannot access older task cues, while caching every past observation increases latency and GPU memory with trajectory length.
Go to primary source ↓E03PDF p. 4, Figure 2/caption, Section 3.2, Eqs. (2)–(3)
A causal video VAE feeds a video DiT coupled to a separate action DiT in a mixture-of-transformers architecture. Video prediction supplies training supervision; inference processes a clean observation latent once to update the video KV cache and denoises actions against cached representations.
Go to primary source ↓E04PDF pp. 4–5, Section 3.3, Figure 3/caption, Eqs. (4)–(7)
Hybrid memory combines recent and initial anchor-frame tokens with per-frame gist tokens. Old non-anchor, non-recent visual KVs are evicted; their gist KVs remain. Figure 3 has query rows and key columns: final-frame queries retain access to the anchor and the middle frame's gist but not its full visual tokens. With L=120 and M=8, d=L/M=15; long-term token storage is O(NM), not constant in N.
Go to primary source ↓E05PDF p. 6, Section 4.1, Model architecture
The backbone is Wan2.2-TI2V-5B with T5 and a causal VAE; a 30-block, approximately 1B action expert is initialized by hidden-dimension interpolation from the video DiT, giving approximately 6B total parameters. RMBench uses a 384×320 three-camera mosaic, 120 visual tokens per latent frame, 14-dimensional state/action vectors, horizon 16, two initial frames, four recent frames, and eight gist tokens per frame. Proprioception enters the action expert's text context; gist/video/action positions share the specified 3D RoPE basis.
Go to primary source ↓E06PDF p. 6, Section 4.1, Training setup
Both branches use continuous flow matching with 1000 training timesteps and logit-normal shifts of 5.0 for video and 1.0 for actions. Interleaved clean/noisy sequences use the hybrid visibility mask. All clean video conditioning latents are mixed with Gaussian noise at a uniformly sampled ratio in [0,1]. Equally weighted, scheduler-reweighted video/action MSE losses are optimized with AdamW at 2×10^-4 on eight GPUs, batch size one per GPU.
Go to primary source ↓E07PDF pp. 6–7, Section 4.2 and Figure 4(a–c)/caption
Efficiency curves measure one layer and one forward pass as sequence length varies. TTT/RNN modules augment sliding-window attention by additive outputs. Hybrid memory stays below their latency and memory curves through the reported 1600-frame range. Figure 4(c) reports Press Button success: full attention 87%, TTT 67%, RNN 78%, hybrid 87%.
Go to primary source ↓E08PDF p. 7, Section 4.3, Simulation Experiments
RMBench has nine dual-arm, memory-dependent tasks. The stated protocol trains each method with 50 expert demonstrations per task and evaluates success over 100 rollouts. The compared models are π0.5, FastWAM, LingBot-VA, and MemoryWAM.
Go to primary source ↓E09PDF p. 8, Table 1, all task rows and Average row; adjoining Section 4.3 continuation
Average success is 83.0% for MemoryWAM, 78.2% for LingBot-VA, 5.9% for FastWAM, and 10.4% for π0.5. MemoryWAM's rows are 27%, 100%, 100%, 100%, 94%, 41%, 100%, 98%, and 87% in printed task order. LingBot-VA scores 13% on Observe and Pick Up, 79% on Cover Blocks, and 84% on Press Button. The reported average margin over LingBot-VA is 4.8 percentage points.
Go to primary source ↓E10PDF p. 8, Section 4.4, Figure 5 and Table 2, both task rows
Shell Game requires identifying the cup over a cube after human swaps; Look and Press requires remembered button counts followed by a completion press. Table 2 reports MemoryWAM 18/20 and 15/20, LingBot-VA 13/20 and 14/20, and π0.5 5/20 and 0/20, respectively. The authors attribute some LingBot-VA Shell Game failures to missing swaps during slow inference.
Go to primary source ↓E11PDF pp. 8–9, Section 4.5 and Table 3, Cover Blocks/Press Button/Average rows
Removing anchor or recent full-frame tokens substitutes gist tokens; removing gist tokens removes long-term gist memory. Cover Blocks scores 58/75/96/96/98 percent for no anchors/no gist/no sliding window/full attention/hybrid; Press Button scores 90/5/69/87/87 percent. Averages are 74.0/40/82.5/91.5/92.5 percent. The prose's general degradation claim has a task-level exception: no anchors exceeds hybrid on Press Button.
Go to primary source ↓E12PDF p. 9, Section 5, Limitations and future work
The authors identify limited semantic understanding and reasoning inherited from video diffusion models, and suggest incorporating dual-system architectures or unified models as future directions.
Go to primary source ↓E13PDF p. 14, Appendix A.1, Training setup
Training uses bfloat16, FSDP, activation checkpointing on every DiT block, and gradient clipping at 1.0. Swap T additionally uses action history; Observe and Pick Up uses RoboTwin pretraining. Other tasks omit these additions. The separate Observe and Pick Up comparison without pretraining gives MemoryWAM 5% and LingBot-VA 3%.
Go to primary source ↓E14PDF p. 14, Appendix A.1, Inference protocol
Each transformer block caches the first two and four most recent clean frames, plus eight gist tokens from every past frame, which are never evicted. The policy executes all 16 predicted actions, samples mosaics at sub-step indices {3,7,11,15}, appends them to the observation buffer, re-encodes with the VAE, prefills memory, and denoises the next action chunk. Action denoising uses 50 steps in simulation and 10 in the real world; video generation is disabled.
Go to primary source ↓E15PDF pp. 14–15, Appendix A.2, Hardware Setup and Imitation Learning Details; Figure 6
The platform has ARX dual arms with parallel grippers and a RealSense D455 RGB camera. Shell Game has 50 demonstrations; Look and Press has 100, with numbers from 1 to 5. Inputs are cropped/resized to 256×352, deployment uses one NVIDIA RTX 4090, and control runs at 10 Hz within chunks with approximately 0.3 seconds of inter-chunk latency.
Go to primary source ↓8.5 Primary sources
MemoryWAM: Efficient World Action Modeling with Persistent Memory ↗
PDF · 7,534 extracted words
Source fingerprint
a2beffa47141bcc8d5593d0eb50be681ee286a75f543848cef86e7f56d310c8a