MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation
1. Paper overview
In one sentence: MemoryVAM makes episode history available to both video prediction and action decoding, improving memory-dependent manipulation while retaining a history cache whose cost grows with episode length. e02e03e05e06e10e11e14e15e17
| At a glance | What to know |
|---|---|
| Research problem | Source description A robot can see the same object arrangement after different repetition counts. Recent images then omit the progress needed to choose another manipulation or stop. MemoryVAM asks whether episode history should condition both predicted futures and the controls derived from them. e02 |
| Core mechanism | |
| A key reported result | LIBERO-Mem manipulation success: UNet: 42.5%; DiT: 40.5%. Average success rate (%). 10 simulation tasks; 20 rollouts per task; same demonstrations for trainable baselines, with split/count details unspecified. Corresponding no-memory baselines: 5.0% and 3.0%; Gated Memory Policy: 19.5%. Both backbone families improve. Reader calculation: each gain is 37.5 percentage points. Full UNet still scores 0% on B×7; full DiT scores 35%. No confidence intervals are given. e09e10 |
| Reading caution | Source description History storage and processing grow with episode length; very long deployment may require truncation, subsampling or recurrence. Broader pretraining and model-size scaling were not tested. e15 |
Core contributions
Figure 2. One compressed episode history reaches the video model, the action decoder and the completion gate. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the upper left: episode images become CLIP embeddings, and learned Perceiver queries compress them into the purple memory matrix G. Trace the purple lines into the middle band. Its UNet and DiT diagrams are alternative backbone instantiations, not consecutive stages. The UNet keeps separate text and memory attention; DiT combines their conditioning. At the bottom, memory accompanies denoised video tokens into the action decoder. Section 3.3 separately specifies the projected-memory interface for the DiT action transformer. At the upper right, the gate pools backbone features, language and memory to estimate completion. These arrows describe information flow, not all training gradients. e03e04e05e06e08e10e19
What it supports. History can influence control indirectly by changing the video-derived representation and directly by reaching the action decoder. This explains the two pathways tested in Table 1. The gate adds a goal-dependent stopping decision; the architecture does not identify generated frames themselves as robot commands.
Where the evidence stops. Figure 2 omits the gate’s training-time visual-feature detachment specified in Appendix B.3. Its history ends at t−1, consistent with Section 3.1, while Appendix B.5 appends the current frame before compression. That indexing difference remains unresolved.
2. Motivation
2.1 The problem and the proposed response
A robot can see the same object arrangement after different repetition counts. Recent images then omit the progress needed to choose another manipulation or stop. MemoryVAM asks whether episode history should condition both predicted futures and the controls derived from them. e02
2.2 What this reading follows
Imagine asking a robot to pick up the same toy three times. After each placement, the camera may show nearly the same scene, but the next decision changes when the requested count is reached. MemoryVAM addresses this ambiguity by compressing earlier observations into a shared memory for the video predictor and action decoder. A separate completion gate combines that memory with visual features and the instruction. This reading follows the two information paths, then separates simulation success, physical execution and generated-video diagnostics. The strongest evidence concerns the value of both pathways; reliable long counting and a fully specified reproduction remain open. e02e03e05e06e10e11e14e15e17
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | IDM |
| Quadrant | Q4 · Dual-system × IDM |
3.1 Evidence-based assessment
Insufficient evidence to decide
The memory/long-horizon category and Dual-system architecture are supported: a predictive video backbone feeds a separate action decoder, with staged training. The recorded IDM label is plausible in a broad visual-feature-to-control sense, but the supplied equations define an action policy rather than an explicit inverse-dynamics transition objective. The exact Q4 assignment therefore remains uncertain; the catalog snapshot is preserved. e03e05e08
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Recover progress before asking what happens next
The toy-pickup example is an observation-aliasing problem: the same local scene can correspond to different numbers of completed repetitions. MemoryVAM first maps earlier frames to global CLIP embeddings, then lets learned queries attend over their ordered history. Compression produces a fixed-size interface for the rest of the policy; it does not make the retained history cache constant-cost. Temporal positions tell the compressor when an event occurred, while the validity mask excludes padding. The resulting G enters the predictive backbone before action features are extracted. Reader interpretation: this gives the model a way to distinguish futures that look plausible locally but correspond to different episode progress. The architecture provides that opportunity; the pathway ablation supplies the evidence that this use of history benefits the tested policy. e02e04e05e10e15
Figure 3. Three physical task designs make earlier observations relevant to the next action. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the top orange strip as the memory phase of Shell Game: the toy can begin under different cups, and the cups may be swapped before the visible end state. The adjacent blue strip supplies cup-lifting action examples. In the lower left, follow Init State, Pick, Place and Return; the initial location must survive the intermediate move to the plate. In the lower right, compare Pick 1st and Lift 1st with the later question-mark panels. The gray ‘Visually Similar’ bands mark the ambiguity that a short observation window may leave unresolved. They illustrate the task rationale rather than measuring image similarity. e02e11e17
What it supports. The tests exercise distinct dependencies on history: an occluded object’s location, an earlier placement position and repetition progress. A common memory interface could help all three, but the task montage alone cannot establish that a policy actually uses history. Table 2 supplies the physical success measurements.
Where the evidence stops. These are task illustrations, not a complete evaluation protocol. Demonstration counts, real-robot rollout counts, per-task stopping thresholds and exact rollout limits are absent from the supplied source, so they must remain unspecified.
5.2 Teach memory through change, then train controls
The first training stage combines future-video learning with reconstruction of consecutive CLIP feature differences and a completion objective. Delta reconstruction asks memory to preserve what changed across the episode, with the previous embedding also supplied to the auxiliary decoder. It is removed from the inference mechanism. Completion labels come from demonstration endings, so the method avoids manually marking every intermediate subgoal, but still depends on meaningful episode boundaries. The gate pools history, visual features and language; its visual input is detached for the gate loss. Stage 2 freezes the video backbone while continuing to train memory-related modules and the action policy, leaving both memory pathways active. Reader interpretation: the central separation is between learning a predictive representation and learning controls from it, not between a wholly frozen memory module and a new controller. e06e07e08
5.3 Follow three evidence levels without merging them
First ask whether the robot completes the task. Table 1 supplies simulation success and the VPP pathway ablation; Table 2 supplies physical rollouts. Next ask whether the predicted visual trajectory remains useful. Table 4(a) evaluates generated video at several horizons, and its DiT results show that memory does not improve every horizon. Finally ask what stopping evidence measures. Table 4(b) evaluates gate timing, while Table 4(c) measures motion in generated frames before and after ground-truth completion. A model can improve either diagnostic without establishing reliable physical termination. Reader interpretation: the strongest mechanistic conclusion is that both injection paths help this VPP implementation. Claims about well-calibrated completion probabilities, arbitrary long counting or broadly scalable episodic reasoning require additional tests beyond the reported aggregate gains. e09e10e11e12e13e14e15
5.4 Training and inference
During training
Stage 1 starts from a memoryless checkpoint to avoid early token collapse. Backbone attention, Recap, memory projections, reconstruction decoder and gate train jointly; VAE and CLIP text encoder stay frozen. Video denoising or flow matching is combined with delta reconstruction and gate BCE. e07
Only the final 10 demonstration frames are gate positives, upsampled fivefold. The gate loss detaches visual backbone features while permitting gradients through memory and language branches. Reconstruction is an auxiliary training decoder, not an inference-time control module. e06e07
Stage 2 freezes the video backbone while keeping Pathway 1 active. Appendix B.4 lists Recap, injection projections, gate, Video Former and GCDenoiser as continuing to train with action prediction. Freezing the backbone therefore does not freeze the entire memory system. e08
During inference
Appendix B.5 appends the current CLIP embedding, compresses history, predicts futures/features, and decodes actions. The listed local context is four frames, video horizon 16 frames and action chunk eight actions. Rollout ends above a setting-specific gate threshold or at its maximum step count. e08
The precise refresh/execution cadence is unspecified. History indexing also differs: Section 3.1 ends at t−1, while Appendix B.5 includes the current frame before compression; reproduction must resolve this boundary. e17e19
5.5 Implementation flow
- Compress retained history
CLIP ViT-B/32 supplies one global embedding per frame. Projection, temporal positions and a padding mask feed learned Perceiver queries. Training caps history at 400 embeddings; evaluation retains 600 frames. VPP uses 16 tokens of width 384; Cosmos uses 32 of width 1024. e04
- Condition the predictive representation
UNet adds a separate memory cross-attention output to text attention. DiT appends memory tokens to text conditioning. This first pathway changes the future prediction and the features later supplied to control. e05
- Condition action decoding directly
VPP concatenates memory and denoised video tokens; DiT appends projected memory to visual conditioning in its action transformer. The separate action module receives history directly and through the predictor. It predicts controls rather than executing generated pixels. e03e05
- Estimate completion
The Cue Gate combines pooled backbone features, mean-pooled memory and instruction embeddings through projected branches and an MLP. Its probability represents goal completion, allowing remembered progress and language to influence stopping. e06
6. Experiments & results
MemoryVAM compresses episode history into tokens shared by a video predictor and downstream action decoder, with a learned completion gate. On LIBERO-Mem, UNet success rises from 5.0% to 42.5%; physical DiT rollouts also improve. The evidence supports history-conditioned predictive representations, while long-count reliability, scaling and exact reproduction remain unresolved (e02, e03, e10, e11, e15, e17).
6.1 Read the original evidence
Table 1. Both memory pathways give the highest VPP average, with substantial variation across tasks. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first four rows together before comparing different model families. They hold the VPP backbone family fixed while removing memory or restricting it to one pathway. The rightmost Avg column summarizes ten tasks, each evaluated over 20 rollouts. The columns separate single placements, repeated actions and sequencing/spatial tasks. The caption defines B as bowl, Bt as bottle, P as plate, Cs as cheese and Bskt as basket; it does not expand E. Next compare the two Cosmos rows to assess whether the gain persists with a DiT. Finally, inspect external baselines while retaining their different architectures and information access. e09e10
What it supports. VPP reaches 42.5% with both paths, compared with 28.5% for predictor-only memory, 7.5% for action-only memory and 5.0% without memory. Cosmos improves from 3.0% to 40.5%. The complementary-path interpretation is stronger than a claim that adding any memory module will deliver these gains.
Where the evidence stops. The authors use the same demonstrations for trainable baselines, but Embodied-SlotSSM receives oracle subgoals. No confidence intervals are reported. Full UNet success remains 0% on B×7, so the average does not establish reliable long-count manipulation.
Table 2. Physical success improves, but the longest tested pickup count remains difficult. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the grouped Toy Pickup heading. Its 1×, 3× and 5× columns are three requested repetition counts, and the adjacent Avg averages those pickup settings only. Place and Return and Shell Game are separate task columns. Compare Ours (DiT) against the no-memory Cosmos-RF row within each setting. The source’s physical setup uses PiPER arms with a fixed Intel RealSense D415 camera. Section 4.4 separately reports 78.0% averaged across five settings; that is a different aggregation from the 78.3% pickup average visible here. Neither average replaces the per-setting values, especially the drop at five repetitions. e11
What it supports. Memory-conditioned DiT reaches 95%, 100% and 40% on one, three and five pickups, plus 80% on Place and Return and 75% on Shell Game. These are executed physical tasks. The result supports transfer beyond simulated benchmarks while showing that successful short counting does not guarantee longer-count reliability.
Where the evidence stops. The source does not give real-robot trial counts or uncertainty. Do not infer a denominator from percentage increments or carry the simulation table’s 20-rollout protocol into this table. Task-specific success adjudication is also not fully formalized.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO-Mem manipulation success 10 simulation tasks; 20 rollouts per task; same demonstrations for trainable baselines, with split/count details unspecified. | UNet: 42.5%; DiT: 40.5%. Average success rate (%) | Corresponding no-memory baselines: 5.0% and 3.0%; Gated Memory Policy: 19.5%. Both backbone families improve. Reader calculation: each gain is 37.5 percentage points. Full UNet still scores 0% on B×7; full DiT scores 35%. No confidence intervals are given. e09e10 |
| Dual memory pathway ablation VPP on the same LIBERO-Mem evaluation. | Both pathways: 42.5%. Average success rate (%) | Pathway 1 only: 28.5%; Pathway 2 only: 7.5%; neither: 5.0%. This supports complementary uses of memory within the tested architecture; it does not prove that every possible action-only memory design must fail. e09e10 |
| Physical memory-dependent manipulation DiT policy with PiPER arms and fixed RealSense D415; real-robot trial counts and split unspecified. | Toy Pickup 1×/3×/5×: 95/100/40 (average 78.3); Place and Return: 80; Shell Game: 75. Task success rate (%) | No-memory Cosmos-RF: 0/0/0, 0 and 5 respectively. Executed robot outcomes support physical transfer; the 5× drop limits claims of robust counting. The 78.3% average covers only pickup variants, not all tasks. e11 |
| Autoregressive video prediction LIBERO-Mem; predicted chunks are fed back as context; evaluation sample count/split unspecified. | At H_v/8H_v: UNet 18.32/19.84; DiT 18.65/16.24. FVD (lower is better) | VPP 43.36/37.81; Cosmos-RF 22.54/26.11. DiT worsens at 2H_v and 4H_v: 25.37/22.06 versus 23.78/21.23. Video fidelity improvements are horizon-dependent and do not measure executed task success. e12 |
| Completion timing and generated-video stopping VPP reconstruction ablation on LIBERO-Mem; gate thresholds 0.5 and 0.9. | With reconstruction: 35.5/28.8 frames; stop ratio 0.46. Mean absolute timing error (frames); stop ratio | Without reconstruction: 44.3/62.0 frames and 0.53; no memory: stop ratio 1.11. Stop ratio compares mean frame-difference motion after versus before completion. Lower values concern generated motion, not physical stopping guarantees. Timing error alone does not establish probability calibration. e13e14 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4(a–c). Video fidelity, gate timing and generated motion test different consequences of memory. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the three panels separate. Panel (a) evaluates generated video: SSIM points upward, while LPIPS and FVD point downward. H_v is the 16-frame prediction chunk; the later FVD columns evaluate autoregressive horizons. Panel (b) measures absolute completion-timing error in frames at the displayed gate thresholds. Its w/ and w/o rows mean with and without the reconstruction auxiliary loss, not with and without memory. The same distinction applies in panel (c), where No mem is a separate baseline. Read Ratio using Appendix D.1’s post/pre-completion motion definition; the adjacent Energy column lacks a separate definition or normalization. e08e12e13e14
What it supports. Reconstruction reduces aggregate timing error from 62.0 to 28.8 frames at threshold 0.9, and the generated-video stop ratio is 0.46 with memory and reconstruction versus 1.11 without memory. Improvements are not universal: DiT FVD worsens at 2H_v and 4H_v, and the 1× timing error worsens at threshold 0.5.
Where the evidence stops. These diagnostics do not measure physical stopping success or probability calibration. The motion ratio uses fixed grayscale windows around ground-truth completion; a smaller ratio establishes reduced generated motion within those windows, not correct task execution.
7. Analysis & limitations
7.1 What the evidence leaves open
History storage and processing grow with episode length; very long deployment may require truncation, subsampling or recurrence. Broader pretraining and model-size scaling were not tested. e15
The described shuffled-history diagnostic lacks numerical effect sizes. Third-placement comparisons also change requested repetition count, so language-conditioned gate differences alone do not isolate memory. The cited language-conditioned Recap-query ablation has no reported comparison. e06e16e19
7.2 Questions for discussion
- Would shuffling history still impair progress recognition with instruction, current view and retained frames held fixed? (e06, e16)
- Does reconstruction improve real stopping reliability as well as generated-video motion and gate timing? (e11, e13, e14)
8. Reproducibility audit
8.1 Requirements and known gaps
Rebuild cached CLIP histories and the two attention interfaces. Table 3 supplies a 270,000-step memoryless warm start, 500,000 total Stage 1 steps, video/memory rates 5×10^-6/5×10^-5 and batch 32; Stage 2 uses rate 10^-4, batch 16 and 10 action DDIM steps. e04e05e07e08
Missing requirements include optimizer family, Stage 2 duration, compute hardware/time, software versions, demonstration counts/splits, real-robot trial counts, deployment thresholds and rollout limits. Table 4(c) does not separately define its Energy column. These cannot be filled from the project link. e17e11e14
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Repeat the four-way pathway comparison with matched execution
Reader-proposed, not run: train VPP with neither pathway, predictor-only memory, action-only memory and both pathways using the same demonstrations, common warm-start checkpoint, training budgets and paired seeds. Keep the same modules allocated where feasible and mask the disabled interfaces, documenting any remaining parameter or optimization differences. Evaluate each condition on the same 20 starts per LIBERO-Mem task per seed. Choose one stopping protocol on separate validation data and hold it fixed. Report taskwise success, count overshoots and gate timing with uncertainty. If the full model no longer consistently exceeds predictor-only memory, the claim that direct action injection adds a reliable benefit would weaken. e05e07e08e09e10e17
Check 2: Separate temporal memory from instruction and threshold effects
Reader-proposed, not run: at fixed current observations, instruction, history length and retained frame set, compare ordered versus shuffled histories for models trained with and without reconstruction. Preserve the validity mask and shuffle before assigning temporal positions. Unlike the source’s 3× versus 5× comparison, keep the requested count fixed. Measure absolute and signed gate timing errors at 0.5 and 0.9, missed terminations, false stops and probability calibration on naturally sampled episode frames using the declared boundary-label window. If shuffling has no effect, chronology dependence is weakened; if reconstruction improves timing without improving calibration, retain the timing claim but reject the stronger calibration interpretation. e04e06e13e16
8.3 Reading coverage
Visual audit: Visually inspected the title/author/version page; Figures 1–6; Tables 1–4; the method equations, training and inference text; and appendix architecture, parameter, diagnostic and hardware pages. All five final original crops were separately viewed and checked for legibility and completeness. Figure 2 was cross-checked against Section 3.3 and Appendices A–B: its arrows express information flow, its shared action schematic is supplemented by the DiT-specific text, and gate visual-feature detachment is stated only in the training text. The history-indexing discrepancy is retained in the architecture caution. Pages 9–10 contain references and were read as text. No separate supplements, code or linked media were supplied to this review.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 11, 12, 13, 14, 15, 16. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Sections 1–2: Introduction and Related Work
- Sections 3.1–3.6: history, compression, dual conditioning, gate, backbones and training
- Sections 4.1–4.5: protocols, generation, simulation, real robots and ablations
- Sections 5–6: Conclusion and Limitations
- References, pp. 9–10
- Appendices A.1–A.5, B.1–B.5, C, D.1, E, F and G, pp. 11–16
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Identity note (e01): the inspected title and all seven authors match the catalog. This is arXiv:2606.20679v1, dated 13 June 2026; no title or author revision discrepancy was observed. No other edition was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This extraction limitation was addressed by inspecting all six figures, four tables and supporting method pages in the supplied PDF.
- Separate supplemental material availability has not been fully verified.
- No code, external project materials or separate supplements were inspected; no experiments were reproduced. All five supplied text chunks were read. Reference-list pages were text-read; scientific and title pages were also visually inspected.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin
The title and seven authors match the catalog. The artifact identifies arXiv:2606.20679v1 [cs.RO], 13 June 2026. Jiang and Yu contributed equally. Affiliations are UCLA, Nirvana Robotics, UC San Diego and University of Utah.
Go to primary source ↓e02PDF pp. 1–2, Section 1 and Figure 1
Repeated manipulation can revisit visually similar states while requiring different next actions. Recap-Cue conditions both predictive features and action decoding on episode history.
Go to primary source ↓e03PDF p. 3, Figure 2 and Section 3 equations; p. 5, Section 3.5
The video model predicts future observations and exposes intermediate features; a separate downstream policy predicts an action chunk. Figure 2 shows two backbone alternatives and a three-input Cue Gate.
Go to primary source ↓e04PDF p. 4, Sections 3.1–3.2; p. 11, Appendix A.1; p. 14, Table 3, History cache and Recap Compressor rows
One global CLIP ViT-B/32 embedding represents each frame. Ordered, masked history is compressed by learned Perceiver queries. Training histories have at most 400 embeddings and evaluation retains at most 600 frames; VPP uses 16 memory tokens of width 384 and Cosmos uses 32 of width 1024.
Go to primary source ↓e05PDF p. 4, Section 3.3 equations; p. 11, Appendices A.2–A.3
UNet uses an additive parallel memory cross-attention branch; DiT appends memory to text conditioning. The action interface concatenates memory with denoised tokens for VPP, or appends projected memory to visual conditioning for the DiT action transformer.
Go to primary source ↓e06PDF pp. 4–5, Section 3.4; p. 12, Appendix A.4, Eq. (1); p. 13, Appendix B.3; p. 14, Table 3, Cue Gate rows
The Cue Gate pools visual features and memory and projects a CLIP language embedding into a fused classifier. Final 10 demonstration frames are positive and upsampled fivefold. Backbone features are detached for the gate loss; memory and language branches receive gradients.
Go to primary source ↓e07PDF p. 12, Appendices A.5 and B.1–B.2, Eq. (2); p. 14, Table 3, Training rows
Stage 1 combines video loss, delta reconstruction with weight 1.0 and gate BCE with weight 0.1. It trains attention and added modules, freezing VAE and CLIP text encoder, after a memoryless warm start. Reconstruction predicts consecutive CLIP differences and is training-only. Table 3 lists a 270,000-step warm start and 500,000 total Stage 1 steps, video/memory rates 5×10^-6/5×10^-5 and batch 32.
Go to primary source ↓e08PDF p. 13, Appendices B.4–B.5; p. 14, Table 3, Video and policy, VPP action decoder and Training rows
Stage 2 freezes the video backbone but retains active memory injection and trains Recap, projections, gate, Video Former and GCDenoiser with action prediction. Inference appends the current CLIP embedding, predicts futures/features and an action chunk, and stops above a setting-specific gate threshold or at a maximum step count. Context, video and action horizons are 4, 16 and 8; VPP has 224 video tokens plus 16 memory tokens. Stage 2 rate is 10^-4, batch 16, with 10 action DDIM steps.
Go to primary source ↓e09PDF pp. 5–6, Section 4.1; p. 7, Table 1 caption
Memoryless comparisons retain the backbone and action module. All trainable baselines are fine-tuned on the same demonstrations, with oracle subgoals for Embodied-SlotSSM. LIBERO-Mem covers 10 tasks and Table 1 reports 20 rollouts per task; the supplied description gives no demonstration counts or explicit train/test split.
Go to primary source ↓e10PDF p. 7, Table 1, all method rows and Avg/B×7 columns; p. 8, Section 4.5, Dual injection ablation
Average success is 5.0/28.5/7.5/42.5% for VPP without memory/Pathway 1/Pathway 2/both, and 3.0/40.5% for Cosmos without/with memory. Gated Memory Policy reaches 19.5%, E-SlotSSM 15.0%, MemoryVLA 9.5%. B×7 is 0.0% for the full UNet and 35.0% for the full DiT.
Go to primary source ↓e11PDF p. 5, Figure 3 and caption; p. 6, Section 4.1, Tasks; p. 7, Section 4.4 and Table 2; p. 15, Figure 5 and Appendix F
Physical tasks test repetition counts, restoration of an initial position and hidden-object tracking after cup swapping. DiT success is 95/100/40% for 1/3/5 pickups (78.3% average), 80% for Place and Return and 75% for Shell Game; no-memory values are 0/0/0%, 0% and 5%. Section 4.4 gives 78.0% across five settings. PiPER arms and a fixed RealSense D415 are specified; trial counts, demonstration counts and uncertainty are not.
Go to primary source ↓e12PDF p. 6, Section 4.2; p. 15, Table 4(a), all rows and horizon columns
Final-frame SSIM/LPIPS and autoregressive FVD evaluate LIBERO-Mem generation. UNet FVD improves 43.36→18.32 at H_v and 37.81→19.84 at 8H_v; DiT improves 22.54→18.65 and 26.11→16.24 respectively, but worsens 23.78→25.37 at 2H_v and 21.23→22.06 at 4H_v.
Go to primary source ↓e13PDF p. 8, Section 4.5, Reconstruction loss and completion timing; p. 12, Appendix B.2; p. 15, Table 4(b)
VPP reconstruction ablation reduces aggregate absolute completion error from 44.3 to 35.5 frames at threshold 0.5 and 62.0 to 28.8 at 0.9. At threshold 0.5 the 1× row worsens 29.5→37.2. Authors infer better calibration, but the table reports timing error rather than a calibration metric.
Go to primary source ↓e14PDF p. 13, Appendix D.1 equations; p. 14, Table 3, Stop ratio rows; p. 15, Table 4(c) and caption
Stop ratio divides mean post-completion by mean pre-completion grayscale frame-difference energy, with a stabilizer. Resolution is 64×64, windows are 48 frames before and 72 after, epsilon is 10^-8. Ratios are 1.11 without memory, 0.53 with memory without reconstruction, and 0.46 with both. The Energy column has no separate definition or normalization in the supplied text.
Go to primary source ↓e15PDF p. 8, Section 6, Limitations
Retaining history costs memory and computation that grow with episode length. Truncation, subsampling and recurrent memory remain future work. Results on LIBERO-scale data and two backbones do not establish scaling to broader video pretraining or larger backbones.
Go to primary source ↓e16PDF p. 14, Appendix E, Cue Gate Probability over Time and Decision-Critical Frame Comparison; p. 15, Shuffled Memory Control
The text describes completion scores over repetitions, CLIP similarity above 0.94 at the third placement of 3× versus 5× tasks, and scores above 0.9 versus below 0.2. It reports degraded timing after shuffling history without numerical results or a plotted curve.
Go to primary source ↓e17PDF pp. 12–14, Appendices B–C and Table 3; pp. 6–7, experimental protocols; p. 15, Appendix F
The paper supplies rates, batches, Stage 1 step counts, token dimensions and physical hardware. It does not specify optimizer family, Stage 2 duration, compute hardware/time, software versions, per-task deployment thresholds or maximum step values, action-chunk execution cadence, or complete data/split details.
Go to primary source ↓e18PDF p. 6, Figure 4 and caption; p. 16, Figure 6 and caption
Qualitative sequences compare memory-conditioned and memoryless generation, including physical rollout observations versus predictions and simulated video consistency. These selected frames illustrate the authors’ interpretation, without supplying a separate robot-success measurement.
Go to primary source ↓e19PDF p. 4, Section 3.1; p. 13, Appendices B.3 and B.5; p. 8, Section 4.5
Section 3.1 defines history through t−1, whereas Appendix B.5 appends the current frame before compression. Appendix B.3 refers to a language-conditioned Recap-query ablation in Section 4.5, but that section supplies no such comparison.
Go to primary source ↓8.5 Primary sources
MemoryVAM: Integrating Memory into Video Action Model for Robot Manipulation ↗
PDF · 7,659 extracted words
Source fingerprint
51be0160af2eba64da2d1968c16da9aaa23193fa515f53711121bfc887709a76