MEM: Multi-Scale Embodied Memory for Vision Language Action Models
1. Paper overview
In one sentence: MEM combines compressed semantic history with dense recent video so a robot can remember task progress and adapt its manipulation, at the cost of summary supervision and an incompletely specified training recipe. e-identitye-factorizatione-languagee-traininge-long-ablatione-adaptation
| At a glance | What to know |
|---|---|
| Research problem | Source description A kitchen robot must remember both which ingredients it already retrieved and what happened during its last failed grasp. Keeping all camera history is expensive; compressing everything into language discards spatial detail. MEM assigns these requirements to different representations while targeting within-episode control lasting up to fifteen minutes. e-problem |
| Core mechanism | Source description A learned language-memory update compresses semantic history instead of accumulating every past subtask instruction. e-language |
| A key reported result | Open fridge with an ambiguous hinge direction: Figure 7 labels the gap +62%; its percentage axis indicates a 62-percentage-point separation. Success-rate difference. Correction-trained memory and memoryless policies; opening must take at most four grasps. Memoryless π0.6 trained on the same intervention data. Supports failure-history-conditioned strategy switching under this grasp-budget protocol; exact endpoint rates are not tabulated. e-adaptatione-adaptation-protocol |
| Reading caution | Reader analysis Exact result tables and significance tests are absent. Error bars and task-specific scoring constrain comparisons. The authors’ attribution of robustness against causal confusion to data diversity is not an isolated causal test. e-visual-scopee-long-protocole-dexteritye-pretraining |
Core contributions
- Source description
A learned language-memory update compresses semantic history instead of accumulating every past subtask instruction. e-language
- Source description
A pretrained image encoder becomes a video encoder through attention-pattern changes, retaining the backbone’s current-frame token budget. e-encoder
- Source description
Robot experiments test long-horizon progress, correction-conditioned adaptation, memory representation and memory pre-training. e-long-ablatione-adaptatione-memory-resultse-pretraining
Figure 2. A semantic memory loop supplies subtasks to a policy with short-term visual history. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left panel from the goal, camera observations and previous memory at the bottom to the two text outputs at the top. The dashed loop returns the updated summary to the next high-level decision; a separate dashed connection delivers the predicted subtask to the low-level policy on the right. There, stacked camera frames enter the video memory encoder, and noise enters the action expert before continuous actions emerge. These arrows describe an inference interface, not a training-gradient path. Section III-D separately says action-expert gradients do not flow into the VLM backbone. e-factorizatione-languagee-training
What it supports. MEM divides remembered information by its use: semantic history supports subtask selection, while recent visual context supports continuous control. The external LLM described in III-B generates training labels for summaries; the learned high-level policy is responsible for subsequent summary updates. The diagram does not show a future-world simulator.
Where the evidence stops. The high-level diagram and III-A use current observations, but III-B’s policy notation includes a recent observation window. That input discrepancy is unresolved in the supplied paper. The drawing alone also does not establish whether the two policies share weights.
2. Motivation
2.1 The problem and the proposed response
A kitchen robot must remember both which ingredients it already retrieved and what happened during its last failed grasp. Keeping all camera history is expensive; compressing everything into language discards spatial detail. MEM assigns these requirements to different representations while targeting within-episode control lasting up to fifteen minutes. e-problem
2.2 What this reading follows
Consider a robot preparing a recipe: it must remember that the butter is already on the counter, yet also remember how its last grasp slipped. MEM gives these memories different forms. A high-level policy carries a language summary between decisions and chooses the next subtask; a low-level policy receives recent camera and robot-state history to produce actions. The supplied v2 paper evaluates this design on physical robots, including long kitchen tasks and recovery from failed manipulations. This reading follows the information flow, separates task progress from success rate, and uses the ablations to examine which memory and training choices the evidence actually supports. e-identitye-factorizatione-languagee-traininge-long-ablatione-adaptation
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | Dual-system |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The explicit high-level summary/subtask policy and low-level action policy support the recorded hierarchical, dual-system VLA classification. The encoder represents past observations; neither joint future-world/action prediction nor inverse dynamics is described. World-action prediction quadrants are therefore not applicable. This judgment follows the architecture, not joint training alone. e-factorizatione-encodere-training
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 A summary is a task-state update, not a transcript
The high-level policy needs a compact account of what still matters for the remaining task. In training, an off-the-shelf LLM receives subtask annotations and success/failure indicators and generates successive summaries. The annotations supervise the robot policy; the paper does not require that external LLM to rewrite every deployment-time memory. During execution, the policy predicts both an updated summary and a next subtask, then feeds its summary into the next decision. Compression can merge equivalent details, such as several objects placed in one cabinet, and avoid endlessly appending repeated unsuccessful instructions. Section IV-A argues that this reduces mismatch between efficient demonstrations and retry-heavy rollouts. This is selective retention, not a guarantee that every failed event should always be deleted: the goal remains preserving information needed for future execution. e-factorizatione-languagee-long-ablation
Figure 4. Temporal attention compresses visual history before it reaches the language backbone. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the time axis from the older frames toward the current frame, then follow the layer axis from the input grids to the final representation. White arrows within a frame denote bidirectional spatial attention. The black arrows from the current patch toward earlier matching patches should be read as attention to history, consistent with the causal mask specified in III-C, rather than propagation of future information into the past. Every fourth ViT layer adds this temporal operation. The final representation retains current-timestep patch tokens, so the downstream backbone need not consume a separate token grid for every historical image. e-encodere-attention-appendixe-latency
What it supports. The encoder moves history compression into visual feature extraction while reusing image-encoder parameters. Separately, Figure 3 on this same inspected page reports a video-encoder curve below the 300 ms reference line through 16 frames for four cameras on one H100. That timing supports the architectural motivation in one measured configuration.
Where the evidence stops. Appendix C Eq. (3) nests an inner expression containing both spatial and temporal sets and does not write an explicit causal mask. This is not a transparent specification of III-C’s additive same-patch operation; a faithful implementation requires resolving that algebra rather than silently repairing it.
5.2 Compress history where visual features are formed
The low-level branch still needs details that a short sentence may discard: where an object was before occlusion and what the previous grasp looked like. MEM processes a recent image sequence inside a modified ViT. Spatial attention exchanges information within each frame, while every fourth layer adds causal attention over matching spatial positions across time. Only current-timestep patch representations continue into the VLA backbone, concentrating historical information before the expensive language processing. The encoder reuses existing learned projections and adds a fixed temporal embedding with zero value at the current timestep. Historical robot states are projected into tokens separately. This explains the claimed constant visual-token count at the backbone input; it does not mean the video encoder itself has zero extra computation as history grows. e-encodere-attention-appendixe-traininge-latency
5.3 Separate remembering task progress from adapting a grasp
The experiments answer different questions, so their metrics should stay separate. Figure 6 measures partial progress through kitchen tasks and compares long-term semantic and short-term visual memory. Figure 7 instead asks whether a correction-trained policy can finish a manipulation under a changed table height or a limited fridge-opening grasp budget. Figures 8 and 9 remove long-term language memory to probe visual-history representations and when those representations are learned. My interpretation is that this sequence gives stronger evidence for complementary memory functions than any one aggregate bar could. It still leaves causal alternatives: summary compression changes both length and content, and memory pre-training changes the temporal learning experience. Similar average dexterity in Figure 10 establishes compatibility on tested tasks, rather than a general proof that adding history cannot hurt. e-long-protocole-long-ablatione-adaptatione-adaptation-protocole-baselinese-memory-resultse-pretraininge-dexterity
5.4 Training and inference
During training
An off-the-shelf LLM labels summary transitions using annotated subtasks and success/failure indicators. It retains information relevant to future execution and compresses redundant details; the trained high-level policy performs deployment-time updates. e-language
Gemma3-4B initialization supports FAST action-token prediction plus an 860M-parameter flow-matching expert. Expert gradients do not enter the VLM; this does not establish that the entire backbone is frozen. Images are 448×448 with up to four streams. e-training
Pre-training mixes demonstrations, policy rollouts, human corrections and vision/video-language tasks. Six observations are sampled one second apart; post-training reaches 18 frames and a reported 54 seconds. Adaptation fine-tuning retains failed attempts preceding human corrections. e-data-runtimee-adaptation
During inference
Update semantic memory and the subtask, generate an action chunk, execute using asynchronous real-time chunking, and condition subsequent decisions on new observations. The paper uses inference-time or training-time RTC without a complete per-task schedule. e-factorizatione-data-runtime
5.5 Implementation flow
- Maintain semantic state
The high-level policy predicts the next subtask and updated memory from the goal, observations and previous summary. Figure 2 feeds the summary back and passes the subtask to the low-level policy. III-A specifies current observations; III-B instead writes a recent observation window, leaving this input scope ambiguous. e-factorizatione-language
- Encode recent history
Every fourth ViT layer adds causal temporal attention across the same patch location alongside bidirectional spatial attention. Historical patch outputs are discarded before the VLA backbone; current-timestep features carry compressed history. Historical proprioception uses one projected token per state. e-encodere-training
- Produce actions
The low-level policy conditions on the subtask and recent observations. Its flow-matching expert maps noise to continuous action chunks. No future-video rollout or inverse-dynamics action extraction is described. e-factorizatione-training
6. Experiments & results
MEM adds two forms of history to π0.6: a recurrent language summary for semantic task progress and a video encoder for recent manipulation context. A high-level policy updates the summary and selects a subtask; a low-level policy produces robot actions. Physical-robot experiments support complementary memory benefits and correction-conditioned adaptation, while leaving training details and parts of the attention specification unresolved.
The supplied paper reports its quantitative experiments as plots (Figures 3 and 6–10) and contains no numerical results table. This edition therefore uses original result plots and ablation plots in place of a table. Exact unlabeled bar values and unreported uncertainty values are not reconstructed. e-visual-scope
6.1 Read the original evidence
Figure 7. Remembering a failed manipulation helps a correction-trained policy change its next attempt. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row as a task-specific test, not as an additional stage of one procedure. The upper row tests chopstick pickup after training at higher table settings and evaluation at the lowest setting; success includes depositing the chopstick in a bin. The lower row tests a fridge whose hinge direction lacks obvious visual cues, with success restricted to opening in at most four grasps. Both policy variants receive correction data, but MEM can condition on the failed attempt retained in recent history. The images illustrate strategy changes; the adjacent bars provide the success-rate comparison under these definitions. e-adaptatione-adaptation-protocole-long-protocol
What it supports. The plot labels gaps of +11% for chopsticks and +62% for opening the fridge. Because the arrows span differences on a percentage success-rate axis, the corresponding interpretation is 11 and 62 percentage points, respectively. These results support correction-conditioned adaptation at execution time rather than demonstrating deployment-time weight updates.
Where the evidence stops. Exact bar endpoints are not tabulated. The paper states ten rollouts per policy and task or recipe generally, but does not detail how these particular percentage labels were aggregated. The snapshots do not establish that every successful rollout recovered through the same strategy.
Figure 8. Short-term visual memory supports several kinds of hidden-state reasoning. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the photos and task labels to connect each bar group to what must be remembered. Mug swapping tracks which cups have already visited the machine; finding an object requires recalling the correct drawer; groceries require recognizing when the bag is empty. Coffee tests counting, grilled cheese includes timing, and window cleaning tracks covered regions. Yellow denotes MEM, gray pooled history, green proprioceptive history and brown no memory. Crucially, this experiment removes MEM’s long-term language summary and implements alternatives on the same π0.6 backbone. It therefore evaluates observation-memory representations rather than the entire two-level system shown in Figure 2. e-baselinese-memory-resultse-task-rubricse-pretraining
What it supports. MEM has the highest displayed mean in every task group. The alternatives’ weaknesses vary with the information required: robot-state history cannot directly retain a human’s drawer choice, while aggressive visual pooling can discard distinctions among past object states. The breadth of the pattern is more informative than an invented exact aggregate score.
Where the evidence stops. Task progress combines different scoring rules, including binary completion and partial credit. These are six targeted robot tasks, not a universal memory benchmark. Backbone alignment does not by itself establish identical training histories or compute; Figure 9 addresses memory pre-training separately.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Long-horizon recipe setup and kitchen cleaning 42 training recipes; five seen recipes tested in unseen kitchens with unseen objects. Ten rollouts per policy and task or recipe are stated. | MEM has the highest plotted mean for both tasks; exact values are not tabulated. Task progress; mean ± standard error | No memory, video only, text only, and naive concatenated text plus video. Supports complementary memory roles. Completed-item/subtask progress is not whole-episode success. e-long-protocole-long-ablation |
| Open fridge with an ambiguous hinge direction Correction-trained memory and memoryless policies; opening must take at most four grasps. | Figure 7 labels the gap +62%; its percentage axis indicates a 62-percentage-point separation. Success-rate difference | Memoryless π0.6 trained on the same intervention data. Supports failure-history-conditioned strategy switching under this grasp-budget protocol; exact endpoint rates are not tabulated. e-adaptatione-adaptation-protocol |
| Chopstick pickup at an unseen table height Correction data from the upper half of the height range; test at the lowest setting. Success requires pickup and deposition. | Figure 7 labels the gap +11%, interpreted as 11 percentage points on its percentage axis. Success-rate difference | Memoryless π0.6 fine-tuned on the same intervention data. A smaller adaptation gain than fridge opening; neither label is a relative-percent improvement. e-adaptatione-adaptation-protocol |
| Six-task visual-memory suite and pre-training ablation Shared π0.6 backbone; MEM evaluated without long-term language memory. | MEM leads all six Figure 8 mean bars; memory pre-training improves all six Figure 9 mean bars. Task progress; plotted mean ± standard error | Pooled visual memory, proprioceptive memory, no memory, and memory added only during post-training. Supports representation and training benefits without isolating the responsible pre-training data component. e-baselinese-memory-resultse-pretraininge-long-protocole-task-rubrics |
| Dexterous manipulation without an explicit memory requirement Five out-of-the-box tasks; batch folding and box building use fine-tuning. | Figure 10 shows similar aggregate means, with task-specific gains and losses. Task progress | Memoryless π0.6. Compatibility with strong dexterity is supported; universal improvement or statistical equivalence is not established. e-dexterity |
| Video-memory inference scaling Four camera streams on one NVIDIA H100. | The video-encoder curve remains below 300 ms through the plotted 16-frame setting. Inference time in seconds | Naively encoding frames separately exceeds the plotted barrier. Configuration-specific measured latency, not a complete robot-loop timing guarantee. e-latency |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 6. The two memory channels and learned text compression each matter in long kitchen tasks. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: yellow is full MEM, green retains only video, pale gray retains only text, blue concatenates naive text history with video, and brown removes memory. Compare variants within each task before looking at the shaded aggregate. The vertical axis measures task progress, with error bars described as standard errors in IV-A. Recipe setup awards credit for requested items correctly retrieved and placed; kitchen cleaning awards credit per completed subtask. The recipe evaluation uses five training recipes in unseen kitchens with unseen objects. Thus these columns combine semantic bookkeeping with the physical ability to finish the requested operations. e-long-protocole-long-ablatione-language
What it supports. Full MEM has a clear visual advantage in both task groups, while neither memory channel alone reproduces it. The naive text-plus-video variant also falls short. The authors connect that last gap to repeated failed subtasks at inference creating histories unlike mostly efficient demonstrations, and propose compressed summaries as the remedy.
Where the evidence stops. The paper supplies no exact table for these bars. Task progress is not the fraction of entirely successful episodes, and the aggregate’s weighting is not specified. The compression comparison supports usefulness but does not separately control summary length, semantic selection and repeated-instruction robustness.
Figure 9. Learning to use history during pre-training improves later memory-dependent control. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the green and yellow bar within each task: green adds the video encoder during target-task post-training, while yellow includes memory in the earlier training stage. The green model is not trained from scratch; the text says it starts from a π0.6 checkpoint already trained on the same diverse robot data. This distinction makes the question whether the policy learned to use history early, rather than whether it had any pre-training at all. Read the six task pairs before the shaded average, and retain the standard-error bars when judging how strongly any individual task separates the variants. e-pretraininge-data-runtimee-task-rubricse-long-protocol
What it supports. Memory pre-training raises all six displayed means, with especially visible separations for grocery unpacking and window cleaning. This supports treating temporal feature learning as part of broad pre-training instead of relying solely on a small task-specific phase. It does not identify a numerical threshold of pre-training data sufficient to obtain the effect.
Where the evidence stops. The ablation does not independently vary robot data, non-robot video, optimization budget and memory horizon. It therefore cannot assign the gain to internet video alone or prove that data diversity causes robustness against causal confusion. No exact bar-value table is supplied.
7. Analysis & limitations
7.1 What the evidence leaves open
Exact result tables and significance tests are absent. Error bars and task-specific scoring constrain comparisons. The authors’ attribution of robustness against causal confusion to data diversity is not an isolated causal test. e-visual-scopee-long-protocole-dexteritye-pretraining
Appendix C Eq. (3) uses an inner expression with both spatial and temporal sets and omits an explicit causal mask. It does not transparently specify III-C’s additive, same-patch temporal operation. Figure 4’s backward attention arrows are consistent with querying history; the algebra remains unresolved. e-attention-appendixe-encoder
Memory beyond a single episode, including continual deployment over weeks or years, is future work. e-future
7.2 Questions for discussion
- Which failed attempts should semantic compression discard, and which must persist to prevent repeated mistakes? [e-language, e-long-ablation]
- Would a fully specified causal encoder preserve the gains under matched data, compute and memory horizons? [e-encoder, e-attention-appendix, e-pretraining]
8. Reproducibility audit
8.1 Requirements and known gaps
Reconstruction needs the base checkpoint, robot/video data mixture, subtask outcome annotations, summary-label generator and RTC implementation. The paper omits optimizer settings, training budgets, exact mixture weights, summary-model identity/prompt and comprehensive per-task memory schedules. e-languagee-traininge-data-runtimee-long-protocole-attention-appendix
Proposed checks: match summary token budgets while varying repeated failures, and test encoder causality plus single-image initialization equivalence. These target compression and the unresolved attention specification. e-languagee-long-ablatione-encodere-attention-appendix
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate semantic compression from a shorter prompt
Reader-proposed check, not performed: on the recipe and kitchen tasks, compare learned summaries, concatenated subtask history truncated to the same token budget, and a deduplicated subtask log. Keep the video branch, checkpoint, prompts and evaluation scenes fixed. Introduce a controlled number of repeated failed subtasks before recovery, then score omitted/repeated completed steps and the original task-progress metric. Record summary length and latency as well. A growing advantage for learned summaries at matched length would support semantic retention beyond a prompt-size benefit; equal performance after deduplication would weaken the claim that learned summarization is necessary to handle repeated-instruction shift. e-languagee-long-ablatione-long-protocol
Check 2: Verify the temporal mask before testing memory gains
Reader-proposed check, not performed: first obtain clarification of Appendix C Eq. (3) and the exact temporal/spatial composition. At initialization, compare a single-image encoder output with the modified encoder using the paper’s zero-current-time embedding, holding image weights and preprocessing fixed. Then perturb later frames in a sequence and test that representations of earlier timesteps do not change under the claimed causal mask. Finally, on the hidden-drawer task, compare correct history against history shuffled across episodes while keeping the current observation and policy weights fixed. Initialization mismatch, future leakage, or unchanged performance after removing relevant history would each falsify a different part of the claimed mechanism. e-attention-appendixe-encodere-traininge-task-rubricse-memory-results
8.3 Reading coverage
Visual audit: Inspected the title, all ten original figures, method/training pages, result plots, task rubrics and Appendix C equations. All six final crops were separately viewed at their saved dimensions. Figure 4’s arrows point from current queries toward earlier patch positions, consistent with its causal caption; the unresolved mismatch is Appendix C’s algebra. III-A and III-B differ in high-level observation conditioning. Reference-only pages 10–12 were read in the complete text but were not visually inspected. No quantitative table is present.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 13, 14, 15. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; I. Introduction
- II. Related Work
- III. Multi-Scale Embodied Memory for VLAs (A–D)
- IV. Experimental Evaluation (A–C)
- V. Conclusion; Acknowledgements; References
- Appendix A. Contributions
- Appendix B. Task Details (all tasks)
- Appendix C. Video encoder with Space-Time separable attention
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- The supplied PDF is arXiv:2603.03596v2, dated 8 March 2026; the catalog submission date is 4 March 2026. Title and all 17 authors agree. No earlier version was supplied, so revision changes were not compared.
- The acquisition warning that text extraction does not reconstruct figure images was addressed by inspecting original PDF pages and all six final crops. Reference-only pages 10–12 were read as text, not rendered.
- Separate supplemental material availability has not been fully verified.
- No external project page, linked model card, code or dataset was inspected, and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title, author block, affiliation footnote and arXiv margin
The title and all 17 authors match the catalog. The artifact is arXiv:2603.03596v2, 8 March 2026. Affiliations are Physical Intelligence, Stanford University, UC Berkeley and MIT.
Go to primary source ↓e-problemPDF pp. 1–3, abstract and Sections I–III-A
Dense recent observations and compressed semantic history address different requirements; experiments target within-episode tasks lasting up to fifteen minutes.
Go to primary source ↓e-factorizationPDF p. 3, Section III-A, unnumbered policy factorization; p. 4, Figure 2
A high-level policy predicts a subtask and updated summary; the low-level policy predicts continuous action chunks from recent observations and the subtask. Figure 2 shows the recurrent summary and subtask interface.
Go to primary source ↓e-languagePDF p. 3, Section III-B
An off-the-shelf LLM labels summary transitions from subtask annotations and success/failure indicators, compressing irrelevant details. III-B conditions the high-level policy on recent observations, whereas III-A writes only the current observation.
Go to primary source ↓e-encoderPDF pp. 4–5, Section III-C and Figure 4
Every fourth ViT layer adds temporal attention at matching spatial patches with a causal mask. Spatial attention remains bidirectional; only current-timestep representations pass to the VLA backbone. No new learnable encoder parameters are introduced.
Go to primary source ↓e-attention-appendixPDF p. 15, Appendix C, position embedding and Eqs. (1)–(3)
Temporal embedding satisfies e(0)=0; query/key/value projections are reused with RMSNorm. Equation (3) nests attention with nonempty spatial and temporal sets inside spatial attention rather than explicitly specifying III-C’s same-patch causal operation.
Go to primary source ↓e-latencyPDF p. 4, Figure 3, axes, legend and caption
With four camera streams on one NVIDIA H100, the video-encoder curve stays below the 300 ms barrier through the displayed 16-frame setting; naive frame encoding exceeds it.
Go to primary source ↓e-trainingPDF p. 5, Section III-D
Gemma3-4B initialization, FAST action-token prediction and an 860M flow-matching expert are used. Expert gradients do not enter the VLM. Images are 448×448 with up to four streams; historical states use linear projections.
Go to primary source ↓e-data-runtimePDF p. 5, Section III-D, right column
Pre-training mixes demonstrations, policy rollouts, human corrections and vision/video-language tasks. Six observations are spaced one second apart; post-training reaches 18 frames and 54 seconds. On-robot execution uses inference-time or training-time RTC.
Go to primary source ↓e-long-protocolPDF p. 5, Section IV-A; p. 13, Appendix B.1
Training uses 42 recipes; five seen recipes are evaluated in unseen kitchens with unseen objects. The paper states ten rollouts per policy and task or recipe and mean ± standard error in graphs. Recipe and kitchen scores count completed items/subtasks.
Go to primary source ↓e-long-ablationPDF p. 6, Figure 6 and Section IV-A
Combined MEM has the tallest progress bars for both long-horizon tasks. Video-only, text-only and uncompressed subtask-history variants underperform. The authors attribute naive history failures to repeated subtask instructions at inference.
Go to primary source ↓e-adaptationPDF p. 7, Figure 7 and Section IV-B
Memory and memoryless policies receive correction data; MEM retains the preceding failed attempt in visual history. Figure 7 labels success-rate gaps +11% for chopsticks and +62% for opening the fridge.
Go to primary source ↓e-adaptation-protocolPDF p. 14, Appendix B.2(a–b)
Chopstick training uses table heights in the upper half of the range; evaluation uses the lowest setting. Success requires pickup and deposition. Fridge success requires opening in at most four grasps.
Go to primary source ↓e-baselinesPDF p. 7, Section IV-C, Comparisons
Pool Memory averages independently encoded past observations while retaining the current observation separately; Proprio Memory retains robot states. Comparisons share the π0.6 backbone and omit long-term language memory from MEM.
Go to primary source ↓e-memory-resultsPDF p. 8, Figure 8 and Section IV-C
MEM has the highest displayed progress across six memory tasks; pooled and proprioceptive histories have task-dependent weaknesses. Values are plotted without exact numerical tables.
Go to primary source ↓e-pretrainingPDF pp. 7–8, MEM-Posttrain-Only description, Figure 9 and discussion
The post-training-only variant starts from pre-trained π0.6 but gains the video encoder during task post-training. Memory pre-training yields higher mean bars across six tasks.
Go to primary source ↓e-task-rubricsPDF p. 14, Appendix B.3(a–f)
Tasks test nonrepeated mug placement, finding a hidden object without opening an incorrect drawer, complete grocery unpacking, exactly two coffee scoops and lid replacement, timed sandwich cooking, and complete window wiping.
Go to primary source ↓e-dexterityPDF p. 9, Figure 10; pp. 14–15, Appendix B.3(g–m)
The comparison separates five out-of-the-box tasks from batch folding and box-building fine-tuning. Aggregate bars are similar, with task-specific gains and losses; scoring ranges from item counts to binary success.
Go to primary source ↓e-futurePDF p. 9, Section V
Memory across episodes, lasting weeks, months or years, is proposed as future work.
Go to primary source ↓e-visual-scopePDF p. 4, Figures 2–4; pp. 6–9, Figures 5–10; pp. 13–15, Appendix A–C
The paper contains architecture diagrams, task photos and quantitative plots, but no numerical result table. The appendix supplies task descriptions and attention equations rather than tabulated plot values.
Go to primary source ↓8.5 Primary sources
MEM: Multi-Scale Embodied Memory for Vision Language Action Models ↗
PDF · 10,871 extracted words
Source fingerprint
213bbe173026561d1ca4b9aaa626e51991bcfb49d343516ed7fd6658709e5475