Key-Gram: Extensible World Knowledge for Embodied Manipulation
1. Paper overview
In one sentence: Key-Gram gives a VLA policy addressable linguistic memory that improves compositional manipulation, while leaving parser reliability and the mechanics of protected knowledge expansion unresolved. e02e03e04e05e06e11e13e19e21
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that dense language–vision fusion makes sparse instruction tokens compete with visual tokens, while backbone adaptation can overwrite reusable task knowledge. They propose separating linguistic storage from online scene reasoning. Modality competition and interference are motivations, not directly measured causal effects here. e02 |
| Core mechanism | |
| A key reported result | LIBERO-to-LIBERO-Plus transfer: π0-KG 72.8; π0.5-KG 87.7. Reported success (%). Models trained on LIBERO; no target-domain fine-tuning for this column. π0 53.6; π0.5 83.9; OpenVLA-OFT 69.6. Gains over corresponding backbones are 19.2 and 3.8 percentage points; reported relative gains are 35.8% and 4.5%. e08e11 |
| Reading caution | Source description The authors acknowledge evaluation on only π0 and π0.5, reliance on parser quality, and insufficient evidence across broader architectures or longer continual-learning settings. e19 |
Core contributions
Figure 1. Instruction priors enter before attention; visual foresight and action decoding branch from shared backbone information. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start on the left with the instruction, which is decomposed into entries in external world knowledge. Its arrow enters the VLM alongside visual observations. The upward branch reaches a VAE, while the rightward KV-cache arrow supplies the action expert; noise enters that expert from below and actions leave above. On the right, follow the instruction upward through task phrases, hash heads, embeddings and concatenation. Two linear projections support the scaled-dot-product gate and value stream. Their product passes through convolution. The small Transformer diagram places this residual module before attention, matching Section 3.3 and Equations (3)–(4). e03e04e05e06e17e21
What it supports. The key design is where retrieved language affects computation: a visual token can receive a context-dependent amount of instruction memory before standard attention. The action expert uses the resulting backbone context. The diagram supports a modular backbone/expert architecture and a separate visual prediction pathway, rather than an action decoder consuming a rendered future frame.
Where the evidence stops. The parser logo does not identify an exact model/version. The graphic omits the σ operation shown in Equation (3), so that equation governs the gate formulation. No loss specification establishes how the future branch was trained or used during deployment.
2. Motivation
2.1 The problem and the proposed response
The authors argue that dense language–vision fusion makes sparse instruction tokens compete with visual tokens, while backbone adaptation can overwrite reusable task knowledge. They propose separating linguistic storage from online scene reasoning. Modality competition and interference are motivations, not directly measured causal effects here. e02
2.2 What this reading follows
A robot can see the same objects yet need different actions when an instruction changes their destinations. Key-Gram treats those linguistic relations as reusable priors stored outside the main visual computation. A parser compresses the instruction into task phrases, deterministic hashes retrieve embeddings, and selected Transformer layers decide how strongly to inject them. The paper tests this addition on π0 and π0.5, including simulated transfer and physical object recombination. Read the visuals as evidence for memory-conditioned control: the architecture also describes future-visual prediction, but the experiments do not separately establish its contribution or the long-term retention promised by extensible memory. e02e03e04e05e06e11e13e19e21
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | Dual-system |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Insufficient evidence to decide
VLA and Dual-system are supported by the memory-guided backbone and separate KV-conditioned action expert. However, Section 3.4 explicitly describes future-visual and action prediction, so 'Not applicable' cannot mean no predictive world representation. Its training and inference contribution are insufficiently specified for a firm WAM quadrant assignment. Fine-tuning is documented, but a distinct data-augmentation contribution is not established. Preserve the catalog snapshot while flagging these qualifications. e03e06e08e11e21
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 An instruction becomes an address, then a visual prior
Consider the paper's instruction about putting yellow and white mugs into a microwave and closing its door. Key-Gram first asks a language parser to produce reusable task phrases. Word identifiers, padding, deterministic odd multipliers and XOR turn each phrase into several memory addresses. The retrieved vectors are concatenated before projection, so the fusion module receives a combined instruction memory rather than independently attending over a list of textual documents. Visual hidden states then determine token-wise gate values from the projected memory key. Gated values pass through convolution and enter a residual connection before attention. This ordering matters: language can change the representations on which later scene reasoning operates. It also exposes a dependency: if parsing drops a destination or relation, deterministic retrieval cannot recover it by observing that the instruction was misunderstood. e03e04e05e17e19
Figure 2. Memory size can grow along instruction coverage and address capacity, but fusion must still accommodate retrieved entries. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel (a) indexes physical tables by key-gram slot s vertically and hash head h horizontally. The purple logical table aggregates their outputs; it should not be read as one dense semantic search over all rows. Panel (b) illustrates longer instructions activating more phrases and corresponding table groups. Panel (c) links greater task complexity to collision pressure and sketches additional capacity. Section 3.5 describes both enlarging row capacity and appending parallel tables whose retrieved outputs are concatenated. Follow the arrows as alternative memory-allocation operations, not as measured performance or timing curves. None of these panels presents an empirical scaling sweep. e07e05e18e21
What it supports. This separation explains how deterministic lookup can avoid a search whose cost grows with the entire memory. However, the real-world configuration does not simply add heads: it increases slots from eight to sixteen while reducing heads from four to two, preserving 1,024 retrieved dimensions and 32 tables per inserted layer.
Where the evidence stops. The diagram's expanded 2V×d table is schematic; the prose also describes output concatenation. Fusion resizing and protection of old entries are unspecified. Constant-time lookup in total table size is not a measured guarantee of constant robot-control latency.
5.2 Shared backbone context does not establish foresight-driven control
Trace the two exits from the memory-guided backbone separately. Section 3.4 describes learnable queries that extract features from final hidden states and produce an upsampled future visual latent. The action expert instead receives the backbone KV cache and performs flow-matching denoising over trajectories. Reader interpretation: this architecture can couple visual and action representations without requiring the action expert to consume the generated future image. The physical success tables demonstrate manipulation performance, but they do not isolate whether the visual prediction branch caused the improvement. Appendix A.5 reports full-parameter training, while leaving visual supervision, loss weighting and optimization details unstated. A reproduction therefore needs the missing objective before it can test the future branch fairly. It should not infer frozen backbone weights from statements that the base architecture remains unchanged. e03e06e12e13e18e20e21
5.3 Separate recombination, adaptation and memory growth
The expansion experiment asks several different questions. Familiar object pairs test execution on trained combinations; unseen pairings test whether object and destination relations recombine; the final retest asks whether the first pairing still works after training on the second. Key-Gram improves the unseen pairings substantially and improves that retest, yet the experiment is shorter than a sustained continual-learning evaluation. Reader interpretation: these results motivate extensible memory without demonstrating all of its promised protection mechanisms. Figure 2 explains adding slots or capacity, but Section 3.3 projects a concatenated memory vector whose width depends on the retrieved entries. Growth may therefore require a changed fusion interface even if the main visual architecture stays fixed. The paper does not give the exact resizing or old-entry update procedure. Recombination success, sequential retention and structural memory expansion should remain separate claims. e05e07e09e13e19e21
5.4 Training and inference
During training
Appendix A.5 explicitly reports full-parameter training. Statements that the backbone and expert are 'unchanged' describe preserved architecture and do not establish frozen weights. The source gives no complete loss formulation, visual/action loss weighting, optimizer, learning-rate schedule, or precise stage schedule. e08e18e20e21
Real-world data comprise 300 trajectories for each long-horizon task and two 50-trajectory expansion training sets. Sequential adaptation further trains on the second pairing before retesting the first. e09
During inference
Parse the instruction, retrieve fixed-address entries, fuse them into visual computation, and decode actions from KV-conditioned denoising. The source describes CPU-resident memory as feasible because lookup is constant-time in total table size; it reports no end-to-end inference latency. e04e05e06e07e21
Figure 1 does not feed a decoded future image into the action expert; both pathways share backbone information. Action horizon, denoising step count, feedback frequency, and the deployed role of the future-visual output remain unspecified. e03e06e21
5.5 Implementation flow
- Extract reusable task phrases
A lightweight language model or API parser produces bounded-length key-grams, rather than contiguous n-grams. The RoboTwin prompt requests eight phrases of two to four words, with at least three action-bearing phrases. The parser model/version is unspecified. e04e17
- Address embedding memory
Word identifiers are zero-padded and hashed with deterministic odd multipliers and XOR. Each head retrieves an embedding; head and phrase outputs are concatenated. These are stored priors, not records of the robot's recent visual history. e04e07
- Fuse before attention
The concatenated memory is projected into a key and value. Incoming visual hidden states query the key through a scaled dot product and gate. A long-span convolution refines gated values before residual addition. Selected zero-based layers are (1, 8, 13). e05e18
- Produce two downstream predictions
Learnable queries cross-attend to final hidden states, followed by projection and two upsampling stages into VAE-compatible future latents. Separately, backbone KV cache conditions a smaller, architecturally isomorphic action expert that denoises trajectories by flow matching. e06
- Allocate memory by instruction regime
Simulation uses eight slots and four heads per slot; physical experiments use sixteen slots and two heads. Both use 32-dimensional head embeddings, 8,192 rows per table, 1,024 retrieved dimensions, and 8.39M embedding parameters per inserted layer. Convolution spans are eight and sixteen respectively. e18
6. Experiments & results
Key-Gram augments π0 and π0.5 with instruction-indexed embedding memory. A parser extracts reusable phrases; hashed lookup retrieves priors that modulate visual hidden states before attention. The backbone supplies future-visual representations and context for a separate action expert. Reported gains are strongest under distribution expansion and compositional manipulation, but the evidence does not isolate future prediction or establish lifelong, update-free knowledge retention.
6.1 Read the original evidence
Table 2. The transfer column gives the strongest separation from π0; in-domain gains are smaller and not universal. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down a single column before comparing across columns. The first column evaluates LIBERO fine-tuning, the middle transfers LIBERO-trained models directly to LIBERO-Plus, and the last permits fine-tuning on LIBERO-Plus itself. Those are different training/evaluation conditions, not stages of one reported rollout. Within a column, pair each KG row with its corresponding backbone above the dividing rule. Parenthesized gains are relative improvements, not percentage-point differences. OpenVLA-OFT is another reported comparator, but the KG-versus-matching-backbone comparisons most directly address the proposed augmentation. The paper reports four standard LIBERO suites in its setup; this table provides aggregate values. e08e11
What it supports. In direct transfer, π0 rises from 53.6 to 72.8 and π0.5 from 83.9 to 87.7: 19.2 and 3.8 percentage points. The same table prevents a blanket improvement claim: LIBERO fine-tuning changes π0.5 from 96.9 to 96.7, even though its LIBERO-Plus results improve.
Where the evidence stops. No error bars or repeated-seed uncertainty are supplied. The 72.8 transfer result must not be presented as the 88.5 target-fine-tuned result, and the reported 35.8% relative gain is not a 35.8-point increase.
Table 4. Unseen object pairings account for the clearest physical generalization gains. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Translate the task numbers using the experimental protocol on page 7. Tasks 1 and 2 use familiar Pen–Charger and Matchbox car–Lighter pairings. Task 3 recombines Pen with Lighter; Task 4 recombines Charger with Matchbox car. Task 5 returns to Pen–Charger after further training on the other familiar pair. Compare columns 3 and 4 to assess compositional transfer, and column 5 to assess the single sequential retest. The Avg. column combines these distinct settings, so it cannot by itself identify improved retention. Parenthesized percentages again express relative gains over each matching base policy. e09e13e19e21
What it supports. For π0, unseen-pair success rises from 52 to 70 and from 48 to 68. For π0.5, it rises from 64 to 76 and from 66 to 80. The sequential retest also improves, to 88 versus 80 and 90 versus 86, but it probes only one update-and-retest sequence.
Where the evidence stops. The source gives two 50-trajectory training sets but does not state real-world evaluation trial counts or uncertainty. Do not treat those training counts as trial denominators. Neither this table nor selected Figure 4 rollouts establishes indefinite protection against forgetting.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin2.0 manipulation 50 tasks; Easy and Hard settings; 100 configurations with 100 trials each in unseen environments. | π0-KG: 80.3 Easy, 75.6 Hard; π0.5-KG: 89.0 Easy, 84.4 Hard. Average success rate (%) | π0: 65.9/58.4; π0.5: 82.7/76.8 (Easy/Hard). Reported 29.5%/9.9% relative gains are Hard-setting gains, not averages across both settings. Individual tasks can regress. e08e10e16 |
| LIBERO-to-LIBERO-Plus transfer Models trained on LIBERO; no target-domain fine-tuning for this column. | π0-KG 72.8; π0.5-KG 87.7. Reported success (%) | π0 53.6; π0.5 83.9; OpenVLA-OFT 69.6. Gains over corresponding backbones are 19.2 and 3.8 percentage points; reported relative gains are 35.8% and 4.5%. e08e11 |
| LIBERO and LIBERO-Plus fine-tuning Separate in-domain fine-tuned columns, not the transfer protocol. | LIBERO: 94.6/96.7; LIBERO-Plus: 88.5/92.6 for π0-KG/π0.5-KG. Reported success (%) | Corresponding backbones: 94.2/96.9 and 84.0/90.4. Transfer gains coexist with a 0.2-point LIBERO decline for π0.5-KG; improvement is not universal. e11 |
| Physical long-horizon manipulation Piper dual-arm platform; picking, sorting, assembly. | π0-KG 80.0; π0.5-KG 88.7. Mean reported success (%) | π0 69.3; π0.5 82.0. Assembly: 52→66 and 70→82. These are executed manipulation outcomes. The table reports relative mean gains of 15.4%/8.1%, without uncertainty estimates. e09e12 |
| Physical compositional expansion and sequential adaptation Task 3: unseen Pen–Lighter; Task 4: unseen Charger–Matchbox car; Task 5: retest Pen–Charger after further training. | π0-KG Tasks 3/4/5: 70/68/88; π0.5-KG: 76/80/90. Reported success (%) | π0: 52/48/80; π0.5: 64/66/86. The five-setting means are 81.6 versus 72.4 and 86.8 versus 80.0. One sequential retest cannot establish long-term forgetting resistance. e09e13e19 |
| Layer-placement ablation π0.5; four named RoboTwin tasks; Easy:Hard weighted 1:9, then averaged across tasks. | Vanilla 51.0; layer (1) 71.5; (1,8) 76.8; (1,8,13) 77.2. Weighted score (%) | Adding layer 13 after (1,8) gives 0.4 points at displayed precision. Early insertion carries most of this gain. Added layers also add parameters, so this is not a fixed-capacity isolation of gating or convolution. e14e18 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 5. The greedy sweep favors an early insertion, then more widely separated additions. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the panels as successive searches: begin without a selected memory layer, then condition the next sweep on layer 1, then on layers 1 and 8. The horizontal coordinate is the probed insertion layer. Colored circles and shaded curves use the left score axis; red diamonds use the right normalized-gate axis. Shading marks the score curve, not an uncertainty interval. The score ranges differ markedly across panels, so equal visual heights do not mean equal improvement. The selected peaks correspond to the layer sequence summarized in Table 5. Treat the red traces as diagnostic activity measurements rather than direct causal estimates of knowledge use. e14e15
What it supports. The score curves peak at layer 1 initially, then around layer 8 and layer 13 in later searches. This supports the paper's placement choice on its four-task weighted evaluation. Nearby probed layers have suppressed gates after an earlier insertion, consistent with—but not proof of—the authors' redundancy interpretation.
Where the evidence stops. Page 9 describes a gate floor around 0.1, while the vanilla panel's deep-layer diamonds sit near 0.5. Do not report 0.1 as that plotted plateau. Gate normalization is unspecified, and score axes differ between panels.
Table 5. One early memory insertion supplies most of the measured placement gain. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the rightmost Score column: 51.0 for vanilla, 71.5 for layer 1, 76.8 for layers 1 and 8, and 77.2 for layers 1, 8 and 13. These are not the all-task RoboTwin averages in Table 1. The caption defines an average over four tasks with Easy:Hard weighted 1:9, so the Hard columns dominate. Then inspect individual tasks to see why the aggregate can hide tradeoffs: adding layer 13 changes hanging-mug Hard from 36 to 35 even as pick-dual-bottles Hard rises from 95 to 98. Appendix A.3 confirms zero-based layer indexing. e14e16e18
What it supports. Relative to vanilla, the first insertion adds 20.5 percentage points to this weighted score. Adding layer 8 contributes another 5.3; adding layer 13 contributes 0.4 at the displayed precision. The result favors early access to language memory, while showing diminishing marginal benefit from the greedy additions.
Where the evidence stops. Each added module brings additional embedding parameters, so layer count and capacity change together. No uncertainty is reported for the small final gain. Table 6 contains different per-task values; preserve this table's own ablation protocol and numbers.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge evaluation on only π0 and π0.5, reliance on parser quality, and insufficient evidence across broader architectures or longer continual-learning settings. e19
No controlled future-head, parser, hash-capacity, or convolution ablation is reported. No quantitative future-latent metric establishes dynamics quality; gate activations alone do not prove reduced modality competition. e06e14e15e21
Source inconsistencies remain: the prose's near-0.1 vanilla gate floor differs from Figure 5's near-0.5 deep-layer markers. Table 1 reports 78.0 for π0-KG H=1 Hard, versus 77.9 in Table 6; ablation and appendix per-task rows also differ. e15e16
Equation (1) requires a prime modulus, whereas the configuration lists 8,192 rows. Their relationship is unexplained. Expansion by concatenation also changes fusion input width; the resizing and old-entry protection procedure is not specified. e04e05e07e18
7.2 Questions for discussion
- Does the gain persist with parameter-matched late injection or instruction-independent memory?
- Can old-task behavior be preserved when new key-gram slots change the fusion interface?
- Does the future-visual branch improve executed control when its contribution is isolated?
8. Reproducibility audit
8.1 Requirements and known gaps
Reported compute is eight NVIDIA H200 GPUs, batch 16 per GPU/global 128, about 135 GB per GPU, roughly 128 samples/s, and 1–1.5 days per RoboTwin epoch. These are author-reported requirements, not measured here. e20
A faithful implementation needs parser/version and caching policy, word-ID mapping, hash seeds/moduli, memory initialization/update rules, convolution layout, backbone checkpoints, visual targets/losses, optimizer schedule, and control timing. Several are absent; real-world evaluation trial counts, random seeds, and uncertainty estimates are also unspecified. e04e05e06e09e17e18e21
Proposed checks should hold capacity constant while changing memory injection depth, and separately compare sequential retention with protected versus updateable memory. The illustrated edition specifies controls and falsifiable outcomes; neither check has been run. e07e09e14e18e19
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test insertion depth at matched memory capacity
Reader-proposed experiment, not run: use π0.5 on the same four RoboTwin ablation tasks and the stated 1:9 Easy:Hard weighting. Compare one module at layer 1 with one at layer 8 and one at layer 13, holding slot/head configuration, embedding count, demonstrations, optimizer schedule and evaluation seeds fixed. Add a control with instruction-independent retrieved values while preserving module shape. Report task-level success and uncertainty as well as the weighted score and gate traces. An early-layer advantage that persists at matched capacity would support the placement claim; comparable performance from constant memory would weaken the interpretation that retrieved linguistic content is responsible. Obtain the missing training recipe before treating any result as a faithful reproduction. e04e05e14e15e18e21
Check 2: Test whether protected memory preserves behavior through expansion
Reader-proposed experiment, not run: start from the Pen–Charger training set, adapt on Matchbox car–Lighter, and evaluate the original pair plus both unseen recombinations before and after adaptation. Compare protected old memory with append-only capacity, updateable memory of the same final size, and the matching base policy. Keep demonstrations, backbone update budget and final fusion width matched; explicitly record any projection resizing and hash collisions. Measure old-task success change alongside new-task success over repeated seeds. If protected entries remain identical while old-task behavior still deteriorates, preserving storage alone does not establish protected control knowledge. Longer sequences would then test whether the single retest in Table 4 generalizes. e02e04e05e07e09e13e18e19
8.3 Reading coverage
Visual audit: Inspected the title/authors/version on page 1, motivation on pages 1–2, architecture on page 3, equations and prediction pathways on pages 4–5, allocation on page 6, all main result/ablation tables and figures on pages 7–10, the full per-task table on pages 13–14, parser/configuration text on pages 14–15, and configuration, limitations and compute on page 16. All six final original crops were separately viewed. Figure 1 arrows and pre-attention residual placement were checked against Sections 3.3–3.4; Equation (3)'s σ is not drawn in the graphic. Figure 5's gate traces were checked against its caption and page 9 prose, with the floor/plateau discrepancy retained. Tables 1/5/6 were not silently reconciled. Some prompt lines on page 14 overflow the source page edge. Reference-only pages 11–12 were read in text, not visually inspected. No separate supplement or code was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 13, 14, 15, 16. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work, including 2.1–2.3
- 3 Method, including 3.1–3.5 and Equations (1)–(4)
- 4 Experiment, including 4.1–4.4
- 5 Conclusion
- References
- A.1 Full RoboTwin2.0 Results
- A.2 Key-Gram Extraction Prompt
- A.3 Model Configuration Details
- A.4 Limitations and Future work
- A.5 Computational Resources
Outside the original text pass
- Identity: the inspected title page identifies arXiv:2605.18556v1 [cs.RO], 18 May 2026. Title and all four authors match the catalog; no other revision was supplied or compared.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- That extraction limitation was addressed by inspecting all five figures, all eight tables, and every page supporting retained scientific details.
- Separate supplemental material availability has not been fully verified.
- All five supplied text chunks were read completely, including references and appendices. Reference-only pages 11–12 were read as text but were not visually inspected.
- Some long prompt lines on PDF p. 14 extend beyond the right page edge; their supplied extracted text was read. No missing visual text was reconstructed.
- Code and external resources were not inspected; experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title/author block and arXiv margin
Exact title; Jingjing Fan, Siyuan Li, Botao Ren, Zhidong Deng; Tsinghua departments; arXiv:2605.18556v1, 18 May 2026. Fan and Li share equal contribution.
Go to primary source ↓e02PDF pp. 1–2, Section 1
Motivates functional separation of reusable linguistic knowledge from dynamic scene reasoning and claims reduced interference.
Go to primary source ↓e03PDF p. 3, Figure 1 and caption; p. 4, Section 3.1
Memory-guided VLM sends KV context to a distinct action expert; a separate VAE-compatible visual pathway is described. Key-Gram precedes attention.
Go to primary source ↓e04PDF pp. 4–5, Section 3.2, Equations (1)–(2)
Parser extraction, padded word IDs, odd-multiplier XOR hashing, prime modulus, embedding lookup and head concatenation.
Go to primary source ↓e05PDF p. 5, Section 3.3, Equations (3)–(4) and residual definition
Concatenated memory projects to a key/value; visual queries provide token-wise gates; convolution precedes residual addition.
Go to primary source ↓e06PDF p. 5, Section 3.4, Vision reasoning and Action inference
Learnable visual queries, cross-attention and two upsampling stages; KV-conditioned, smaller isomorphic action expert uses flow-matching denoising.
Go to primary source ↓e07PDF p. 6, Figure 2 and Section 3.5
Slot–head physical tables form logical memory; expansion through slots, row capacity or parallel tables; device partitioning and CPU lookup are described.
Go to primary source ↓e08PDF pp. 6–7, Section 4.1, Benchmarks and Model Configuration
RoboTwin uses all 50 tasks, 100 clean/randomized configurations and 100 trials each; LIBERO uses four suites; transfer and target fine-tuning are separate protocols.
Go to primary source ↓e09PDF p. 7, Section 4.1, Real-world robot
Piper platform; 300 trajectories per long-horizon task; two 50-trajectory pairing datasets; unseen pairings and sequential retesting protocol.
Go to primary source ↓e10PDF p. 7, Table 1, Avg. 50 Tasks row, corresponding backbone/KG Easy and Hard columns
π0 65.9/58.4 versus KG 80.3/75.6; π0.5 82.7/76.8 versus KG 89.0/84.4. Parentheses denote relative improvements.
Go to primary source ↓e11PDF p. 7, Table 2, all rows and three protocol columns
LIBERO, LIBERO-to-Plus transfer, and Plus fine-tuning values; includes π0.5-KG's 96.7 versus 96.9 LIBERO result.
Go to primary source ↓e12PDF p. 8, Table 3, all task columns and Avg.; Figure 3
Long-horizon averages 69.3/82.0 for base models and 80.0/88.7 for KG; assembly 52/70 versus 66/82; selected physical rollouts.
Go to primary source ↓e13PDF p. 9, Table 4, Tasks 1–5 and Avg.; Figure 4
Expansion results distinguish familiar pairs, unseen recombinations and sequential adaptation. KG means 81.6/86.8 versus base 72.4/80.0.
Go to primary source ↓e14PDF pp. 8–9, Section 4.4; p. 10, Table 5 and caption
Four tasks: hanging mug, move stapler pad, pick dual bottles, stack blocks three. Easy:Hard 1:9 weighting; scores 51.0, 71.5, 76.8, 77.2.
Go to primary source ↓e15PDF p. 9, Section 4.4, gate-probing paragraph; p. 10, Figure 5 and caption
Colored/shaded curves encode score; red diamonds encode normalized gates on a separate axis. Vanilla deep-layer markers sit near 0.5 despite prose referring to a floor around 0.1; panel score axes differ.
Go to primary source ↓e16PDF p. 7, Table 1, Avg. H=1 Hard; p. 10, Table 5; pp. 13–14, Table 6
Table 6 reports 77.9 versus Table 1's 78.0 for π0-KG H=1 Hard. For π0.5-KG hanging mug Hard, Table 5 gives 35 versus Table 6's 36; tables are retained as distinct reports. Per-task regressions are present.
Go to primary source ↓e17PDF pp. 14–15, Appendix A.2, introductory paragraph and extraction prompt
RoboTwin parser prompt requests exactly eight phrases, two to four words each and at least three with action verbs. Some long PDF lines overflow the page.
Go to primary source ↓e18PDF pp. 15–16, Appendix A.3 and Tables 7–8
Simulation S=8,H=4; physical S=16,H=2; head width 32, memory width 1024, rows 8192, 32 tables and 8.39M embedding parameters per layer; spans 8/16; zero-based layers (1,8,13).
Go to primary source ↓e19PDF p. 16, Appendix A.4
Authors limit generality to two tested backbones, acknowledge external-parser dependence, and call for larger suites and longer continual-learning studies.
Go to primary source ↓e20PDF p. 16, Appendix A.5
Full-parameter training uses eight H200 GPUs; batch 16 per GPU/128 global, approximately 135 GB per GPU, 128 samples/s and 1–1.5 days per RoboTwin epoch.
Go to primary source ↓e21PDF pp. 4–7, Sections 3–4.1; pp. 8–10, Sections 4.3–4.4; pp. 14–16, Appendices A.2–A.5
Available method, experiment and configuration descriptions do not specify a complete training objective, visual loss/targets, optimizer schedule, parser identity or deployment timing; no dedicated future-prediction evaluation or component ablation beyond placement is reported.
Go to primary source ↓8.5 Primary sources
Key-Gram: Extensible World Knowledge for Embodied Manipulation ↗
PDF · 7,783 extracted words
Source fingerprint
a0319207ab18de8d8d615a4e7df9231d3cdaa0abb9986ed33697098644edbc8a