RT-1: Robotics Transformer for Real-World Control at Scale
1. Paper overview
In one sentence: RT-1 combines early language-conditioned visual compression with discrete action prediction to learn broad kitchen manipulation skills, trading limited motion novelty for practical feedback control. identityproblem-dataarchitectureaction-lossoverall-resultssimulation-transfersaycandata-ablationmodel-ablationevaluation-discrepancy
| At a glance | What to know |
|---|---|
| Research problem | Source description Can one policy learn many connected manipulation instructions while remaining fast enough for feedback control? RT-1 addresses both demonstration diversity and the latency of a high-capacity controller. Its learned target is an action, rather than a future scene. problem-dataarchitectureaction-losspolicy-training |
| Core mechanism | Source description A compact tokenizer and Transformer combine early language conditioning, visual token reduction and discretized outputs in one policy. architectureaction-loss |
| A key reported result | Unseen instruction generalization: 76% Reported success rate. Held-out combinations of previously seen skills and objects. Main text says 21 instructions; Appendix I and Table VII say 53. Gato 52%; BC-Z XL 43%; BC-Z 19%. A 24-percentage-point advantage over Gato, rather than the introduction’s 25% summary. This tests compositional reuse, not entirely new motions. overall-resultsevaluation-discrepancylimitations |
| Reading caution | Source description Authors acknowledge demonstration dependence, inability to generalize to entirely unseen motions, similar kitchen environments and limited dexterity. limitations |
Core contributions
- Source description
A compact tokenizer and Transformer combine early language conditioning, visual token reduction and discretized outputs in one policy. architectureaction-loss
- Source description
Approximately 130,000 demonstrations from 13 robots over 17 months cover 744 listed instructions. Instructions distinguish object–skill combinations, not 744 distinct motor primitives. problem-data
- Author claim
The authors argue that broad task coverage is especially valuable and that the policy can absorb simulation and other-robot experience; data ablations and separate transfer evaluations test these properties. data-ablationsimulation-transfercross-robot-transfer
Figure 2. Instruction-conditioned visual compression connects camera history to robot commands. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the blue instruction branch into Universal Sentence Encoder, then across into the FiLM layers of the image encoder. The multiplications by one plus gamma and additions of beta are language-conditioned affine modulation, consistent with the text’s identity initialization. The stacked image branch processes six observations. Each spatial feature grid has 81 tokens; TokenLearner combines them into eight per frame. Follow the downward arrows to the 48-token history, positional encoding and self-attention blocks. The final green tokens specify control mode, arm/gripper and base. The bottom photograph illustrates execution, not a predicted future image. architectureaction-losspolicy-trainingimplementation-gaps
What it supports. The architecture puts language conditioning before the visual bottleneck, allowing task-relevant information to reach the controller compactly. Its eight-layer Transformer operates on a short token sequence even though it receives six images. This explains the intended information flow and computational economy; measured control performance must be assessed in the experiments.
Where the evidence stops. The diagram does not specify an attention-mask matrix or a freezing policy. Decoder-only does not imply autoregressive action generation here: Appendix M explicitly removes that behavior from the final model. No learned world-state rollout is shown.
2. Motivation
2.1 The problem and the proposed response
Can one policy learn many connected manipulation instructions while remaining fast enough for feedback control? RT-1 addresses both demonstration diversity and the latency of a high-capacity controller. Its learned target is an action, rather than a future scene. problem-dataarchitectureaction-losspolicy-training
2.2 What this reading follows
A robot asked to pick an apple from a drawer must connect language to the right visual features, then issue usable commands quickly enough to respond to what happens. RT-1 addresses this with a compact Transformer policy trained on a large collection of demonstrations. The paper’s central evidence comes from physical execution, architectural ablations and changes to the training data. Read the six visuals as a sequence: follow information into the controller, establish what its success rates measure, examine transfer and composed long-horizon execution, then ask which data and model choices explain the results. The supplied RSS edition’s reporting inconsistencies remain explicit throughout. identityproblem-dataarchitectureaction-lossoverall-resultssimulation-transfersaycandata-ablationmodel-ablationevaluation-discrepancy
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | VLA |
| Architecture | One Model |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
Architecture supports the recorded discrete-action VLA and One Model policy classification: language-conditioned features feed one action-predicting Transformer. Modular pretrained encoders do not constitute a separate future predictor or inverse-dynamics model. World/action prediction quadrants are not applicable. SayCan’s external planner and affordance functions belong to the long-horizon system, not internal RT-1 world modeling. architecturepolicy-trainingaction-losssaycan
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Use the instruction before throwing visual information away
RT-1 makes the visual representation depend on the requested behavior before applying its strong compression. The instruction becomes a Universal Sentence Encoder embedding, which supplies the FiLM conditioning inside EfficientNet-B3. Figure 2 shows a scale of one plus gamma and an additive beta; initializing the conditioning projections to zero preserves the pretrained encoder initially. Each frame then supplies a spatial feature grid, but the Transformer receives only eight learned combinations from that grid. Across six frames, this produces 48 tokens. The ordering is important to understanding the proposal: compression receives features already conditioned on the task. The authors suggest this helps reject distractors, and visualize selected attention maps. Those maps illustrate the proposed explanation; they do not by themselves isolate early fusion as the cause of higher success. architectureattention-analysis
5.2 Follow a learned action through the real control loop
Behavioral cloning teaches the policy which action is likely in a demonstrated situation. It does not teach an explicit predicted next image or supply a search over imagined trajectories. The instruction and recent camera history determine discrete output bins for the arm, gripper, base and control mode. Appendix M clarifies that the final model avoids autoregressive action generation. The resulting commands still pass through a physical control stack: Cartesian tool moves become joint trajectories using inverse kinematics, constraints limit what executes, and base moves block policy execution until completion. Cached visual tokens reduce repeated computation, while the fixed waiting mechanism addresses timing consistency. Consequently, the reported 15 ms network time is not the whole response time. The 3 Hz system behavior depends on the surrounding sensing, communication and controller arrangement. policy-trainingaction-losscontrolmodel-ablation
5.3 Treat generalization as several experiments with different boundaries
Reader interpretation: RT-1’s generalization evidence is strongest when each test keeps its own denominator and source of novelty. Table II asks whether a policy can recombine familiar concepts or tolerate changed visuals; its main-text and appendix instruction counts remain inconsistent. Table IV introduces objects through simulation and measures physical transfer, distinguishing a familiar pairing from a new skill–object pairing. The cross-robot experiment additionally remaps actions and alters the target gripper’s appearance. SayCan evaluates yet another system, with planning and manipulation reported separately. Finally, the data ablation removes rare tasks rather than merely fewer random episodes. Together these experiments show several useful forms of reuse, but they do not establish arbitrary new motions or universal environment transfer. A reproduction should preserve these distinctions and publish the exact evaluated instructions and raw outcomes. overall-resultsevaluation-discrepancysimulation-transfercross-robot-transfersaycandata-ablationlimitations
5.4 Training and inference
During training
Behavioral cloning minimizes action negative log-likelihood on successful demonstrations using categorical cross-entropy. Binary episode success defines evaluation and the successful-data assumption; the core policy has no described online reward-optimization stage. policy-trainingaction-loss
Demonstrators use two VR remotes with matched action mappings. Software requests instructions and scene randomization. Checkpoint selection uses a 551-task simulator with RetinaGAN image translation to estimate policy rankings, rather than replace real-world evaluation. collection-selection
During inference
The policy reuses cached tokens from overlapping history windows. Final action components are not generated autoregressively, despite the main text’s general reference to causal masking. Network inference is reported as 15 ms; the system targets 3 Hz and uses a 280 ms capture-to-application waiting mechanism to reduce jitter. action-lossmodel-ablationcontrol
Tool/gripper control is nonblocking; base moves block until completion. Inverse kinematics and constraint checks mediate execution. Long instructions use SayCan language-model planning and separate MT-OPT affordance values, with RT-1 providing manipulation skills. controlsaycan
5.5 Implementation flow
- Condition perception before compression
Universal Sentence Encoder embeds the instruction and modulates EfficientNet-B3 through FiLM. Zero-initialized conditioning layers initially preserve the ImageNet-pretrained encoder’s behavior. Each image yields a 9×9×512 feature map. architecture
- Compress history and predict control
TokenLearner soft-selects combinations of 81 spatial tokens into eight per frame. The resulting 48 history tokens receive positional encodings and enter an eight-layer decoder-only Transformer. The system has approximately 35M parameters. architecturebaseline-design
- Represent and execute actions
Seven arm/gripper dimensions, three base dimensions and a mode variable form the output. Variables map to 256 uniform bins within their bounds. Low-level controllers convert commands to checked trajectories; token prediction and physical execution are distinct. action-losscontrol
6. Experiments & results
RT-1 learns a language-conditioned robot policy by compressing six camera observations into task-relevant tokens and predicting discretized actions. It joins a 35M-parameter architecture with a large, varied demonstration collection. Real-robot results support compositional instruction generalization and heterogeneous-data transfer within kitchen manipulation. Long-horizon results require SayCan, and evaluation-count inconsistencies limit exact protocol reconstruction.
6.1 Read the original evidence
Table II. RT-1 leads the same-data architecture comparisons, with background changes remaining difficult. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each column as a separate evaluation family, not successive stages of one episode. Seen instructions still vary in object placement; unseen instructions recombine familiar skills and objects. Distractors and backgrounds probe different visual shifts. All baselines are trained on the RT-1 data, and the Gato comparison uses a smaller model adapted to the control-latency constraint. Values are success percentages under the paper’s completion and failure rules. Compare RT-1 with the strongest baseline separately in each column: BC-Z for seen tasks, distractors and backgrounds, but Gato for unseen instructions. overall-resultsevaluationevaluation-discrepancybaseline-designimplementation-gaps
What it supports. The unseen-instruction column reports 76% for RT-1 and 52% for Gato, a 24-percentage-point difference. RT-1 also reaches 97% seen, 83% distractor and 59% background success. These are real-robot execution outcomes; the lower background score limits how broadly the headline result should be generalized.
Where the evidence stops. Section IV-A gives over 200 seen and 21 unseen instructions; Appendix I gives 744 and 53. Their relationship is unresolved. The introduction’s 25% unseen advantage differs from the table’s 24-point gap; no uncertainty intervals are supplied.
Table IV. Simulation adds useful object experience that transfers to real picking and new skill combinations. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the training-data column: the second row augments real demonstrations with successful simulated picking trajectories. All three outcome columns are measured in the real world. The left outcome is a control using objects present in real training. The middle uses objects introduced only in simulation, with a skill–object pairing seen there. The right is harder: the object was introduced through simulated picking, but the evaluated skill–object combination was absent from both training domains. Parentheses show percentage-point differences from real-only training. Appendix J identifies 518,000 successful simulated trajectories and restricts this test to picking and move-near skills. simulation-transferpolicy-trainingimplementation-gaps
What it supports. For objects absent from real training, success rises from 23% to 87% on seen pairings and from 7% to 33% on unseen pairings. That supports both domain transfer and some compositional reuse. The remaining 33% result also shows how much harder transfer becomes when the skill–object pairing changes.
Where the evidence stops. The control falls from 92% to 90%, so the caption’s claim of no impact is approximate, not literal equality. No uncertainty establishes equivalence. The added experience comes from a separate simulation/RL data-generation pipeline, not RT-1 predicting futures.
Table VI. Stronger manipulation improves execution while the SayCan planning rates remain equal. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each kitchen, separate Planning from Execution. The three reimplemented systems share SayCan planning and all report 87% planning success; their manipulation policies differ. Read the Execution columns to compare the effect of those policies. Appendix K defines 15 high-level instructions, averaging 9.6 steps and 2.4 manipulation skills, and identifies separate MT-OPT value functions for affordances. The retained footnote matters: Original SayCan used a different prompt, so its planning score is not a matched comparison. Kitchen2 probes a less familiar scene than the kitchen used to model the training classroom. saycan
What it supports. SayCan with RT-1 achieves 67% execution in each kitchen. BC-Z reaches 53% and 13%, and Gato reaches 33% and 0%. Holding the reported planning result constant makes execution quality the central comparison. This evidence concerns a composed planning-and-control system, rather than an autonomous long-horizon planner inside RT-1.
Where the evidence stops. The separately described up-to-50-step executions are not the denominator for this table. Do not interpret 67% as a measured success rate on 50-step tasks. Original SayCan’s missing Kitchen2 entries are unreported results, not zeros.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Seen instruction execution Real robot classroom with randomized placements. Main text specifies over 200 evaluated instructions; Appendix I says 744 and skill-weighted aggregation. | 97% Reported success rate | BC-Z 72%; Gato 65%; BC-Z XL 56%. Strong aggregate performance under human-rated completion and failure constraints; not a verified rate for every listed instruction. overall-resultsevaluationevaluation-discrepancybaseline-design |
| Unseen instruction generalization Held-out combinations of previously seen skills and objects. Main text says 21 instructions; Appendix I and Table VII say 53. | 76% Reported success rate | Gato 52%; BC-Z XL 43%; BC-Z 19%. A 24-percentage-point advantage over Gato, rather than the introduction’s 25% summary. This tests compositional reuse, not entirely new motions. overall-resultsevaluation-discrepancylimitations |
| Distractor and background robustness Real-robot tests with clutter/occlusion and changed countertop or kitchen appearance; same-data architecture comparisons. | 83% / 59% Success rate, distractors / backgrounds | Best baseline BC-Z: 47% / 41%. Gains are 36 and 18 percentage points, with substantial remaining background sensitivity. overall-resultsevaluationevaluation-discrepancybaseline-design |
| Simulation-to-real object transfer Add 518k successful simulated picking trajectories; evaluate picking and move-near skills physically on objects absent from real training. | 87% / 33% Success on seen / unseen skill–object combinations | Real-only: 23% / 7%; real-object control changes from 92% to 90%. Supports transfer and recombination while preserving the observed two-point control decrease. simulation-transfer |
| Transfer of Kuka bin-picking experience Mobile-manipulator evaluation with a matching bin and altered gripper appearance; appendix specifies 72 grasping trials. | 39% Bin-picking success rate | Mobile-manipulator-only: 22%; Kuka-only: 0%. Classroom control: 90% mixed versus 92% original. Mixed training helps this adapted setting; arbitrary morphology transfer without appearance adjustments remains untested. cross-robot-transfer |
| SayCan long-horizon execution 15 instructions in each of two kitchens; average 9.6 steps and 2.4 manipulation skills per instruction. | 67% / 67% Execution success, Kitchen1 / Kitchen2 | BC-Z: 53% / 13%; Gato: 33% / 0%. All three planning rates: 87%. The gain concerns execution within SayCan. Separately described 50-step demonstrations are not a 50-step benchmark success estimate. saycan |
| Data coverage ablation Remove rare tasks versus cap common-task examples; same RT-1 architecture. | Full data: 73%; 75% tasks / 97% data: 54%. Table XII aggregate generalization success | 100% tasks / 51% data: 50%. Removing little data can hurt when task coverage disappears. Task identity and frequency also change; this is not a universal scaling law. data-ablation |
| Action representation and latency ablation Table XIII: discrete outputs versus continuous Gaussian outputs with MSE, and autoregressive outputs. | RT-1: 97% / 76% / 83% / 59%; 15 ms. Seen / unseen / distractor / background success; network latency | Continuous: 68% / 43% / 37% / 35%; 16 ms. Autoregressive: 85% / 71% / 67% / 65%; 36 ms. Discrete prediction beats the tested Gaussian alternative. Autoregression worsens three axes and latency but improves background success. model-ablation |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table XII and accompanying plot. Rare-task coverage has substantial value even when it accounts for few episodes. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. In the upper block, task coverage stays at 100% while common tasks lose demonstrations through per-task caps. In the lower block, 25% of tasks disappear but 97% of episodes remain. Compare the aggregate Generalization All column before inspecting its unseen, distractor and background components. The plot uses blue for seen success, gray for generalization and partially filled markers for reduced task coverage. Its horizontal axis measures data quantity, not task coverage. The green and purple arrows are schematic trend annotations, not additional measured points or fitted laws; use the table and actual markers for exact values. data-ablation
What it supports. Full data gives 73% aggregate generalization. Removing rare tasks reduces it to 54% despite retaining 97% of episodes, while retaining every task with 51% of the data gives 50%. Thus small data removal can be costly when it removes task coverage, although the two reduced conditions do not have identical scores.
Where the evidence stops. Narrowing changes task identity and frequency as well as coverage; quantities are not matched. This supports a collection heuristic, not a universal causal scaling law. Prose gives 22.5% for the smallest dataset while the table displays 22%.
Table XIII and accompanying plot. Action representation, pretraining and history affect different parts of the performance–latency tradeoff. Original paper, p. 21 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use full RT-1 as the reference row. Parentheses and the lower bars show success-rate changes relative to it; these differences are percentage points despite the plot’s percent-sign labels. Keep the rightmost inference-time column separate from success. The continuous-action ablation uses a Gaussian output with MSE, whereas the default predicts discrete bins. Removing pretraining or history changes other components independently. Autoregressive actions increase inference from 15 to 36 ms and reduce seen, unseen and distractor scores, but the green background bar rises: background success improves from 59% to 65%. Network timing excludes the rest of the robot pipeline. model-ablationaction-lossimplementation-gaps
What it supports. The tested Gaussian alternative drops seen success from 97% to 68% and distractor success from 83% to 37%. Removing pretraining reduces unseen success to 43%; removing history reduces distractor success to 50%. These patterns support the design’s particular combination, without establishing that every design choice helps every evaluation axis.
Where the evidence stops. The no-Transformer row prints 86 (−13), although 86−97 equals −11; preserve this source error. Appendix M also conflicts on reduced-model sizes. No hardware specification or uncertainty accompanies latency/results, and Gaussian-vs-discrete changes both representation and loss.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors acknowledge demonstration dependence, inability to generalize to entirely unseen motions, similar kitchen environments and limited dexterity. limitations
Tables lack uncertainty estimates and evaluation counts remain unreconciled. Table III ties RT-1 with BC-Z XL at L2 and BC-Z at L3 despite stronger surrounding prose. Appendix M gives conflicting reduced-model sizes. Table XIII prints 86 (−13), although 86 minus 97 is −11. evaluation-discrepancyimplementation-gapsrealistic-resultsmodel-ablation
Attention highlights suggest explanations without causally testing language fusion. The continuous-action ablation changes output family and loss, so it does not rule out richer continuous policies. attention-analysismodel-ablation
7.2 Questions for discussion
- Would a matched continuous multimodal head close the gap to discretization?
- Does rare-task coverage remain beneficial with fixed episode budget, optimization steps and test composition?
- How much cross-robot transfer survives without changing the target gripper’s appearance?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the demonstrations, pretrained encoders, splits, randomization, action bounds and controller timing. The PDF omits complete optimizer settings, batch size, training duration, seeds, accelerator hardware, software versions and an explicit freezing policy. A code link does not establish that these gaps are resolved. architectureaction-losscontrolcollection-selectionimplementation-gaps
Cross-robot replication must preserve the 2:1 RT-1:Kuka ratio, action remapping, pick-anything relabeling and altered evaluation appearance; otherwise it tests another setting. cross-robot-transfer
Proposed checks compare discrete and Gaussian outputs under controlled training and measured latency, then compare broad and narrow coverage at equal episode budgets with fixed test splits. Record raw counts and uncertainty; no reproduction was performed. model-ablationdata-ablationevaluation-discrepancy
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Separate action expressiveness from inference-time cost
Reader-proposed experiment, not performed: keep the instruction/image encoder, six-frame input, demonstration split, optimization budget and robot trial scenes fixed. Compare the paper’s discrete head with its Gaussian/MSE alternative and an autoregressive discrete head. Measure both offline action likelihood or prediction error appropriate to each head and physical success on matched seen, held-out and distractor instructions. Record per-stage latency and apply the same controller timing to all policies; publish seeds, raw trial counts and confidence intervals. A persistent discrete-head advantage under matched control timing would support the reported representation effect. If the advantage disappears when timing or training budget is controlled, the original explanation would need revision. Gaussian failure alone should not be generalized to every continuous output family. architectureaction-lossmodel-ablationcontrolevaluationimplementation-gaps
Check 2: Test whether rare-task coverage helps at a fixed data budget
Reader-proposed experiment, not performed: build broad and narrow training subsets with equal episode counts and training steps. The broad subset samples across all available instruction families; the narrow subset excludes rare families and reallocates its episode budget to retained ones. Repeat the subset selection and training across seeds. Freeze a held-out compositional test list before either selection, and compute seen-task results only on instructions present in both subsets. Match kitchen scenes, object placements and distractor difficulty, reporting each category separately rather than only an aggregate. If broad coverage reliably improves held-out success at equal quantity, it strengthens the paper’s diversity interpretation. If the difference vanishes, Table XII’s original frequency and task-composition changes become plausible explanations. Resolve the source’s evaluation-count discrepancy before claiming an exact replication. data-ablationevaluationevaluation-discrepancyimplementation-gaps
8.3 Reading coverage
Visual audit: All 22 supplied PDF pages were rendered and visually inspected, including the title/author block, Figures 1–13, Tables I–XIII, appendix implementation details and reference pages. All six final original crops were separately opened and checked for legibility and complete relevant labels. Figure 2’s FiLM operations and direction of information flow were checked against Section III-B; the final non-autoregressive action behavior comes from Appendix M. Table XII’s arrows are treated as schematic, and Table XIII’s inconsistent no-Transformer delta is disclosed. The SayCan crop retains the original-prompt footnote. Supporting uncropped pages cover the control loop, evaluation discrepancies, transfer preparation, limitations and proposed-check premises. No separate supplemental files, linked sites or code were inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF pp. 1–2: title, abstract and I. Introduction
- PDF pp. 2–3: II. Related Work
- PDF pp. 3–5: III. RT-1, robot setup, model and data
- PDF pp. 5–9: IV. Experiments, A–E
- PDF pp. 9–10: V. Conclusions and Limitations
- PDF pp. 10–13: References
- PDF pp. 13–22: complete appendix, contributions and A–N, including controls, collection, model selection, baselines, evaluation and ablations
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The verified edition is the supplied 22-page RSS 2023 proceedings PDF. No numbered revision or revision history is supplied; differences from other editions cannot be established. The exact title and 51-author identity agree with the catalog. Observed forms include Alex Herzog, Nikhil J Joshi, Michael Ryoo, Pannag Sanketi and Huong Tran rather than the catalog’s expanded or punctuated forms; metadata preserves the title-page credits.
- The supplied extraction notes: Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This limitation was addressed by rendering and visually inspecting all 22 PDF pages and all six final original crops.
- Separate supplemental material availability has not been fully verified. No separate supplement was supplied or read.
- The linked code and project sites were not accessed; code was not inspected and no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, proceedings header, title and complete author/affiliation block
The title matches RT-1: Robotics Transformer for Real-World Control at Scale. The header identifies Robotics: Science and Systems 2023, Daegu, July 10–14, 2023. All 51 alphabetically listed authors and three affiliations are visible; several names use shorter forms than the catalog.
Go to primary source ↓problem-dataPDF pp. 2–5, Sections I and III-C; p. 4, Table I, Total row
Approximately 130,000 demonstrations were collected with 13 robots over 17 months. Table I totals 744 instructions, distinguished from verb-based skills. Section III-C describes later additions to the skill collection.
Go to primary source ↓architecturePDF p. 3, Section III-B and Figure 2; p. 4, TokenLearner and Transformer paragraphs
Six 300×300 RGB images pass through ImageNet-pretrained EfficientNet-B3, conditioned by Universal Sentence Encoder embeddings using identity-initialized FiLM. Each 9×9×512 feature map becomes 81 tokens, reduced to eight; 48 history tokens receive positional encoding before an eight-layer, 19M-parameter decoder-only Transformer. The visual tokenizer has approximately 16M parameters.
Go to primary source ↓action-lossPDF p. 4, Section III-B, Action tokenization, Loss and Inference speed; p. 20, Appendix M, final paragraph
Seven arm/gripper dimensions, three base dimensions and an arm/base/termination mode form the output. Variables are mapped to 256 uniform bins within their bounds. Categorical cross-entropy and causal masking are described; Appendix M explicitly says the final model does not generate actions autoregressively. TokenLearner and cached overlapping-window features support 3 Hz control.
Go to primary source ↓policy-trainingPDF pp. 13–14, Appendix A, Robot learning, Transformers and Imitation learning
The policy maps an instruction and image history to an action distribution and applies actions to the robot. Behavioral cloning minimizes negative log-likelihood on successful demonstrations. Binary episode success defines the task objective. No future-image or future-state prediction target is specified for RT-1.
Go to primary source ↓controlPDF p. 14, Appendix B, Model inference, and Appendix C, Low-level control details
A 280 ms state-capture-to-action waiting mechanism addresses jitter. Tool and gripper position controllers are nonblocking; base control is blocking. Cartesian tool trajectories use inverse kinematics and high-gain joint position control; execution stops before constraint violations. Base trajectories are similarly checked.
Go to primary source ↓collection-selectionPDF pp. 14–15, Appendices E–G and Figure 7
Operators use two VR remotes mapped to the policy action space, with software-specified instruction and scene randomization. A simulator supporting 551 tasks and RetinaGAN-transformed images supplies approximate checkpoint rankings for real-data policies. The authors claim useful ordering, not equality of real and simulated success.
Go to primary source ↓evaluationPDF pp. 5–6, Section IV-A, Success Metric, Seen task performance, Unseen tasks generalization and Robustness
Human-rated success requires completion without environmental collision, touching irrelevant objects, unsafe behavior or exceeding 100 actions, approximately 33 seconds. Main-text evaluations specify over 200 seen instructions, 21 unseen instructions, 30 distractor tasks and 22 background tasks.
Go to primary source ↓evaluation-discrepancyPDF p. 5, Section IV-A; pp. 15–17, Appendix I, Table VII and Figures 9–11
Appendix I describes 744 seen tasks with skill-weighted aggregation and 53 held-out instructions, listed in Table VII. It defines three distractor-test instructions and six background-test instructions with difficulty levels. These counts are not reconciled with the main text. Training distractors are described as 0–4 in text and 2–4 in Figure 11’s caption.
Go to primary source ↓baseline-designPDF p. 6, Section IV-B, opening paragraphs; p. 15, Appendix H
Comparisons retrain architectures on RT-1 data. Gato is reduced to 37M parameters for control latency, versus its original 1.2B configuration; RT-1 has 35M. BC-Z uses a ResNet, continuous actions and no temporal history; BC-Z XL increases its size.
Go to primary source ↓overall-resultsPDF p. 7, Table II, all rows and columns, and Section IV-B; p. 2, introduction summary
RT-1 reports success percentages 97/76/83/59 for seen/unseen/distractor/background evaluation. Gato gives 65/52/43/35, BC-Z 72/19/47/41 and BC-Z XL 56/43/23/35. The unseen gap to Gato is 24 percentage points, although the introduction says 25%.
Go to primary source ↓realistic-resultsPDF p. 7, Table III and Generalization to realistic instructions; p. 16, Appendix I, Realistic instructions
RT-1 scores 70 overall and 88/75/50 at L1/L2/L3. BC-Z XL scores 55 overall and 63/75/38; BC-Z reaches 50 at L3. RT-1 therefore ties a baseline at L2 and L3. Increasing levels add new kitchen appearance, distractors, then new objects or locations.
Go to primary source ↓simulation-transferPDF p. 8, Table IV and Section IV-C, Absorbing simulation data; p. 17, Appendix J and Table VIII
Adding 518k successful simulated picking trajectories yields real-world success of 90/87/33 for real objects with seen skills, sim-only training objects with seen skills, and sim-only training objects with unseen skill combinations. Real-only training gives 92/23/7. Evaluation focuses on picking and moving near objects; simulation trajectories are produced using the described real-to-sim and multitask-RL procedure.
Go to primary source ↓cross-robot-transferPDF pp. 8–9, Section IV-C and Table V; pp. 17–18, Appendix J, Figure 12 and Table IX
Mixing 209k successful Kuka QT-Opt episodes with mobile-manipulator data improves mobile-manipulator bin-picking success from 22% to 39%, while classroom success changes from 92% to 90%; Kuka-only gives zero in both. Kuka actions are remapped, instructions relabeled pick anything, and training uses a 2:1 RT-1:Kuka ratio. Evaluation uses a matching bin and modified mobile-gripper appearance; Appendix J specifies 72 grasping trials.
Go to primary source ↓saycanPDF p. 9, Table VI and Section IV-D; pp. 18–19, Appendix K, Tables X–XI
SayCan uses language-model decomposition and MT-OPT value functions for affordances, with RT-1 as manipulation policy. Fifteen instructions average 9.6 steps and 2.4 manipulation skills. RT-1 execution is 67% in both kitchens, versus BC-Z 53%/13% and Gato 33%/0%; all three planning rates are 87%. Original SayCan has a different prompt. Up-to-50-step execution is separately described without a dedicated aggregate table.
Go to primary source ↓data-ablationPDF p. 9, Section IV-E and Figure 6; pp. 19–20, Appendix L and Table XII
Retaining 75% of tasks and 97% of data gives 86 seen and 54 aggregate generalization, versus 97/73 at full data. Retaining all tasks but 51% of data gives 71/50. Quantity reductions cap common-task episodes at 200, 100 or 50; narrowing removes the least represented tasks. The smallest subset is 22.5% in prose and 22% in Table XII.
Go to primary source ↓model-ablationPDF pp. 20–21, Appendix M and Table XIII, full-model and ablation rows
RT-1 reports 97/76/83/59 and 15 ms inference. Continuous Gaussian actions with MSE give 68/43/37/35 and 16 ms. No pretraining gives 43 unseen; no history gives 50 distractor success. Autoregressive actions give 85/71/67/65 and 36 ms. Without Transformer, seen success is printed 86 with an inconsistent −13 relative to 97. Reduced size is inconsistently described as 35M→21M and 31M→25M.
Go to primary source ↓limitationsPDF p. 10, Section V, limitations paragraphs
Authors identify imitation-learning limitations, inability to generalize to entirely new motions, evaluation in kitchens still similar to training, and a broad but not highly dexterous manipulation set.
Go to primary source ↓attention-analysisPDF pp. 21–22, Appendix N and Figure 13
Selected attention examples highlight objects and drawers. Authors propose early language fusion and compact representations as explanations for robustness; these visualizations alone do not establish causal necessity.
Go to primary source ↓implementation-gapsPDF pp. 3–4, Section III-B; pp. 13–15, Appendices A–H; pp. 20–21, Appendix M and Table XIII
The supplied implementation descriptions specify architecture, cloning objective, controller timing and data procedures, but do not give a complete optimizer schedule, batch size, training duration, seeds, accelerator model/count, software versions, numeric bounds for every action variable or explicit module-freezing policy. Result tables give point estimates without confidence intervals or seed dispersion.
Go to primary source ↓8.5 Primary sources
RT-1: Robotics Transformer for Real-World Control at Scale ↗
PDF · 16,740 extracted words
Source fingerprint
e55066b074f0cbd6d1627397ba68e1a0b1d9ac54858a8c06983d03b52ac9fe67