Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation
1. Paper overview
In one sentence: GR-1 uses a shared transformer to transfer human-video forecasting into robot action learning, with strong task-chain gains but incomplete evidence about which aspects of prediction cause those gains. overviewformulationencodersablationcalvin-resultsreal-results
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot demonstrations are costly, whereas human interaction videos are abundant but lack robot states and action labels. GR-1 asks whether a language-conditioned video prediction model can transfer directly to multi-task robot imitation through a compatible token interface. overviewformulation |
| Core mechanism | Source description The same GPT-style transformer supports video-only pretraining and joint future-image/action prediction during robot fine-tuning, with modality-specific encoders and output heads. overviewencoderstoken-mask |
| A key reported result | CALVIN unseen-scene generalization: 85.4%; 40.1%; 3.06 First-task success; five-task success; average completed chain length. ABC→D: train on A/B/C and evaluate unseen environment D under the same chain protocol. RT-1 has the best baseline first-task success, 53.3%; MT-R3M has the best baseline average length, 0.93. Different baselines lead different columns. This tests scene transfer within CALVIN's task family. calvin-resultscalvin-splitscalvin-protocol |
| Reading caution | Reader analysis No seed variation or uncertainty is reported for the main quantitative comparisons. Real-robot evaluation denominators and precise success thresholds are absent, limiting statistical interpretation. calvin-resultsreal-resultsreal-setup |
Core contributions
- Source description
The same GPT-style transformer supports video-only pretraining and joint future-image/action prediction during robot fine-tuning, with modality-specific encoders and output heads. overviewencoderstoken-mask
- Author claim
The authors attribute improved scene and language generalization to large-scale video pretraining and retained pretrained representations. The experiments support gains, but these causal explanations remain hypotheses. calvin-resultslanguage-evalablation
Figure 1. A compatible token interface carries video pretraining into robot fine-tuning. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the legend: red, blue and green blocks are language, image and robot-state inputs; striped blocks are learned prediction tokens. On the left, language and video frames enter GR-1 and the purple output represents a predicted image. The central Copy Weights arrow carries this model into robot fine-tuning. On the right, state inputs and action tokens join the sequence, and the model has both image and action outputs. The CALVIN and real-robot panels show application settings. Read the Deploy arrows as use of the resulting policy; this overview does not depict a search over imagined trajectories. overviewformulationtoken-maskobjectives
What it supports. The architecture permits transfer before robot action labels are available. Section 3.3 specifies video-only pretraining followed by robot fine-tuning with both prediction and imitation losses. The useful continuity is the shared sequence model; state and action interfaces extend what it can learn from robot trajectories.
Where the evidence stops. The graphic is not an attention diagram. Section 3.2.2 specifies that [OBS] and [ACT] positions cannot be attended to by other tokens; the stripes identify learned queries, not generated frames fed back into control.
2. Motivation
2.1 The problem and the proposed response
Robot demonstrations are costly, whereas human interaction videos are abundant but lack robot states and action labels. GR-1 asks whether a language-conditioned video prediction model can transfer directly to multi-task robot imitation through a compatible token interface. overviewformulation
2.2 What this reading follows
A human video contains observations and language, but no robot action labels. GR-1 bridges that gap by keeping the sequence model compatible across two training stages: first predict future images, then add robot state and action interfaces while continuing image prediction. The important reading question is how this shared representation helps an executed policy. Follow the two architecture figures to separate inputs, prediction queries and output heads; then read the simulation and physical-robot tables under their different protocols. The appendix ablations help distinguish the contribution of pretraining from that of auxiliary image supervision, while also exposing an untested combination. overviewformulationencodersablationcalvin-resultsreal-results
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The shared causal transformer with [OBS] and [ACT] outputs supports One Model × Joint prediction. Separate encoders and heads do not form a video-planner-plus-controller pipeline. Joint prediction here means shared conditional image/action learning, without an explicit action-conditioned simulator or runtime planning claim. formulationencoderstoken-maskobjectives
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Transfer a forecasting task before action labels exist
The pretraining problem is deliberately compatible with robot imitation. A language description and a history of human-video frames suffice to define the future-image target, so no robot action annotation is needed at this stage. GR-1 first learns this task on Ego4D clips while retaining frozen language and image encoders. Fine-tuning then inserts robot-state and action-query tokens into the sequence and adds action losses while preserving the image objective. The visual decoder continues receiving observation-query outputs from the same causal transformer. Notice what transfers: parameters trained to relate language and temporal visual context, rather than an action vocabulary copied from human motion. The actual mapping from robot observations to arm and gripper commands is learned from robot demonstrations. This is the architectural rationale for using video data despite its different embodiment. formulationencoderstoken-maskobjectivespretrain-data
Figure 2. Separate output heads learn from a shared transformer history. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the upward arrows on the left: CLIP encodes the instruction, pose and gripper status pass through projections, and ViT supplies visual tokens. Its global CLS feature bypasses the perceiver, while patch features pass through the resampler. Snowflakes mark the frozen text and image encoders. Across the dashed divider, follow the output paths upward from GR-1: observation-query hidden states join positional mask tokens in the vision decoder, while the action-query hidden state enters an MLP that branches into arm and gripper outputs. The two GR-1 bars are a layout device showing inputs and outputs of the shared model. encoderstoken-maskobjectivesimplementation
What it supports. The image head reconstructs patches, whereas the action head predicts control commands. Their losses train a common backbone, which supports the catalog's joint-prediction classification. Appendix A.1 further specifies a three-layer action MLP and that the image target is normalized patch-wise before reconstruction loss is computed.
Where the evidence stops. No arrow here carries a decoded future image into the action MLP. Treat a useful predictive representation as the proposed training mechanism; this diagram does not demonstrate inference-time planning or establish that image quality determines action quality.
5.2 Separate predictive supervision from online planning
A reader interpretation of the information flow is that GR-1 benefits from learning two compatible targets from a shared observed history. The attention rule matters: [OBS] and [ACT] tokens are prediction queries, and the network masks them as positions that other tokens could attend to. A decoded future image is therefore not specified as an intermediate input to an inverse-dynamics controller. Equation (2) instead maps the same language, image and state history directly to a future observation and the current action. During training, the image and action losses shape shared parameters; at deployment, the action head supplies the policy output. This supports joint prediction without establishing model-predictive control. The component ablation motivates the representation-learning explanation, but cannot prove it uniquely because a pretrained model fine-tuned without the image loss is absent. formulationtoken-maskobjectivesablation
5.3 Connect the headline to the actual evaluation unit
Start evaluation with the unit being counted. CALVIN gives a policy an instruction chain and advances only after the current task succeeds. First-task success, completion of all five tasks and average completed length therefore answer different questions. The unseen-scene split changes the environment while keeping the task family; the paraphrase test changes wording rather than adding skills. Physical experiments ask a different question again: whether actions transport particular objects or manipulate a drawer under the stated scene conditions. As a reader, use the component ablations to investigate the training mechanism, and the horizon ablation to assess one design choice within its no-pretraining setting. Finally, inspect failure cases alongside averages. Table 7's uneven task performance and Table 2's category-transfer drop prevent the strongest aggregate results from being read as uniform manipulation competence. calvin-protocolcalvin-splitslanguage-evalreal-setupreal-resultsablationhorizontask-diagnostic
5.4 Training and inference
During training
Ego4D pretraining uses 800,000 three-second clips containing 8 million frames. Frames are sampled one-third of a second apart; prediction targets the next sampled frame. Robot fine-tuning retains image prediction and adds behavior cloning. CLIP and MAE remain frozen in both phases. pretrain-dataimplementationobjectives
The backbone has 12 layers, 12 heads and hidden size 384; the complete system has 195M parameters, 46M trainable. Input sequences have length 10. Standard robot fine-tuning predicts three timesteps ahead for static and gripper cameras, with patch-normalized image targets and random-shift augmentation. implementation
During inference
At the current timestep, language, observed images and measured state history feed the action head. The stated formulation directly predicts the current action alongside a future image. It specifies no inverse-dynamics stage, action-candidate search or model-predictive planning loop; prediction losses supply training supervision rather than an explicit runtime planner. formulationtoken-maskobjectives
5.5 Implementation flow
- Encode and compress the history
Frozen CLIP encodes language; a frozen MAE-pretrained ViT supplies a global CLS token and local patch tokens. A trainable perceiver resampler reduces local tokens. Robot pose and gripper status are encoded separately, and projections align embedding dimensions. encodersobjectives
- Use prediction queries within one causal transformer
Each robot timestep orders language, state, image, [OBS], then [ACT] tokens. Language is repeated at every timestep; learned temporal embeddings are shared across modalities at that timestep. Tokens can attend to earlier positions except [OBS] and [ACT]. These prediction queries therefore do not become an attended history of generated outputs. token-mask
- Decode two targets
[OBS] hidden states and positional mask tokens enter a vision decoder that reconstructs future-image patches. The action head maps [ACT] output to arm and gripper predictions. Appendix A.1 specifies a three-layer action MLP with two final heads. CALVIN actions are delta XYZ, delta Euler angles and a binary gripper command. encodersobjectivesimplementationcalvin-splits
6. Experiments & results
GR-1 transfers language-conditioned future-frame prediction on human videos into a robot policy. A shared causal transformer predicts images and actions after robot fine-tuning. The experiments show better CALVIN task-chain completion and real-robot manipulation than the tested baselines, while ablations support contributions from both video supervision and pretraining. These are executed-policy results; generated images alone do not establish control success.
6.1 Read the original evidence
Table 1. Read chain completion and data splits before comparing headline success. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read one horizontal block at a time. ABCD→D includes the test environment in training, whereas ABC→D holds D out. The 10% block uses 2,244 labeled trajectories; the unseen-language block evaluates full-data models on a common sample of generated paraphrases. Columns 1–5 are success fractions for completing that many tasks consecutively, and Avg. Len. summarizes completed tasks per chain. Appendix A.2 evaluates 1,000 chains with up to five instructions and a 360-timestep timeout per task. To understand the headline 0.949, compare the first-task column with the much more demanding five-task value, 0.731. calvin-resultscalvin-splitscalvin-datalanguage-evalcalvin-protocol
What it supports. GR-1 improves ABCD→D average chain length from HULC's 3.06 to 4.21. In ABC→D it reaches 3.06, compared with the strongest baseline average of 0.93 from MT-R3M. RT-1 instead leads baseline first-task success at 0.533, versus GR-1's 0.854; the best comparator depends on the metric.
Where the evidence stops. Section 4.1 gives MCIL/HULC access to labeled and unlabeled play, while GR-1/RT-1/MT-R3M use the labeled 1%. These are not identical training-data budgets, and the table supplies no uncertainty estimates.
Table 2. Physical manipulation improves, with a clear limit on category transfer. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the three transportation columns as different generalization tests. Seen objects includes the training objects and scenes with added distractors or background changes. Unseen instances replaces the eggplant, broccoli and pepper with new instances; unseen categories asks for tomato or yellow peach, absent as transportation targets in robot training. The final column concerns opening and closing a drawer. Section 4.2 reports 1,775 transportation demonstrations and 2,856 drawer trajectories. Appendix A.3 identifies a Kinova Gen2 with wrist RealSense and static Kinect Azure views. These are reported physical action outcomes, unlike the separate qualitative image predictions on page 9. real-resultsreal-setupfailures-video
What it supports. GR-1 leads all four columns, with success 0.79, 0.73, 0.30 and 0.75. The modest decline from seen objects to unseen instances contrasts with the much lower unseen-category result. That distinction supports instance transfer in the tested setup while keeping claims of broad category generalization limited.
Where the evidence stops. The PDF does not report physical evaluation trial counts, confidence intervals or precise success thresholds. Table 2 also does not separate the three seen-object scene conditions, so 0.79 cannot be assigned to each disturbed scene.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| CALVIN multi-task learning ABCD→D; language-labeled robot training subset; 1,000 instruction chains, at most five tasks, 360 timesteps per task. | 94.9%; 73.1%; 4.21 First-task success; five-task success; average completed chain length | HULC: 88.9%; 38.3%; 3.06. HULC and MCIL use labeled and unlabeled play; GR-1, RT-1 and MT-R3M use the labeled 1%. The 94.9% headline is first-task success, not completion of an entire five-task chain. The comparison is not matched in robot-data usage. calvin-resultscalvin-datacalvin-protocol |
| CALVIN unseen-scene generalization ABC→D: train on A/B/C and evaluate unseen environment D under the same chain protocol. | 85.4%; 40.1%; 3.06 First-task success; five-task success; average completed chain length | RT-1 has the best baseline first-task success, 53.3%; MT-R3M has the best baseline average length, 0.93. Different baselines lead different columns. This tests scene transfer within CALVIN's task family. calvin-resultscalvin-splitscalvin-protocol |
| CALVIN limited data and unseen language Separate ABCD→D experiments: 2,244 sampled labeled trajectories for limited data; full-data models evaluated on a common sampled set of generated instruction paraphrases. | Limited data: 77.8%, 2.00. Unseen language: 76.4%, 2.17. First-task success; average completed chain length | HULC: 66.8%, 1.11 and 71.5%, 1.82, respectively. Data scarcity and paraphrase transfer are distinct protocols; unseen language does not introduce new manipulation tasks. calvin-resultscalvin-datalanguage-evalcalvin-protocol |
| Real-robot transportation and drawer manipulation Kinova Gen2; 1,775 transportation demonstrations and 2,856 drawer trajectories. Transportation evaluates seen objects, unseen instances and unseen categories. | Seen objects 0.79; unseen instances 0.73; unseen categories 0.30; drawer manipulation 0.75. Reported task success rate | RT-1: 0.27/0.13/0.00/0.35; MT-R3M: 0.15/0.13/0.10/0.30 in the same column order. Physical execution improves in every reported setting, but category transfer remains weak. Trial counts and confidence intervals are not supplied. real-setupreal-results |
| Contribution of video prediction and pretraining CALVIN component ablation; compare neither component, video prediction without video pretraining, and both. | ABCD→D: 3.33→3.82→4.21; ABC→D: 2.40→2.65→3.06; limited data: 1.04→1.52→2.00. Average completed chain length | The scratch variants retain pretrained visual/text encoders. Figure 7(b) shows video prediction without pretraining underperforming the action-only variant on real-robot picking/transporting. Gains are not uniform across domains. The missing pretrained-but-no-video-loss condition prevents a complete factorial attribution. ablationobjectives |
| Future-frame horizon ablation CALVIN, without video pretraining; compare prediction offsets 1, 3 and 5. | 3.61, 3.82 and 3.67, respectively. Average completed chain length | Offset 3 is best among the three reported settings. This local optimum is not established for the fully pretrained system; Table 5 does not explicitly label its data split. horizon |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. Video supervision and pretraining each improve average CALVIN chain completion. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Within each split, read the two checkmark columns before the scores. The first row removes video pretraining and [OBS] prediction; the second retains video prediction but starts without video pretraining; the gray row uses both. These scratch variants concern GR-1's video-training stage: Section 3.3 still freezes pretrained CLIP and MAE encoders. Compare first-to-second rows for the added image objective, then second-to-third for video initialization. Use Avg. Len. to track the overall chain effect. On ABC→D the first-task score actually falls from 0.823 to 0.815 when only video supervision is added, even as average length rises. ablationobjectives
What it supports. Average lengths rise 3.33→3.82→4.21 on ABCD→D, 2.40→2.65→3.06 on ABC→D, and 1.04→1.52→2.00 with limited data. This is consistent with complementary contributions from image supervision and pretraining, but the benefit is not a uniform increase in every success column.
Where the evidence stops. The pretrained model fine-tuned without image supervision is missing. Moreover, Figure 7(b) and Appendix A.4 report that prediction without pretraining hurts the real-robot ablation relative to action-only training. CALVIN's pattern should not be generalized across domains.
Table 5. A moderately distant target works best in the reported scratch comparison. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read Future Step as the offset of the supervised image target, not an action-chunk length or the number of tasks in a chain. The three rows compare prediction one, three and five timesteps ahead; the remaining columns retain CALVIN's chain metrics. The setup sentence on page 15 explicitly disables video pretraining for this experiment. Appendix A.1 motivates different temporal offsets by denser robot observations than the sampled human-video frames. The discussion proposes that very near frames are too similar, while farther frames may become less informative for immediate local action prediction. Those explanations are hypotheses, not measurements of an internal mechanism. horizonimplementationfailures-videoqualitative
What it supports. The average completed length is 3.61 at offset one, 3.82 at three and 3.67 at five. Thus the tested objective benefits from moving beyond immediate-frame prediction, but performance does not increase monotonically with temporal distance. The results motivate a horizon choice rather than a general rule that farther prediction is better.
Where the evidence stops. Table 5 does not explicitly state the data split, and its no-pretraining setting differs from the full system. Figures 6 and 10–12 use Δt=1 for displayed predictions; standard robot fine-tuning in A.1 uses Δt=3.
7. Analysis & limitations
7.1 What the evidence leaves open
No seed variation or uncertainty is reported for the main quantitative comparisons. Real-robot evaluation denominators and precise success thresholds are absent, limiting statistical interpretation. calvin-resultsreal-resultsreal-setup
Reported GR-1 failures include confusing similarly colored pepper/peach objects, incomplete drawer closure and failure to engage the handle. Qualitative predictions omit some occluded details; they are not a calibrated dynamics benchmark. failures-video
Figures 6 and 10–12 show predictions trained with Δt=1, whereas Appendix A.1 specifies Δt=3 for standard robot fine-tuning. These are distinct displayed settings and should not be silently equated. failures-videoqualitativeimplementation
7.2 Questions for discussion
- Would pretraining still improve control if image supervision were removed only during robot fine-tuning?
- Does the best prediction horizon persist across pretraining conditions and unseen scenes?
8. Reproducibility audit
8.1 Requirements and known gaps
Use AdamW with cosine decay and dropout 0.1. Pretraining/CALVIN fine-tuning use batch sizes 1024/512, learning rates 3.6e−4/1e−3, warmup 5/1 epochs and training 50/20 epochs. Real-robot training changes batch size to 64 and epochs to 30. implementationreal-setup
A faithful attempt needs the Ego4D clip selection, frozen encoder checkpoints, labeled CALVIN trajectories, camera preprocessing and action conventions. The PDF omits compute hardware/time, software versions, exact pretrained checkpoint identifiers and sufficient physical evaluation detail for an exact replication. encoderspretrain-dataimplementationreal-setup
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Complete the missing pretraining-by-supervision comparison
Reader-proposed check, not performed: run a two-by-two CALVIN experiment crossing video pretraining with image supervision during robot fine-tuning. Keep CLIP/MAE checkpoints frozen and identical, retain the same token layout and output heads in all conditions, and disable the image-loss gradient rather than removing tokens. Match robot trajectories, optimizer steps and paired random seeds. Evaluate the same 1,000 chains on ABCD→D and ABC→D, reporting first-task success, full-chain success and average length with uncertainty. If pretrained action-only fine-tuning matches the jointly supervised pretrained model, continued image supervision is unnecessary for that gain; if it consistently loses, it supports an additional fine-tuning benefit beyond initialization. objectivestoken-maskimplementationcalvin-protocolablation
Check 2: Test whether the prediction horizon depends on pretraining
Reader-proposed check, not performed: repeat offsets 1, 3 and 5 both with and without video pretraining, explicitly identifying ABCD→D and ABC→D results. Fix observation history, frame sampling, robot fine-tuning data, update count and paired seeds. Use Appendix A.2's chain protocol, and separately measure held-out patch-normalized image error at each model's target offset; do not treat cross-offset pixel errors as equally difficult tasks. The falsifiable question is whether offset 3 remains best for control after pretraining and scene transfer. A changed ranking would limit the scope of Table 5's choice, while improved image error without improved control would weaken image fidelity as a sufficient explanation of policy gains. implementationcalvin-splitscalvin-protocolhorizonfailures-video
8.3 Reading coverage
Visual audit: The title/author page, both architecture figures, all seven tables, all twelve figures and every page supporting retained method, training, evaluation and reproduction details were visually inspected. Six final original crops were also inspected at their delivered sizes; legends, branch arrows, snowflakes, checkmarks and table headers remain intact. Figure 1's overview and Figure 2's paths were checked against Sections 3.1–3.3; the prediction-token attention rule comes from page 5 rather than an inferred diagram connection. The Δt=1 qualitative figures were kept distinct from A.1's standard Δt=3 robot setting. References and the remaining related-work page were read in the complete text pass. External videos and separately hosted supplemental material were outside this inspection.
PDF pages inspected for this edition: 1, 2, 4, 5, 6, 7, 8, 9, 14, 15, 16, 17, 18, 19, 20, 21, 22. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; Section 1 Introduction (pp. 1–2)
- Sections 2.1–2.3 Related Work (pp. 2–3)
- Sections 3.1–3.3 Problem Formulation, Architecture, Training (pp. 4–5)
- Sections 4.1–4.3 CALVIN, Real Robot Experiments, Video Prediction (pp. 6–9)
- Section 5 Conclusion; Acknowledgement and References (pp. 9–13)
- Appendix A.1–A.6: network/training, evaluation, real robot, ablations, task success, additional results (pp. 14–22)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- All six supplied text chunks were read individually, including references and the complete included appendix. The inspected title page identifies the ICLR 2024 conference edition; all nine authors match the catalog. Capitalization, line breaks and extraction spacing account for the title presentation difference. No numbered revision or revision history is supplied, so differences from earlier editions cannot be established.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by inspecting the original PDF figures, tables and supporting pages.
- Separate supplemental material availability has not been fully verified.
- External project videos and code were not inspected; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block and conference header
The title is Unleashing Large-Scale Video Generative Pre-training for Visual Robot Manipulation; the ICLR 2024 page lists Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li and Tao Kong, affiliated with ByteDance Research.
Go to primary source ↓overviewPDF pp. 1–2, Abstract, Section 1 and Figure 1
GR-1 transfers language-conditioned video prediction to robot imitation by copying weights and adding state/action interfaces.
Go to primary source ↓formulationPDF p. 4, Section 3.1, Equations (1)–(2)
Pretraining predicts a future frame from language and images; robot fine-tuning jointly predicts a future observation and current action from language, image history and robot-state history.
Go to primary source ↓encodersPDF p. 4, Section 3.2.1 and Figure 2(a–e)
CLIP encodes text; MAE ViT provides global and local image representations; a perceiver resamples patches; robot states use projections. Separate image and action decoders read the shared model.
Go to primary source ↓token-maskPDF p. 5, Sections 3.2.1–3.2.2
Language is repeated per timestep, temporal embeddings align modalities, and causal attention excludes [OBS]/[ACT] positions as attended tokens.
Go to primary source ↓objectivesPDF p. 5, Sections 3.2.3–3.3, Equation (3)
Image decoding uses [OBS] outputs plus positional mask tokens; action prediction uses [ACT]. Training combines MSE, Smooth-L1 and BCE; CLIP and MAE are frozen in both phases.
Go to primary source ↓pretrain-dataPDF p. 5, Section 3.3, Pre-Training paragraph
Three-second Ego4D clips yield 800,000 clips and 8M frames for language-conditioned video pretraining.
Go to primary source ↓implementationPDF p. 14, Appendix A.1 and Table 3
The source specifies transformer size, parameter counts, three-layer action head, sequence length, frame spacing, offsets, two-camera prediction, patch normalization, augmentation and optimization hyperparameters.
Go to primary source ↓calvin-splitsPDF p. 6, Section 4.1, Experiment Setup
CALVIN contains 34 tasks; actions use delta position/orientation and binary gripper commands. ABCD→D includes D in training; ABC→D holds D out.
Go to primary source ↓calvin-protocolPDF p. 14, Appendix A.2
Evaluation uses 1,000 unique chains, up to five tasks, a 360-timestep timeout, progression only after success and neutral reset after each chain.
Go to primary source ↓calvin-dataPDF p. 7, Section 4.1, Baseline Methods and Data Efficiency
MCIL/HULC use all play data; RT-1/MT-R3M/GR-1 use the language-labeled 1%. Limited data samples 66 trajectories for each of 34 tasks, totaling 2,244 from 22,966.
Go to primary source ↓calvin-resultsPDF p. 6, Table 1, all four protocol blocks and columns 1–5/Avg. Len.
GR-1 reports 0.949/0.731/4.21 for first/fifth/average in ABCD→D and 0.854/0.401/3.06 in ABC→D; limited-data and language results retain their separate table blocks. No uncertainty columns appear.
Go to primary source ↓language-evalPDF p. 7, Section 4.1, Zero-Shot Unseen Language Generalization; p. 16, Table 6
Fifty GPT-4-generated paraphrases per task are sampled for a common evaluation set; all methods use ABCD→D-trained models. Table 6 provides examples.
Go to primary source ↓real-setupPDF p. 8, Sections 4.2.1–4.2.2 and Figure 5; pp. 14–15, Appendix A.3
The paper specifies demonstration totals, scene/instance/category distinctions, Kinova Gen2, wrist RealSense/static Kinect Azure and real-robot training changes. It does not state evaluation trial totals or detailed success thresholds.
Go to primary source ↓real-resultsPDF p. 8, Table 2, RT-1/MT-R3M/GR-1 rows and four task-setting columns
Reported GR-1 rates are 0.79/0.73/0.30/0.75 versus RT-1 0.27/0.13/0.00/0.35 and MT-R3M 0.15/0.13/0.10/0.30; uncertainty is not tabulated.
Go to primary source ↓failures-videoPDF p. 8, Section 4.2.1 Results; p. 9, Section 4.2.2 continuation, Figure 6 and Section 4.3
Failures include pepper/peach confusion and drawer closure/handle engagement. The paper notes missing occluded details. Figure 6 distinguishes green ground truth from blue predictions and specifies Δt=1.
Go to primary source ↓ablationPDF p. 15, Appendix A.4, Figure 7 and Table 4
Three conditions compare no video prediction/pretraining, prediction only and both. Table 4 gives average lengths 3.33/3.82/4.21, 2.40/2.65/3.06 and 1.04/1.52/2.00. Figure 7(b) and its discussion show the prediction-only variant below action-only in real-robot tests.
Go to primary source ↓horizonPDF pp. 15–16, Appendix A.4, Different Future Predictions and Table 5
Pretraining is disabled. Offsets 1/3/5 yield average lengths 3.61/3.82/3.67. Authors hypothesize distant images become less useful for local actions; the table lacks an explicit split label.
Go to primary source ↓task-diagnosticPDF p. 16, Appendix A.5; p. 17, Table 7, stack block and push blue block right rows
Block-manipulation gains vary by task: stack block is 80.1% versus 45.7% without prediction/pretraining; push blue block right remains 53.6% for GR-1.
Go to primary source ↓qualitativePDF pp. 18–22, Figures 8–12 and captions
Figures 8–9 show selected executed rollouts. Figures 10–12 compare green ground truth and blue predicted images for Δt=1; they supply qualitative examples rather than image-quality metrics.
Go to primary source ↓8.5 Primary sources
UNLEASHING LARGE -SCALE VIDEO GENERATIVE PRE-TRAINING FOR VISUAL ROBOT MANIPULATION ↗
PDF · 9,543 extracted words
Source fingerprint
b4d3d9f82a0793db9ff5db3d93ae9ae950bc4692e22508136e6770cdea93babd