GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation
1. Paper overview
In one sentence: GR-2 transfers large-scale video prediction into a policy that generates future images and action trajectories, achieving strong manipulation results while leaving the causal role of visual planning incompletely tested. e01e03e04e07e08e09e10e11
| At a glance | What to know |
|---|---|
| Research problem | Source description Robot demonstrations are costly to collect. GR-2 asks whether text-conditioned prediction of everyday videos can supply temporal and semantic knowledge for a policy that follows language instructions across tasks and visual changes. The target is manipulation from observation and robot-state histories, including unfamiliar objects and environments. e01e03 |
| Core mechanism | |
| A key reported result | Physical multi-task manipulation: Simple: 97.7% Reported task success rate (%). 105 teleoperated tasks; training-like environment; approximately 400 trajectories per task, without augmentation. The small-data GR-2 condition with about 50 trajectories per task reaches 73.9%; Figure 6 also shows GR-1 below full-data GR-2. Strong in-setting execution; the demonstration-budget comparison is not an isolated pre-training ablation. Physical evaluation trial counts and confidence intervals are not supplied. e06e07 |
| Reading caution |
Core contributions
Figure 1. Video prediction supplies the pre-training task; robot data adds action prediction. Original paper, p. 2 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the blue examples: their text descriptions accompany human activities and robot videos, illustrating the video-language pre-training mixture. Then move to the orange examples, which depict downstream robot tasks such as operating a toaster and transferring objects between baskets. The two headings identify training stages, while the central GR-2 block connects them. Cross-reading the caption with Section 2.1 is essential: the model predicts discrete future image tokens during pre-training and adds robot-state inputs and cVAE action trajectories during fine-tuning. The double-headed arrows summarize the overview; they do not specify bidirectional attention, backpropagation paths or the order of token generation. e02e03e04
What it supports. The transferable task is text-conditioned prediction of visual change. The method section describes adapting that capability to produce multi-view future images and action trajectories together. The illustration clarifies the two training stages, but the scientific interface comes from Equations (1)–(2) and the surrounding encoder and decoder description.
Where the evidence stops. This is an overview rather than a detailed architecture diagram. Neither the arrows nor the faint central icons resolve how future-image information reaches the cVAE, which parameters are shared, or which attention connections are permitted.
2. Motivation
2.1 The problem and the proposed response
Robot demonstrations are costly to collect. GR-2 asks whether text-conditioned prediction of everyday videos can supply temporal and semantic knowledge for a policy that follows language instructions across tasks and visual changes. The target is manipulation from observation and robot-state histories, including unfamiliar objects and environments. e01e03
2.2 What this reading follows
A robot can learn what a task should look like from video, but it still needs to produce executable motion. GR-2 connects these problems through video generative pre-training and robot fine-tuning with image and action outputs. The supplied October 2024 v1 report evaluates physical table-top manipulation, basket-to-basket picking and simulated CALVIN instruction chains. Read its evidence in that order: first the training interface, then execution under different shifts, then the scaling and rollout diagnostics. The strongest measurements concern completed robot tasks. The attractive interpretation that generated video acts as a plan remains less directly tested, because the report lacks a causal ablation and a detailed future-to-action interface. e01e03e04e07e08e09e10e11
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The stated architecture is one GPT-style policy π whose outputs include future images and actions, with a cVAE action generator. This supports One Model × Joint prediction at the described policy level, beyond merely sharing a training dataset. The report does not present a separately trained inverse-dynamics model; its visual-planner language does not establish such a decomposition. Missing token-level connectivity limits a more detailed architectural judgment. e03e04e10
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn visual change before learning robot trajectories
Start from the supervision available in each stage. A text-video pair can teach the model what the described activity looks like over time without providing a robot action label. GR-2 converts images to discrete VQGAN tokens and trains a GPT-style transformer to predict future image tokens conditioned on language and preceding images. The text encoder and VQGAN are frozen during GR-2 training, while robot states later enter through trainable linear encoders. The corpus mixes human activities with robot videos and uses hand filtering and re-captioning. During robot fine-tuning, the input interface expands to multi-view image histories and end-effector/gripper states, and the output interface adds actions. This establishes what transfers between stages; the report does not specify the token layout or loss weights needed to implement that transfer exactly. e03e04
Figure 12, oven and toaster examples. Paired visual futures and recorded rollouts illustrate the proposed planning interpretation. Original paper, p. 11 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. For each instruction, read left to right within the orange Pred row, then inspect the corresponding blue GT row beneath it. The first pair depicts opening the oven; the second depicts pressing the toaster switch. Pred means an autoregressively generated video, while GT is the recorded physical rollout. Compare the arm's approach, interaction with the appliance and resulting scene change. The overall task progression is similar, but individual poses and timing do not coincide exactly. Section 3.4 interprets such correspondence as actions replaying a predicted visual trajectory; this crop lets the reader inspect that qualitative evidence directly. e10e12
What it supports. The examples support the narrower observation that generated images can depict task-relevant motions resembling real execution. They make the proposed visual-plan interpretation understandable and expose differences that a quantitative alignment measure would need to capture. They do not by themselves show that the action generator depends causally on these particular generated frames.
Where the evidence stops. These are selected qualitative examples with no error metric, failure frequency or controlled removal of the visual prediction path. The paper's broader claim that improving video generation can improve action prediction remains a hypothesis requiring an intervention.
5.2 Separate a predicted visual future from executed motion
Equation (2) gives one policy two outputs: a future observation and an action trajectory. Section 2.1 assigns action generation to a conditional VAE, and Section 3.4 describes inferring actions from a generated visual trajectory. Read together, they motivate a policy that uses visual prediction to guide motion, but they do not specify the conditioning graph needed to verify every dependency. Execution adds another stage: the model's Cartesian trajectory is smoothed and tracked through whole-body control, which produces low-level joint actions. The reported 200 Hz belongs to that controller, not to the policy's video generation rate. Reader interpretation: separating these interfaces prevents plausible images, feasible trajectories and successful physical interaction from being treated as interchangeable achievements. The paired rollout figure supports correspondence; an intervention is needed to establish dependence. e03e04e05e10e12
5.3 Read generalization as several different tests
Begin with the physical multi-task plot and keep the demonstration budget visible: the roughly 50-trajectory condition answers a different question from the roughly 400-trajectory condition. Next separate appearance changes from unfamiliar manipulation. Backgrounds and kitchens can change while the underlying skill remains recognizable; Unseen Manipulation also includes unfamiliar object categories and instances. The augmentation recipe addresses appearance through object insertion and background replacement, and its uneven gains suggest a narrower benefit than universal task generalization. Bin picking then tests a fixed any-object instruction across object novelty and clutter, whereas CALVIN tests consecutive language-conditioned tasks in simulation. Finally, the scaling plot changes model capacity and uses a subset of physical settings. Reader interpretation: these results are complementary evidence, but pooling them would hide their different data, novelty and success definitions. e06e07e08e09e11
5.4 Training and inference
During training
Pre-training mixes Howto100M, Ego4D, Something-Something V2, EPIC-KITCHENS and Kinetics-700 with RT-1 and Bridge videos. Hand filtering and re-captioning are described, but selection thresholds and a complete optimization recipe are absent. e04
Multi-task fine-tuning uses approximately 40,000 teleoperated trajectories over 105 tasks and eight skills, about 400 demonstrations per task; the small-data condition uses about 50. Augmentation inserts objects with diffusion, segments backgrounds with SAM, and propagates an inpainted frame through video generation while preserving robot motion. e06
The default model has 230M total parameters, with 95M described as trainable. Section 3.5 separately states that its scaling experiment includes in-domain robot videos during pre-training and freezes pre-trained parameters during robot fine-tuning; that statement is not an unambiguous freeze specification for every experiment. e05e11
During inference
The policy receives language and recent observations/states and produces future images and an action chunk. Section 3.4 describes actions inferred from a generated visual trajectory, but supplies no token-order algorithm, replanning schedule, chunk execution fraction or policy latency. The reported 200 Hz concerns low-level joint execution. e03e05e10
The visual-planning interpretation is consistent with paired predictions, but the report does not establish a search over candidate futures or isolate inference-time reliance on generated frames. e10e12
5.5 Implementation flow
- Encode observations and state
A frozen text encoder tokenizes the instruction. A VQGAN, trained on Internet and in-domain robot images and then frozen, converts images into discrete tokens. Trainable linear layers encode robot states during fine-tuning. e03
- Predict future images
The GPT-style transformer autoregressively predicts future image tokens from text and image history; the VQGAN decoder reconstructs images. Pre-training uses single-view videos, while robot fine-tuning supports multiple views. e03e04
- Generate an action trajectory
Robot fine-tuning predicts future images and actions in tandem. A conditional VAE generates the action trajectory. The authors attribute smoother trajectories and real-time feasibility to predicting action sequences; the report does not specify the cVAE conditioning interface or provide a single-action ablation. e03e04
- Execute through a controller
Trajectory optimization improves smoothness and continuity. Whole-body control converts the Cartesian trajectory to joint actions, incorporating collision constraints and manipulability. Execution uses a Kinova Gen3 arm, Robotiq 2F-85 gripper, static head camera and wrist camera. e05
6. Experiments & results
GR-2 learns language-conditioned video prediction at scale, then adapts a GPT-style policy to predict robot action trajectories alongside future images. Physical multi-task and bin-picking results and simulated CALVIN chains support its practical capability, while qualitative video alignment and model scaling leave the causal contribution of visual planning unresolved.
The supplied v1 PDF presents its quantitative results as Figures 6, 9 and 10, not tables, so this edition retains original result plots rather than inventing a table crop. Figure 1 is a training overview; Section 2.1 provides only a high-level architecture description, without a detailed token/attention diagram. Figure 11 supplies a scaling diagnostic, while Figure 6 compares data budget and combined augmentation; the source has no isolated ablation of generated-video conditioning or the cVAE action interface. These are source limitations despite the full supplied-text reading. e02e03e04e07e08e09e11
6.1 Read the original evidence
Figure 6. Augmentation helps environment shifts, but its gains do not extend uniformly to every setting. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each group against the legend before comparing heights. Red is GR-1 with about 400 demonstrations per task; orange is GR-2 with about 50; light blue is GR-2 with about 400; dark blue adds data augmentation at that larger budget. Simple resembles training, Distractor adds clutter, and the three Unseen groups test background, environment and manipulation changes. The last group aggregates the unseen settings rather than introducing another test condition. Use the light-blue versus dark-blue comparison to examine augmentation, and the orange versus light-blue comparison to examine data budget. These comparisons answer different questions from the GR-1 comparison. e06e07
What it supports. Adjacent source text reports 97.7% Simple success for full-data GR-2 and 73.9% for the small-data version. For unseen environments, augmentation raises the reported result from 71.7% to 87.0%, a calculated 15.3 percentage-point gain. The plot also shows lower augmented bars in Simple and Unseen Manipulation, so the average gain should not be read as universal improvement.
Where the evidence stops. Exact percentages come from the surrounding results text, not precision estimates from bar heights. The figure gives neither error bars nor trial denominators. Augmentation combines object insertion and background editing, so this comparison does not separate their effects.
Figure 9. Object novelty and added clutter leave GR-2's bin-picking success comparatively stable. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The orange and blue bars compare GR-1 and GR-2 under one fixed instruction: transfer any object from the right basket to the left basket. Seen and Unseen distinguish whether object identities occurred in robot training. Cluttered Seen and Cluttered Unseen increase the initial basket population beyond the training-like conditions. Section 3.2 specifies 5–9 objects for the first pair of settings and 12–17 for the cluttered pair. Compare each blue bar with its orange counterpart, then compare the blue bars with one another. This isolates the reported pattern across object novelty and clutter, without turning the task into a test of language diversity. e08
What it supports. The text reports average success of 79.0% for GR-2 versus 33.3% for GR-1, a calculated 45.7 percentage-point difference. The blue bars remain comparatively similar across the four conditions, supporting robustness within this basket-transfer setup. The experiments involve 55 seen and 67 unseen objects after training on approximately 94,000 trajectories.
Where the evidence stops. The plot has no exact per-bar labels or uncertainty intervals, and the text does not specify the averaging weights or trial denominators. These success results do not provide cycle time, sustained throughput or a general industrial reliability guarantee.
Figure 10. GR-2 completes more of the five-task instruction chains in the reported ABCD-D simulation protocol. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The first five groups use the left success-rate axis and represent completing increasingly long prefixes of a five-instruction chain. They are not five unrelated single-task tests. The final group, separated by a dashed line, uses the right axis for average completed tasks per chain. Dark blue denotes GR-2; the other colors identify RT-1, MT-ACT, HULC, RoboFlamingo and GR-1. Read the printed GR-2 labels directly, then use Section 3.3 for the baseline values and evaluation protocol: ABCD-D, 34 tasks and 1,000 instruction chains. Keep this simulated evaluation separate from the physical multi-task and bin-picking experiments. e09
What it supports. GR-2's labeled completion rates decrease from 98.6% for the first task to 85.9% for all five; its mean chain length is 4.64. The source reports GR-1 at 73.1% for all five and 4.21 average length. The calculated gains are 12.8 percentage points and 0.43 completed tasks per chain.
Where the evidence stops. The final bar group measures tasks, not percentages. The reported comparison is specific to ABCD-D and the listed baselines; the figure alone does not show equal pre-training data or compute across methods, or transfer to another split.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Physical multi-task manipulation: Simple 105 teleoperated tasks; training-like environment; approximately 400 trajectories per task, without augmentation. | 97.7% Reported task success rate (%) | The small-data GR-2 condition with about 50 trajectories per task reaches 73.9%; Figure 6 also shows GR-1 below full-data GR-2. Strong in-setting execution; the demonstration-budget comparison is not an isolated pre-training ablation. Physical evaluation trial counts and confidence intervals are not supplied. e06e07 |
| Physical multi-task manipulation: generalization Unseen Backgrounds, Unseen Environments and Unseen Manipulation; approximately 400 trajectories per task. | GR-2: 71.4% backgrounds, 71.7% environments, 55.8% manipulation; GR-2 with augmentation: 87.0% environments and 74.7% mean across the three unseen settings. Reported success rate (%) | The environment result improves by 15.3 percentage points with augmentation, calculated from reported values. Figure 6 shows no improvement for unseen manipulation. Appearance robustness and novel-object manipulation remain distinct challenges; augmentation is not uniformly beneficial. e06e07 |
| Physical end-to-end bin picking Approximately 94,000 training trajectories using 55 objects; evaluation spans 122 objects, including 67 unseen, across Seen, Unseen and their cluttered counterparts. | 79.0% Reported average success rate (%) | GR-1: 33.3%; a calculated difference of 45.7 percentage points. Supports object and clutter generalization under the fixed any-object transfer instruction. It does not measure instruction diversity or industrial throughput; trial denominators and uncertainty are absent. e08 |
| CALVIN ABCD-D instruction chains Simulation; more than 20,000 expert demonstrations covering 34 tasks; 1,000 evaluation chains of five successive instructions. | 1–5 tasks: 98.6%, 96.1%, 93.1%, 90.1%, 85.9%; average length 4.64. Consecutive-task completion rate (%); mean completed tasks per chain | GR-1: 94.9% for one task, 73.1% for five, average length 4.21. Five-task completion improves by 12.8 percentage points and average length by 0.43 tasks. This is the reported ABCD-D protocol; its results should not be conflated with other benchmark splits. e09 |
| Model-size scaling diagnostic GR-2-S/B/L/XL: 30M/95M/312M/719M trainable parameters; video validation and a subset of physical multi-task settings. | Qualitative plotted trend: larger models have lower video loss and higher physical success. Video validation loss; physical success rate | Four model sizes evaluated in Figure 11; exact robot percentages and the task subset are not enumerated in the text. Supports scaling within this setup. It neither establishes compute-matched superiority nor isolates visual prediction as the cause of action improvement. e11 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 11(a)–(d). Larger models improve both video loss and robot success in the reported scaling setup. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panels (a), (b) and (c) plot validation loss against training iterations on Ego4D, RT-1 and the authors' robot data, respectively. Their vertical ranges differ, so compare model ordering within each panel rather than absolute loss across datasets. The legend progresses from GR-2-S in red to GR-2-XL in dark blue; Section 3.5 gives 30M, 95M, 312M and 719M trainable parameters. Panel (d) changes the quantity to physical task success, with a vertical axis beginning above zero. Its bars come from a subset of multi-task settings. The subsection specifically includes in-domain video pre-training and frozen pre-trained parameters during fine-tuning. e11
What it supports. The ordering is consistent: larger models have lower validation losses and higher robot success in this experiment. This supports the authors' claim that the approach benefits from scale under the reported recipe. It provides a useful diagnostic connection between prediction and control, but model size changes both capabilities together.
Where the evidence stops. This is a scaling diagnostic, not an isolated ablation of visual planning. The task subset, exact success values, uncertainty and matched compute budgets are unspecified. The truncated success axis also makes bar-height ratios unsuitable as effect-size estimates.
7. Analysis & limitations
7.1 What the evidence leaves open
Novel-shape grasp failures and selection of the wrong unseen object remain explicit failure modes. The authors identify unseen manipulation as a target for future improvement. e07e13
Predicted/real rollouts demonstrate qualitative correspondence. There is no quantitative video-action alignment metric or intervention showing that better generated videos themselves improve control. e10e12
Comparisons with GR-1 change more than one factor, including pre-training scale and architecture. Real-robot success plots omit uncertainty, per-setting trial counts and detailed success adjudication, limiting statistical assessment. e07e08e13
7.2 Questions for discussion
- Would blocking generated-future information reduce executed success when model capacity and training data are controlled?
- Why does appearance augmentation improve unseen environments while leaving unseen manipulation unimproved?
- How would the reported gains change with fixed compute and a fully specified physical evaluation protocol?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the curated video mixture, trained frozen encoders, robot trajectories, augmentation pipeline and controller integration. Missing details include optimizer and loss weights, cVAE architecture, token ordering, image resolution, history/chunk lengths, filtering thresholds, accelerator type/count and training duration. e03e04e05e06e11
A reader-proposed mechanism check would hold robot data, backbone size and controller fixed while comparing valid predicted-future conditioning with a matched ablation of that information path. Measure executed success and latency as well as prediction quality; similar success despite degraded futures would weaken the planner interpretation. e03e04e10
A reader-proposed augmentation check would separately vary background replacement and object insertion at fixed demonstration count and training steps, evaluating appearance shifts and unseen manipulation independently. Report repeated-seed uncertainty rather than only their pooled average. e06e07
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does generated-future information change executed actions?
Reader-proposed, not performed: first obtain an explicit description of the future-to-action conditioning interface, which this report omits. If that interface consumes generated-future information, compare the same checkpoint on the same 1,000 CALVIN ABCD-D chains using its normal futures versus temporally shuffled futures. Hold language, observed history, robot state, action sampling procedure and controller fixed; repeat paired trials to estimate variability. Record action changes, completed-chain length, five-task success and latency. If changing the futures leaves actions and success unchanged despite visibly different predicted sequences, the inference-time planner interpretation is weakened. A success drop supports dependence, but does not alone establish physically accurate prediction; repeat with a model trained without that conditioning path to distinguish useful information from sensitivity to corrupted inputs. e03e04e09e10
Check 2: Which augmentation causes the generalization gain?
Reader-proposed, not performed: run a two-by-two comparison of background replacement on/off and object insertion on/off, keeping the approximately 400-demonstrations-per-task budget, starting checkpoint, optimizer budget and video-propagation procedure fixed. Audit edited clips for unchanged robot motion. Evaluate the same physical starts separately in Simple, Distractor, Unseen Backgrounds, Unseen Environments and Unseen Manipulation, reporting trial counts and uncertainty across training seeds. A gain restricted to appearance shifts, with flat or lower unseen-manipulation success, would support the narrower reading of Figure 6. A reproducible unseen-manipulation gain from one augmentation component would identify an effect obscured by the combined recipe. Keep each setting visible rather than accepting improvement only in the unseen average. e06e07
8.3 Reading coverage
Visual audit: The title and author page, method/equation/training pages, all result plots and all 17 original figures were visually inspected. Pages 3–5 verify the encoders, cVAE description, data, hardware and controller; pages 7–8 and 10 verify numerical protocols/results and scaling; pages 11–16 verify predicted/real rollouts; page 17 supports the comparison and stated future work. All six final crops were individually inspected, including the corrected scaling crop. Figure 1's overview arrows were cross-checked with its caption and Equations (1)–(2); they are not treated as attention or causal wiring. Figure 10's two axes and Figure 11's panel labels and truncated success axis were checked. References and the continuation of acknowledgements on pages 18–21 were read as text but were not included in the visual pass. No separate supplement, project page, code or experiment execution was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract (p. 1)
- 1 Introduction (pp. 1–2)
- 2 Methods (pp. 2–3)
- 2.1 Model & Training (pp. 3–4)
- 2.2 Real-Robot System & Deployment (p. 4)
- 3 Experiments (p. 4)
- 3.1 Real-World Multi-Task Learning (pp. 5–8)
- 3.2 End-to-End Bin Picking of Different Objects (pp. 7–9)
- 3.3 CALVIN Benchmark (pp. 8, 10)
- 3.4 Autoregressive Video Generation (pp. 10–16)
- 3.5 Scaling (p. 10)
- 4 Related Work (p. 17)
- 5 Conclusions (p. 17)
- Contributions & Acknowledgements (pp. 17–18)
- References (pp. 18–21)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The entire supplied 21-page text was read. Original PDF pages 1–17 and all 17 figures were visually inspected, resolving the figure-image limitation of text extraction for this reading. No appendix is present in this PDF.
- The title page identifies arXiv:2410.06158v1 [cs.RO], 8 October 2024. Its title and all twelve authors match the catalog; ByteDance Research is the observed affiliation. Later revisions or editions were not supplied or compared.
- No separate supplement was supplied. The project page, code, datasets and cited works were not accessed; no experiments were reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title block, arXiv margin, Abstract and §1
The observed title matches the catalog. The twelve listed authors are Chi-Lam Cheang, Guangzeng Chen, Ya Jing, Tao Kong, Hang Li, Yifeng Li, Yuxiao Liu, Hongtao Wu, Jiafeng Xu, Yichu Yang, Hanbo Zhang and Minzhao Zhu, affiliated with ByteDance Research. The margin identifies arXiv:2410.06158v1 [cs.RO], 8 Oct 2024. The introduction motivates video pre-training by robot-data collection costs.
Go to primary source ↓e02PDF p. 2, Figure 1 and introduction contribution bullets
Figure 1 distinguishes blue video-language pre-training examples from orange video-language-action fine-tuning examples around GR-2. Its double-headed overview arrows are not a token-level computation graph. The text claims lossless transfer and introduces whole-body control.
Go to primary source ↓e03PDF p. 3, §2 and §2.1, Eqs. (1)–(2) and encoder paragraphs
A universal policy takes instruction, observation history and robot-state history to predict actions; Eq. (2) also outputs a future observation. The model is GPT-style, using frozen text and VQGAN encoders, trainable linear state encoders and autoregressive future image tokens. Robot states comprise end-effector position/rotation and gripper state. No training-loss equation or action-conditioning graph is supplied.
Go to primary source ↓e04PDF p. 4, §2.1, pre-training corpus and multi-view fine-tuning paragraphs
The video mixture includes five human-video datasets plus RT-1 and Bridge, with hand filtering and re-captioning; 38 million clips correspond to approximately 50 billion tokens. Fine-tuning outputs future images for each view and cVAE action trajectories. The authors assert benefits of trajectory prediction but do not give a single-step-action comparison here.
Go to primary source ↓e05PDF p. 4, §2.2 and §3 opening paragraph
Hardware is a 7-DoF Kinova Gen3 with Robotiq 2F-85, static head and end-effector cameras. Cartesian trajectory optimization and WBC yield joint actions executed at 200 Hz, with collision and manipulability considerations. The default model has 230M total parameters and 95M trainable parameters.
Go to primary source ↓e06PDF p. 5, §3.1; p. 7, continuation of generalization settings
The authors collect approximately 40,000 demonstrations of 105 tasks/eight skills and compare about 400 versus 50 trajectories per task. Augmentation combines diffusion object insertion, SAM background extraction and video propagation. Evaluation distinguishes Simple, Distractor, unseen tablecloth backgrounds, two unseen kitchens and novel object categories/instances.
Go to primary source ↓e07PDF p. 7, Figure 6 and §3.1 Results; p. 8, opening continuation
Text reports 97.7% Simple, 71.4% Unseen Backgrounds, 71.7% Unseen Environments and 55.8% Unseen Manipulation for GR-2; augmentation gives 87.0% Unseen Environments and 74.7% unseen average. The small-data model gives 73.9% Simple. Figure 6 shows augmentation below unaugmented GR-2 in Simple and Unseen Manipulation. The text names novel-shape grasping and wrong-object selection failures. No trial-count or uncertainty annotation accompanies the plot.
Go to primary source ↓e08PDF p. 8, §3.2 and Figure 9; p. 7, Figure 7
Bin picking uses about 94,000 trajectories of 55 objects and evaluates 122 objects, including 67 unseen. The fixed instruction transfers any object between baskets. Seen/Unseen begins with 5–9 objects; cluttered conditions use 12–17. Mean success is 79.0% for GR-2 versus 33.3% for GR-1. Figure 9 shows higher GR-2 bars in all four settings, without exact bar annotations, trial counts or uncertainty.
Go to primary source ↓e09PDF p. 8, §3.3; p. 10, Figure 10 and caption
CALVIN uses ABCD-D, more than 20,000 demonstrations, 34 tasks and 1,000 five-instruction chains. Figure 10 labels GR-2 completion rates 98.6, 96.1, 93.1, 90.1 and 85.9%, and mean length 4.64. Text gives GR-1 one/five-task rates 94.9/73.1% and mean 4.21. Mean length uses a separate right-hand axis.
Go to primary source ↓e10PDF p. 10, §3.4
The authors describe an action trajectory inferred from a generated visual trajectory and interpret predicted/real video alignment as replaying that visual plan. The subsection supplies qualitative comparisons, not a causal intervention, quantitative alignment metric, token-order algorithm or policy timing measurement.
Go to primary source ↓e11PDF p. 10, §3.5 and Figure 11(a)–(d)
Trainable sizes are 30M, 95M, 312M and 719M. Panels (a) Ego4D, (b) RT-1 and (c) robot data plot video validation loss against iterations; larger models have lower losses. Panel (d) shows increasing success on an unspecified subset of §3.1 settings. The subsection includes in-domain videos in pre-training and freezes pre-trained parameters during fine-tuning. Compute budgets, exact success values and uncertainty are not provided.
Go to primary source ↓e12PDF pp. 11–16, Figures 12–17; specifically p. 11, Figure 12 oven and toaster rows
Orange Pred rows show generated sequences; blue GT rows show captured rollouts. Figure 12 includes opening an oven and pressing a toaster switch, with broadly corresponding motion but visible differences in pose and timing. Figures 13–17 extend the qualitative comparisons to more tasks, bin picking and CALVIN views; no alignment metric is displayed.
Go to primary source ↓e13PDF p. 17, §4 Robot Learning with Pre-training and §5 Conclusions
The comparison to GR-1 states an increase from 0.8 million to 38 million pre-training videos and an architectural change. The conclusion describes correlation between generated videos and predicted actions, and prioritizes unseen-manipulation generalization and robustness.
Go to primary source ↓8.5 Primary sources
GR-2: A Generative Video-Language-Action Model with Web-Scale Knowledge for Robot Manipulation ↗
PDF · 7,597 extracted words
Source fingerprint
8b9bf7c3de182ce263ca65d103fc3a84aa0c70d13aa88f6b8dd5cdf5df2f5f3a