CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models
1. Paper overview
In one sentence: A shared multimodal model predicts an image of the desired future before generating robot actions, improving several manipulation results at the cost of slow image generation and delayed feedback. e-probleme-architecturee-formulatione-controle-componentse-oraclee-limitations
| At a glance | What to know |
|---|---|
| Research problem | Source description Direct vision-language-to-action prediction lacks an explicit intermediate manipulation target. The authors propose future images as naturally available supervision that also exploits videos without action labels, avoiding additional keypoint or bounding-box annotation pipelines. e-probleme-formulation |
| Core mechanism | Source description A shared VILA-U-based model generates subgoal images and then goal-conditioned actions, obtaining observation feedback after each executed chunk. e-architecturee-control |
| A key reported result | Franka-Tabletop adaptation: 78.8% Average task success rate. Previously unseen Franka setup; six tasks, separate model per task, 10–150 demonstrations each; three single-instruction and three multi-instruction settings. OpenVLA 67.3%, Diffusion Policy 51.2%, Octo 41.5%. +11.5 percentage points over OpenVLA, approximately 17.1% relative. CoT-VLA does not win every task. Per-task trial counts and uncertainty are not provided here. e-evaluatione-franka |
| Reading caution | Author claim Authors report 7× average inference slowdown at chunk size 10 from generating 256 image tokens, poorer image quality than diffusion, chunk-boundary discontinuities and missing high-frequency feedback. Entirely new-task goal generation remains difficult. e-limitationse-backbone |
Core contributions
- Source description
A shared VILA-U-based model generates subgoal images and then goal-conditioned actions, obtaining observation feedback after each executed chunk. e-architecturee-control
- Source description
Hybrid attention combines causal image/text prediction with parallel action decoding. Ablations examine successive additions of chunking, full action attention and visual CoT. e-attentione-components
Figure 2. Predict a visual target, generate an action chunk, execute, and observe again. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Begin with the blue observation tokens and green instruction below the shared CoT-VLA block. The upper blue branch represents the generated subgoal image; the yellow branch represents a chunk of actions. The left-hand boxes describe training sources, rather than three separate deployment inputs. Follow the dotted line on the right from action execution back to a new observation: Algorithm 1 obtains that observation after executing the chunk. The caption and Equations (2)–(3) establish the sequential dependence of actions on the generated goal, even though the overview places both output branches above the same block. e-architecturee-formulatione-backbonee-control
What it supports. The architecture supports a one-model interpretation: the shared multimodal model generates both goals and actions. Action-less videos can supervise the image branch, while demonstrated actions supervise control. The predicted image is used during deployment, so its role extends beyond an auxiliary training objective.
Where the evidence stops. Figure 2 labels the action endpoint a_n, whereas Section 3.1 uses n for the image horizon and m for actions. Algorithm 1 also uses inclusive m+1 indexing. Treat these as unresolved notation inconsistencies, not distinct verified horizons.
2. Motivation
2.1 The problem and the proposed response
Direct vision-language-to-action prediction lacks an explicit intermediate manipulation target. The authors propose future images as naturally available supervision that also exploits videos without action labels, avoiding additional keypoint or bounding-box annotation pipelines. e-probleme-formulation
2.2 What this reading follows
CoT-VLA places a visible prediction between an instruction and a robot command: an image of a future subgoal. That image then conditions an action chunk, after which a fresh camera observation restarts the process. This provides a way to train on captioned videos without pretending those videos contain robot action labels. The important evidence is therefore not just whether predicted images look plausible, but whether conditioning on them helps executed manipulation. Read the architecture alongside the component ablations and the ground-truth-goal intervention. Together they support a useful control interface while leaving open how much improvement comes specifically from video data, semantic goal quality or additional computation. e-probleme-architecturee-formulatione-controle-componentse-oraclee-limitations
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Not assigned |
| Architecture | Not assigned |
| Prediction paradigm | Not assigned |
| Quadrant | Not assigned |
This table preserves the labels recorded at reading time. The current major category is WAMs. View the current classification.
3.1 Evidence-based assessment
Insufficient evidence to decide
The snapshot is unassigned, so there is no assigned quadrant to confirm. A shared multimodal backbone predicts images and actions, supporting a one-model interpretation despite separate vision and depth modules. Sequential future-image prediction followed by goal-conditioned actions is inverse-dynamics-like; the image predictor is not action-conditioned rollout simulation. This assessment does not update the catalog. e-architecturee-formulatione-backbonee-control
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Use the future image as a control interface
Start with the two conditionals in Section 3.1. The first predicts a future observation from the current image and instruction; it never receives the actions whose consequences it would simulate. The second predicts actions from the current observation, instruction and that future image. Both conditionals use the same model, but their supervision differs: captioned videos suffice for image prediction, whereas robot demonstrations provide action labels. At deployment, Algorithm 1 generates the goal before sampling the action chunk and observes again after executing it. Consequently, the visible sequence of goal images in Figure 5 is repeatedly conditioned on new observations. Those generated intermediate images should be distinguished from the figure's observed initial and final states; selected successful examples do not measure overall reliability. e-formulatione-architecturee-controle-qualitative
Figure 3. Image generation remains causal while action queries share full attention. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the mask from its upper image/text portion to the bottom yellow action-query rows. The dark staircase in the upper portion matches the caption's causal attention, while the lower action rows extend across all action columns as well as the preceding context. The special tokens labeled x, theta and g stand for parallel decoding slots in the figure; the paper states that a complete action uses seven tokens. Section 3.3 discretizes each action dimension independently into 256 bins, but lets the queries interact through attention. Thus independent discretization does not imply that the model ignores relationships among action dimensions. e-attentione-objectivese-components
What it supports. The mask provides a concrete architectural change beyond generating a subgoal: action outputs need not be decoded one token at a time. Figure 6 tests adding this hybrid mechanism before adding visual CoT, allowing the reader to distinguish those successive system changes.
Where the evidence stops. This schematic does not specify every implementation detail of query construction or loss reduction. Equation (5) repeats a probability term without an explicit dependence on its summation index; the exact action-loss reduction should be resolved before reproduction.
5.2 Separate spatial image positions, residual codes and action queries
The model's image representation has two levels: a 16 × 16 grid of visual positions and four residual codes at each position. The LLM supplies code embeddings and the depth transformer predicts residual tokens. Reader interpretation: the 256 spatial positions help explain the limitations section's reference to generating 256 image tokens, while the backbone section counts four residual codes per position. During CoT-VLA training, the vision tower is frozen while the LLM, projector and depth transformer are optimized. Image generation remains causal; action queries use full attention and each action dimension is discretized into a bin. Parallel action decoding therefore reduces one computational burden without removing autoregressive image generation, which the authors identify as the main latency bottleneck. e-backbonee-objectivese-attentione-limitations
5.3 Ask which intervention the experiments actually isolate
Reader analysis: the strongest mechanism question is whether the predicted goal contributes useful information beyond chunking and attention. Figure 6(a) addresses this incrementally, because the CoT variant is compared with a model that already has both additions. Table 3 addresses a complementary question: can the existing policy do better when supplied with demonstrated goals? Its positive result indicates room at the goal interface, but does not tell us which generator property matters. Figure 6(b) is broader still, changing the entire pretraining stage, including robot and action-less data. The supplement makes that mixture reproducible in outline, but supplies no robot-only control with matched compute. These interventions support different conclusions and should not be combined into a single causal claim that human video improves robotic reasoning. e-componentse-oraclee-pretrain-ablatione-datae-compute
5.4 Training and inference
During training
Pretraining updates the LLM, projector and depth transformer, freezing the vision tower. Robot examples include demonstrated future images and actions; action-less videos supervise visual prediction only. Training adds visual residual-code prediction and action cross-entropy losses. e-formulatione-backbonee-objectives
Selected third-person, single-arm OpenX demonstrations are mixed with Something-Something V2 and EPIC-KITCHEN-100. Table 4 assigns Bridge 24.14% and each video dataset 3.45%. Future-frame offsets are uniformly sampled within manually chosen dataset-specific ranges; the stated action chunk size is 10. e-pretraininge-data
Downstream fine-tuning retains the same trainable modules. LIBERO and Franka use 150 epochs at constant learning rate 10^{-5}. Bridge receives additional fine-tuning on 45k trajectories until training action accuracy reaches 95%; this threshold is not robot task success. e-controle-evaluatione-compute
During inference
Algorithm 1 samples a goal, predicts actions, executes the complete chunk and obtains a fresh observation. Feedback closes the loop between chunks, with no per-action observation update shown inside execution. No candidate-plan search or learned success verifier is specified. e-control
5.5 Implementation flow
- Encode observations
The frozen VILA-U vision tower encodes 256 × 256 images into a 16 × 16 grid with residual depth four. A projector feeds discrete visual features to the LLM; a depth transformer predicts residual codes. e-backbone
- Predict a visual target
The shared model autoregressively predicts an image n frames ahead from the instruction and current image. Robot and action-less video sequences both supervise this phase. It is an instruction-conditioned goal predictor, without candidate actions as input. e-formulatione-architecture
- Predict a chunk
Given observation, instruction and goal image, action queries use full attention. Each action has seven tokens; each dimension uses 256 bins spanning its training distribution's first-to-99th-percentile interval. The least-used 256 text-vocabulary tokens encode these bins. e-attentione-objectives
6. Experiments & results
CoT-VLA makes a 7B multimodal model generate a future subgoal image before predicting a robot action chunk. The shared model learns image prediction from demonstrations and captioned videos, while action supervision requires demonstrations. Experiments support useful goal conditioning and stronger average manipulation performance, with uneven task gains, substantial inference overhead and reporting inconsistencies. Visual chain of thought here means an explicit predicted image used for control; general reasoning is not independently established.
6.1 Read the original evidence
Table 1. CoT-VLA leads three suite columns, but the reported aggregate is internally inconsistent. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare rows within a suite before reading the leftmost Average column. The source caption reports mean success and standard error across three seeds, each with 500 episodes per suite. CoT-VLA is highest in Spatial, Goal and Long; Diffusion Policy is highest in Object. LIBERO's four suites each contain ten tasks with fifty demonstrations per task, and the evaluation follows the stated pause-removal, resizing and rotation pipeline. The 69.0% Long result is an executed simulation-task outcome, not an image-quality score. Bold and underlined formatting denote first and second place in the original table. e-liberoe-evaluation
What it supports. The printed average is 81.13 ± 0.6% versus OpenVLA's 76.5 ± 0.6%, an apparent gain of 4.63 percentage points, about 6.1% relative. The Long column shows a larger difference: 69.0% versus 53.7%. The Object column prevents a claim of dominance across every suite.
Where the evidence stops. The four displayed CoT-VLA suite means average 83.925%, rather than the printed 81.13%. Both are preserved without selecting a corrected value. The source provides no reconciliation, so its aggregate needs verification before use as a benchmark reference.
Figure 4. Real-robot gains depend on the task and instruction setting. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the small task photographs to identify the six settings, then read the labeled bars with the legend above them. The dashed group is the cross-task average. The arrows below split the remaining groups into three single-instruction tasks and three multi-instruction tasks; each task has its own trained model. CoT-VLA's yellow bars show strong average performance, but competing methods win individual tasks: Diffusion Policy on putting corn in a bowl, OpenVLA on flipping the pot upright, and Octo on covering an object. The source describes ten to 150 demonstrations per task on a Franka setup absent from pretraining. e-frankae-evaluation
What it supports. CoT-VLA averages 78.8%, compared with OpenVLA's 67.3%, Diffusion Policy's 51.2% and Octo's 41.5%. The OpenVLA gap is 11.5 percentage points, approximately 17.1% relative. These values contextualize the abstract's roughly 17% real-world improvement rather than implying a 17-point increase.
Where the evidence stops. Figure 4 supplies no uncertainty bars or per-task trial counts. Section 4.2 incorrectly refers to these results as Table 4; the actual Table 4 contains dataset weights. Preserve the distinction between the six task-specific models and one universal deployment policy.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO four-suite manipulation Four suites, each with 10 tasks and 50 demonstrations per task; 500 evaluation episodes per suite per seed, three seeds. Preprocessing removes pauses, resizes images to 256 × 256 and rotates them 180 degrees. | Reported average 81.13 ± 0.6%; Spatial 87.5 ± 1.4%, Object 91.6 ± 0.5%, Goal 87.6 ± 0.6%, Long 69.0 ± 0.8%. Success rate, mean ± standard error | OpenVLA average 76.5 ± 0.6%, Long 53.7 ± 1.3%; Diffusion Policy leads Object at 92.5 ± 0.7%. Printed averages imply +4.63 percentage points, about 6.1% relative. However, CoT-VLA's four suite means average 83.925%, not 81.13%; the source does not reconcile this discrepancy. e-evaluatione-libero |
| Franka-Tabletop adaptation Previously unseen Franka setup; six tasks, separate model per task, 10–150 demonstrations each; three single-instruction and three multi-instruction settings. | 78.8% Average task success rate | OpenVLA 67.3%, Diffusion Policy 51.2%, Octo 41.5%. +11.5 percentage points over OpenVLA, approximately 17.1% relative. CoT-VLA does not win every task. Per-task trial counts and uncertainty are not provided here. e-evaluatione-franka |
| Bridge-V2 generalization WidowX, 10 trials per category, partial-credit scoring; baseline checkpoints are used while CoT-VLA receives additional Bridge fine-tuning. | Visual 65%, Motion 60%, Semantic 50%, Language 70%. Reported success rate with partial credit | OpenVLA: 75%, 45%, 40%, 75%; SUSIE: 30%, 10%, 20%, 40%; Octo: 35%, 10%, 0%, 40%, respectively. Motion and semantic gains coexist with lower visual and language scores. Small partially scored samples and different adaptation histories limit comparisons. e-evaluatione-baselinese-bridge |
| LIBERO component ablation Same VILA-U backbone; successive additions of chunking, hybrid attention and visual CoT on Spatial and Goal. | Spatial: 67.5 → 73.3 → 81.8 → 87.5%; Goal: 54.9 → 74.9 → 79.7 → 87.6%. Success rate | The final CoT addition improves the chunked, hybrid-attention model by 5.7 and 7.9 percentage points. Supports successive additions; this is not a full factorial study and shows no uncertainty bars. e-components |
| Franka-Tabletop pretraining ablation Combined OpenX/action-less-video pretraining versus directly fine-tuning VILA-U on Franka demonstrations. | 78.8% with pretraining versus 53.7% without. Success rate | +25.1 percentage points; authors report 46.7% relative improvement. The intervention does not isolate action-less video's contribution. e-pretrain-ablation |
| Ground-truth-goal intervention on unseen Franka tasks Novel scallion-to-apple-covered-book and cauliflower-to-bear-covered-book compositions; one demonstration supplies goals and five trials evaluate each task per condition. | Table 3 columns 'Sub-task 1'/'Sub-task 2': generated goals 20%/0%; ground-truth goals 60%/40%. Success rate | +40 percentage points in each column. Privileged goals reveal usable action capability with residual failures. Table subtask labels differ from the text's two-task description; no stronger mapping or scalable-reasoning conclusion is assumed. e-oracle |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Figure 6(a–b). Visual goals add gains after chunking and hybrid attention, while the pretraining experiment changes an entire stage. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Panel a should be read in legend order within each suite: vanilla VLA, action chunking, hybrid attention and finally CoT. Section 4.3 says the variants share the VILA-U backbone. Compare the blue and yellow bars to examine the incremental visual-CoT addition after the other two modifications. Panel b asks a different question: it compares direct Franka fine-tuning against the complete OpenX-plus-video pretraining stage followed by fine-tuning. The supplement records Bridge at 24.14% of that mixture and each action-less video dataset at 3.45%; panel b does not provide a robot-only pretraining control. e-componentse-pretrain-ablatione-data
What it supports. Adding CoT increases Spatial from 81.8% to 87.5% and Goal from 79.7% to 87.6%, gains of 5.7 and 7.9 percentage points. The separate pretraining comparison rises from 53.7% to 78.8%, a 25.1-point gain, described by the authors as 46.7% relative.
Where the evidence stops. The bars show no uncertainty. Successive additions are not a full factorial ablation, and panel b jointly changes robot/video pretraining. It cannot establish the marginal benefit of action-less videos or disentangle extra training compute from data composition.
Table 3. Supplying demonstrated goals improves performance, but leaves many failures. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read vertically within each column: the model acts with its generated goal in the first row and with a ground-truth goal in the second. Section 4.4 describes two unseen task compositions involving moving a scallion to an apple-covered book and a cauliflower to a bear-covered book. For each task, one demonstration supplies ground-truth goals and five evaluation trials are conducted per condition. The table calls its columns Sub-task 1 and Sub-task 2, while the prose describes two tasks; retain those printed labels without inferring an undocumented finer-grained scoring protocol. This is a goal substitution experiment rather than training a better generator. e-oraclee-objectivese-control
What it supports. The reported change is 20% to 60% in one column and 0% to 40% in the other: forty percentage points each. Better goals can expose action capabilities that generated goals fail to activate. Residual failure with demonstrated goals shows that goal prediction is not the sole bottleneck.
Where the evidence stops. Five trials per condition provide little precision. Demonstrated goals are privileged information and may improve several properties at once, including task identity and geometry. This comparison does not prove that visual realism alone causes better control or that new-task generalization is solved.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors report 7× average inference slowdown at chunk size 10 from generating 256 image tokens, poorer image quality than diffusion, chunk-boundary discontinuities and missing high-frequency feedback. Entirely new-task goal generation remains difficult. e-limitationse-backbone
Action training uses demonstrated future images, whereas deployment uses generated goals. The oracle comparison tests this interface, but cannot isolate realism, task correctness or distribution shift as the causal factor. e-objectivese-controle-oracle
7.2 Questions for discussion
- Which aggregation produced Table 1's reported CoT-VLA average?
- Would matched-compute robot-only pretraining retain the mixed-video model's gains?
- Can shorter chunks preserve goal-conditioning gains while reducing feedback-related failures?
8. Reproducibility audit
8.1 Requirements and known gaps
The supplement specifies pretraining learning rate 10^{-4}, cosine decay, batch 2048 and 10 epochs; 12 nodes with eight A100 GPUs each, totaling 11K A100 GPU-hours. Fine-tuning takes 10–24 hours on one A100 node. Exact optimizer settings, software versions and decoding parameters are not supplied. e-compute
Resolve source ambiguities: Figure 2 labels actions through n although equations use n for image horizon; Algorithm 1 executes m+1 indexed actions despite the m-action prose. Supplement horizon prose reverses lower/upper names while Table 4 lists ascending endpoints. Table 5 labels 10 'Action Token Size'. e-architecturee-formulatione-controle-datae-compute
Proposed checks: freeze the action decoder while swapping generated, demonstrated and mismatched goals; separately compare robot-only versus mixed-video pretraining at matched updates and compute. These test goal dependence and video benefit separately. e-oraclee-pretrain-ablatione-data
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Hold the policy fixed and intervene on goal content
Reader-proposed experiment, not performed: fix a downstream checkpoint, action queries, chunk length, initial-state set and instruction. Compare generated goals, demonstrated goals, and goals shuffled across tasks; also include an unchanged current-image goal. Use equal trial counts over several seeds and record task success, wrong-object actions and grasp failures. Match goal resolution and measure inference latency separately so privileged image supply does not hide its speed advantage. A reproducible demonstrated-over-generated gain would confirm remaining goal-interface headroom; a fall under shuffled goals would show that goal content influences control. Little change under either intervention would challenge the claimed role of the visual intermediate. e-oraclee-formulatione-controle-bridgee-limitations
Check 2: Isolate action-less video from the rest of pretraining
Reader-proposed experiment, not performed: initialize both models from the same VILA-U checkpoint, use the same trainable modules, losses, token budget and optimizer-update count, and compare the published robot/video mixture with a robot-only mixture that reallocates the two 3.45% video weights proportionally to robot datasets. Retain each robot dataset's documented horizon range, then fine-tune on identical Franka demonstrations for the reported 150 epochs. Record held-out goal task correctness, robot success and compute. Include multiple seeds and confidence intervals. Higher success for the mixed-video model at matched compute would support a video-specific benefit; equal results would weaken that interpretation of Figure 6(b), even if pretraining still helps. e-pretrain-ablatione-datae-pretraininge-backbonee-objectivese-compute
8.3 Reading coverage
Visual audit: Visually inspected the title, all six figures, all five tables, Equations (1)–(6), Algorithm 1, evaluation protocols, limitations and the embedded implementation supplement on PDF pp. 1–8 and 13. Each of the six final original crops was opened and checked for readable labels, complete content and source fidelity. Figure 2's arrows and Figure 3's mask were checked against captions, equations and control pseudocode; notation and aggregation discrepancies are disclosed in the guides. Full-page images include the uncropped Bridge-V2 results, qualitative examples and appendix data/compute details needed to review retained claims. Reference pages 9–12 were read as text only. Separate videos, external resources and any later edition remain uninspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 13. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- PDF p. 1: title, authors, affiliations, arXiv revision marker and Abstract
- PDF pp. 1–2: 1. Introduction
- PDF pp. 2–3: 2. Related Work
- PDF p. 3: 3. CoT-VLA and 3.1. Visual Chain-of-Thought Reasoning
- PDF pp. 4–5: 3.2. The Base Vision-Language Model and 3.3. Training Procedures, including Algorithm 1
- PDF pp. 5–7: 4. Experiments, 4.1. Experimental Setup and 4.2. Evaluations Results
- PDF pp. 7–8: 4.3. Ablation Study and 4.4. Better Visual Reasoning Helps
- PDF p. 8: 5. Conclusion, Limitations and Future Work
- PDF pp. 9–12: References
- PDF p. 13 (supplement p. 1): 6. Implementation Details, 6.1. Data Details, 6.2. Hyperparameters and 6.3. Training
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout. This acquisition limitation was addressed by visually inspecting all six figures, all five tables and the supporting method pages in the supplied PDF.
- Separate supplemental material availability has not been fully verified. The embedded supplementary page was read; the separately referenced execution video was not supplied or inspected.
- Identity verified on PDF p. 1: the exact title and all 15 authors match the catalog. The observed artifact is arXiv:2503.22020v1 [cs.CV], 27 March 2025. The catalog labels the work CVPR 2025; no separate venue edition or later revision was supplied, and equivalence to either is not asserted.
- All five supplied text chunks were read, covering all 13 PDF pages. Visual inspection covered PDF pp. 1–8 and 13; reference-only pages 9–12 were read as text.
- Code, external project resources and cited papers were not inspected. No training, deployment or reproduction experiments were run.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-qualitativePDF p. 7, Figure 5 and caption
Selected LIBERO, Bridge-V2 and Franka examples distinguish observed initial/final states from generated intermediate goals. Each goal is conditioned on the instruction and latest observation; full trajectories are deferred to a supplementary video.
Go to primary source ↓e-identityPDF p. 1, title/byline, affiliation line and left-margin arXiv marker
The title and 15-author sequence match the catalog. The page names NVIDIA, Stanford University and MIT and displays arXiv:2503.22020v1, 27 March 2025.
Go to primary source ↓e-problemPDF pp. 1–2, Abstract, Figure 1 and Introduction
The authors motivate future-image intermediate reasoning and learning from action-less videos alongside demonstrations.
Go to primary source ↓e-architecturePDF p. 3, Figure 2 and caption
One CoT-VLA block generates a goal using causal attention and actions using full attention; execution feeds back a new observation. Action endpoints are labeled a_1 and a_n.
Go to primary source ↓e-formulationPDF p. 3, Section 3.1, Equations (1)–(3) and following paragraph
Robot data include actions, images and language; video data lack actions. Shared P_theta predicts a future image and goal-conditioned actions. Text calls inclusive t through t+m indices m actions.
Go to primary source ↓e-backbonePDF p. 4, Sections 3.2 and 3.3, opening paragraphs
7B VILA-U uses 256 × 256 images, 16 × 16 × 4 residual tokens, a projector and depth transformer. Vision stays frozen while the other three components train.
Go to primary source ↓e-attentionPDF p. 4, Figure 3, caption and Action Tokens Prediction paragraph
The mask is causal for text/images and fully connected across action-query rows; special x, theta and g tokens enable parallel decoding.
Go to primary source ↓e-objectivesPDF p. 4, Visual Tokens Prediction and Action Tokens Prediction, Equations (4)–(6)
The source specifies residual prediction, seven action dimensions, 256 percentile-bounded bins, repurposed vocabulary tokens and summed losses. Equation (4) omits text-described h_j conditioning; Equation (5)'s summand does not explicitly vary with i.
Go to primary source ↓e-pretrainingPDF p. 4, Section 3.3, Pretraining Phase
Selected OpenX data use third-person views and single-arm control. Video sources are EPIC-KITCHENS and Something-Something V2; goal horizons are uniformly sampled per dataset and chunk size is 10.
Go to primary source ↓e-controlPDF p. 5, Adaptation Phase and Algorithm 1
Fine-tuning keeps vision frozen. Deployment samples a goal and action sequence, executes j=0 through m, advances by m+1 and observes again.
Go to primary source ↓e-evaluationPDF pp. 5–6, Section 4.1, LIBERO, Bridge-V2 and Franka-Tabletop setup paragraphs
Four LIBERO suites each have 10 tasks and 50 demonstrations; preprocessing removes pauses, resizes and rotates images. Bridge uses 45k trajectories and a 95% training-accuracy threshold. Franka has six tasks with 10–150 demonstrations each and an unseen setup.
Go to primary source ↓e-liberoPDF p. 5, Table 1, all rows/columns and caption; PDF p. 6, Section 4.2 LIBERO
CoT-VLA reports average 81.13 ± 0.6% and suites 87.5/91.6/87.6/69.0%; OpenVLA averages 76.5 ± 0.6%. Three seeds have 500 episodes per suite each. CoT-VLA's listed suite means arithmetically average 83.925%.
Go to primary source ↓e-frankaPDF p. 6, Figure 4, all bars, legend and caption; Section 4.2 Franka-Tabletop
Separate per-task models average 78.8% CoT-VLA, 67.3% OpenVLA, 51.2% Diffusion Policy and 41.5% Octo. The graph separates single- and multi-instruction tasks. The paragraph incorrectly points to Table 4; these results are Figure 4.
Go to primary source ↓e-baselinesPDF p. 6, Baselines paragraph
Diffusion Policy trains from scratch with chunking, proprioception and language embeddings. OpenVLA/Octo use published Bridge checkpoints and fine-tuning elsewhere. SUSIE uses its Bridge checkpoint.
Go to primary source ↓e-bridgePDF p. 7, Table 2 and caption; PDF p. 6, Section 4.2 Bridge-V2
Four categories have ten trials each with partial credit. CoT-VLA scores 65/60/50/70%, OpenVLA 75/45/40/75%, SUSIE 30/10/20/40% and Octo 35/10/0/40%.
Go to primary source ↓e-componentsPDF p. 7, Section 4.3 Visual CoT, Hybrid Attention, and Action Chunking; PDF p. 8, Figure 6(a)
Same-backbone variants add chunking, hybrid attention and CoT. Spatial: 67.5/73.3/81.8/87.5%; Goal: 54.9/74.9/79.7/87.6%. Bars lack uncertainty.
Go to primary source ↓e-pretrain-ablationPDF pp. 7–8, Section 4.3 Pretraining; PDF p. 8, Figure 6(b)
Franka success rises from 53.7% without combined OpenX/video pretraining to 78.8% with it; authors call this 46.7% relative improvement.
Go to primary source ↓e-oraclePDF p. 8, Section 4.4 and Table 3, both rows and columns
Two unseen task compositions are tested for five trials each per condition, using one demonstration per task for ground-truth goals. Columns labeled Sub-task 1/2 report generated 20/0% versus ground-truth 60/40%.
Go to primary source ↓e-limitationsPDF p. 8, Section 5, limitations paragraph
Authors report 256 generated image tokens, 7× slowdown at chunk size 10, poorer visual quality than diffusion, discontinuities, missing high-frequency feedback and limited new-task goal generalization.
Go to primary source ↓e-dataPDF p. 13 (supplement p. 1), Section 6.1 and Table 4
Weights/endpoints: Bridge 24.14%, 5/10; RT-1 6.90%, 5/10; TOTO, VIOLA, RoboTurk, Jaco Play, Berkeley Autolab UR5 and Berkeley Fanuc each 10.34%, respectively 20/24, 15/20, 1/2, 10/15, 5/10 and 10/15. The two video datasets each use 3.45%, 5/7. Prose reverses lower/upper roles relative to ascending entries.
Go to primary source ↓e-computePDF p. 13 (supplement p. 1), Sections 6.2–6.3 and Table 5
Pretraining: 1e-4 learning rate, cosine decay, batch 2048, 256 × 256 images, 'Action Token Size' 10, 10 epochs. LIBERO/Franka: constant 1e-5, 150 epochs. Training: 12 × 8 A100 GPUs, 11K A100 GPU-hours; fine-tuning one A100 node, 10–24 hours. Optimizer name, software version and sampling recipe are absent.
Go to primary source ↓8.5 Primary sources
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models ↗
PDF · 8,072 extracted words
Source fingerprint
20081e8c7efca4e4bf7f8d6e83ee69793a30961b5069975c0980dddb2cf211f0