Unified Video Action Model
1. Paper overview
In one sentence: UVA shares a video–action representation but separates diffusion decoding, making visual supervision compatible with fast policy inference while leaving task-specific performance and latency tradeoffs. identitymotivationarchitecturedecoders-lossreal-resultshuman-videomasking
| At a glance | What to know |
|---|---|
| Research problem | Source description High-resolution video prediction supplies scene supervision but can slow robot control and propagate image-generation errors into actions. UVA asks whether a shared representation can retain that supervision while decoding dense actions independently. motivation |
| Core mechanism | Source description A shared Transformer feeds two lightweight diffusion heads, supporting joint future prediction and action-only deployment. architecturedecoders-loss |
| A key reported result | Real-world UMI manipulation: Multitask Cup/Towel/Mouse: 0.65/0.70/0.80; single-task Cup: 0.85. Success rate.. ARX X5; 20 rollouts per task; multitask evaluation totals 60 OOD rollouts. DP-UMI: 0.50/0.70/0.40; single-task Cup: 0.95. Cup and Mouse improve by 15 and 40 percentage points; Towel ties. Single-task Cup declines by 10 points. Small test counts limit certainty. real-resultsreal-protocol |
| Reading caution | Reader analysis Reported gains are not universal: visual disturbances still cause failures, including matching object/background colors. Goal-color robustness improves, but UniPi scores higher with background objects. Figure 6 suggests history robustness without reporting error bars. limitationsrobustness |
Core contributions
- Source description
A shared Transformer feeds two lightweight diffusion heads, supporting joint future prediction and action-only deployment. architecturedecoders-loss
- Source description
Masked input/output configurations let the architecture serve as a policy, video generator, forward model, inverse model, or combined policy and planner. task-masks
Figure 2. Joint prediction shares latent computation while preserving separate routes to images and actions. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the diagram from left to right. Each historical image is flattened into N tokens, while its denser action chunk is repeated to match that count. The masked future observation joins these features along the channel dimension; different time positions are then concatenated before the Transformer. On the right, each visual latent token conditions video diffusion, whereas all tokens are aggregated for action diffusion. The epsilon symbols denote predicted noise used by the diffusion process. Section III-C explains that iterative denoising produces the final outputs. The two branches share their conditioning representation without requiring generated pixels to pass into the action branch. architecturemasked-latentsdecoders-loss
What it supports. The architectural benefit is selective decoding: the action head can use a representation trained with visual supervision even when no future image is generated at deployment. This supports the catalog's joint-prediction classification through an explicit shared Transformer and latent representation, while distinguishing it from a sequential video-then-action pipeline.
Where the evidence stops. The masked future images shown at the input are available during training. Policy inference begins without known future-image tokens; reading this diagram as requiring ground-truth future observations at deployment would introduce information the policy does not have.
2. Motivation
2.1 The problem and the proposed response
High-resolution video prediction supplies scene supervision but can slow robot control and propagate image-generation errors into actions. UVA asks whether a shared representation can retain that supervision while decoding dense actions independently. motivation
2.2 What this reading follows
Predicting what a robot will see can help predict what it should do, but generating every future image can be expensive. UVA places the shared reasoning before two separate diffusion heads. Training can supervise both images and actions; deployment can request only actions. This reading follows that separation from the architecture to policy results, video quality, runtime, and masking diagnostics. The strongest practical evidence comes from multitask comparisons, alongside clear exceptions in single-task manipulation. The supplied v3 PDF also includes action-free human-video training and masking studies, which qualify the broader claims made in its discussion. identitymotivationarchitecturedecoders-lossreal-resultshuman-videomasking
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The shared Transformer produces latents consumed by both video and action heads, supporting One Model × Joint prediction architecturally. Separate output heads do not form a video-to-action pipeline. Inverse dynamics is an additional mask configuration; ordinary policy inference skips video, while planning explicitly uses predicted outcomes. architecturedecoders-losstask-masksplanning
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Supervise a shared representation before choosing an output
Begin with the alignment problem. Camera observations need rich spatial content, whereas an action chunk contains denser temporal commands. UVA turns both histories into matching token counts, combines them with masked future-image features, and uses a Transformer to construct shared latents. During training, visual denoising asks these latents to preserve information useful for reconstructing the scene, while action denoising asks them to support the demonstrated motion. Neither loss forces a generated image to become the action decoder's input. That distinction explains how policy deployment can remove the video branch without discarding the representation learned under visual supervision. The action-only ablation tests the value of the larger training design, but its removal of generation capacity means it is not a clean loss-only intervention. architecturemasked-latentsdecoders-lossbaseline-settingssim-results
5.2 Choose the mask, then distinguish the two generation loops
The application mask determines which quantities are conditions and which must be predicted. A policy requests actions from history; a forward model receives actions and predicts observations; inverse dynamics receives observations and predicts actions. Unused components are replaced with learned masks, and only relevant losses apply. Video generation introduces a second choice: how many passes to spend filling visual-token positions. Each pass generates positions across all future frames, rather than completing the next frame in strict temporal order. Within a diffusion head, separate denoising iterations turn noise into an action chunk or visual token. Table IV's one-versus-eight comparison changes autoregressive filling, while Table VII's 16-versus-100 comparison changes denoising. Confusing these counts would produce an incorrect implementation and an invalid speed comparison. task-masksautoregressionmasked-latentsdecoders-lossvideo-resultsruntime
5.3 Separate representation benefits from online model-based control
Ordinary UVA policy inference uses the action head directly and does not need predicted future pixels. The block-pushing experiment uses a different route: an external DP-C policy proposes candidate trajectories, UVA predicts their visual consequences, and an image-based score chooses one for partial execution before replanning. Its increase from 0.38 to 0.60 average success is therefore evidence for outcome-guided selection, while the UMI manipulation table tests directly executed policies. The inverse-dynamics table measures pose reconstruction, and the video table measures feature-distribution similarity. Reader interpretation: these complementary evaluations establish several useful capabilities, but their metrics cannot be pooled into one measure of world-model correctness. A useful next experiment would test whether improvements in predicted outcomes actually improve candidate rankings under a fixed proposal set. decoders-lossplanningreal-resultsinversevideo-results
5.4 Training and inference
During training
Training predicts added noise in action chunks and visual tokens. Applicable losses depend on the sampled application; unused components receive learned mask tokens. The model builds on pretrained MAR-B with substantial modifications. decoders-losstask-masksautoregression
Real-world multitask training uses 500 public UMI episodes per task, 1,500 total. A separate appendix experiment pretrains on 3,175 human videos, then fine-tunes jointly with LIBERO-10. The source does not fully specify optimizer, training duration, default horizons, or which pretrained modules are frozen. real-protocolhuman-videoarchitectureautoregression
During inference
Policy mode masks unavailable futures and decodes actions without generating video. Table I predicts 16 actions and executes eight, except OpenVLA, which is run eight times. Video mode bypasses action decoding and progressively fills token positions across all frames; autoregressive passes and diffusion denoising iterations are different loops. decoders-losssim-protocolautoregression
For block-pushing planning, DP-C supplies 100 candidate 16-action trajectories. UVA predicts images for distance-based target scoring; six selected actions execute before replanning. Inverse mode instead conditions on observations to infer actions, evaluated as camera poses. planninginverse
5.5 Implementation flow
- Align history
A pretrained kl-f16 VAE encodes each image; projection produces N visual tokens. Repeated historical action chunks are projected to matching token counts. Each chunk contains L actions of dimension m; history and future horizons are equal in the experiments. architecture
- Fuse modalities
Concatenate history-image, action, and masked future-image features channel-wise, then concatenate across time. A Transformer produces joint latents Z. Matching spatial positions are masked across frames to limit leakage. Language tasks add repeated CLIP text tokens; the text also describes cross-attention. architecturemasked-latents
- Decode separately
Each visual latent conditions a patch diffusion decoder, followed by VAE image reconstruction. Convolution and an MLP aggregate all tokens for action diffusion. Diffusion iterations run in these heads; the shared Transformer is not the iterative denoiser. decoders-loss
6. Experiments & results
UVA learns shared video–action latents, then decodes future images and action chunks through separate diffusion heads. Policy deployment skips video decoding while retaining joint-training supervision. Multitask gains coexist with single-task losses and deployment latency tradeoffs.
6.1 Read the original evidence
Table I. Multitask gains and the joint-training ablation sit beside a Toolhang loss and faster π0 inference. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the four task columns, keeping single-task and multitask settings separate. UVA leads the listed methods on PushT, PushT-M, and Libero10, while DP-C leads on Toolhang. Next compare UVA with UVA-action: that row removes video generation and trains solely as a policy, so it is the nearest mechanism-oriented comparison. Read the speed column independently: the caption specifies 16 predicted actions and eight executed actions, with OpenVLA invoked eight times to match executed steps. These simulation timings use L40 hardware. The source calls multitask scores average rewards in the protocol and success rates in its caption; retain that ambiguity. sim-resultssim-protocolbaseline-settingsruntime
What it supports. On PushT-M, UVA reports 0.88 versus DP-C's 0.68 and UVA-action's 0.46. On Toolhang, UVA's 0.88 falls below DP-C's 0.95. The table therefore supports substantial benefits in some settings, together with a concrete counterexample to any claim of uniformly superior policy performance.
Where the evidence stops. The π0 variants use different pretraining, inputs, and Libero fine-tuning scope. The appendix also contradicts itself on Libero10 episode counts. UVA-action changes both generation capacity and supervision, so this row alone cannot attribute the gain exclusively to the video loss.
Table II. Real robot results favor UVA on two multitask tasks, with costs in single-task success and runtime. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the two Cup columns as separate experiments. The first trains a cup-only policy; the second belongs to a model trained on all three manipulation tasks. Multitask training uses 500 public demonstration episodes per task, and each policy is evaluated on 20 rollouts per task in changed real-world conditions. Both methods are deployed on an ARX X5 arm. Compare each task within its own column before considering speed. The last column measures a 16-action trajectory with 16 diffusion denoising iterations on an RTX 3080. It is not an action frequency or the complete duration of a manipulation episode. real-resultsreal-protocolruntime
What it supports. UVA improves multitask Cup from 0.50 to 0.65 and Mouse from 0.40 to 0.80, while Towel stays at 0.70. Single-task Cup instead declines from 0.95 to 0.85, and inference rises from 70 to 95 ms. These executed robot results support a useful but conditional multitask advantage.
Where the evidence stops. Only 20 trials support each task entry, and the table supplies no uncertainty estimates. The authors' explanation involving recovery demonstrations and long history is an interpretation of the single-task loss, not an isolated causal test.
Table IV. Additional token-generation passes improve FVD, with different one-pass behavior across datasets. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Lower values are better in both columns. The rows labeled one step and eight steps count autoregressive token-filling passes, not diffusion denoising iterations inside the video head. At each pass UVA predicts a set of spatial token positions across all future frames; later passes condition on tokens already generated. The experiment computes FVD from 500 generated videos per method. Compare the two UVA rows within each dataset to see the effect of iterative filling, then compare against UniPi. The Libero10 and Cup Arrangement columns are distinct datasets and should not be averaged into an unspecified aggregate score. video-resultsautoregressiondecoders-loss
What it supports. Eight passes reduce UVA's FVD from 89.36 to 51.10 on Libero10 and from 51.34 to 29.72 on Cup Arrangement. Eight-pass UVA beats UniPi in both columns. The one-pass model already beats UniPi on Cup Arrangement but remains worse on Libero10, showing why the step setting matters.
Where the evidence stops. FVD compares distributions of video features; it does not demonstrate physically correct action-conditioned transitions or successful execution. The table does not attach a matched latency measurement to the one-versus-eight-pass comparison, so it cannot establish its full speed–quality curve.
Table VII. The runtime budget shows both the savings from skipping video and the remaining encoder/Transformer cost. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the upper block as component costs and the lower block as reported end-to-end model totals. The ordinary 16-step policy combines 40 ms of VAE encoding, 40 ms of Transformer attention, and 15 ms of action diffusion, giving 95 ms. Video diffusion adds 100 ms in its 16-step row, consistent with the 195 ms Policy+Planner total. The 100-step rows show a larger decoding bill. Flash-Attention rows are alternative configurations, not extra modules to add to ordinary attention. Here, steps in the row labels refer to denoising iterations; the generated action trajectory separately contains 16 actions. runtimereal-resultsdecoders-loss
What it supports. Skipping video decoding has a direct computational benefit in the reported decomposition: the 16-step total falls from 195 ms for Policy+Planner to 95 ms for policy alone. Even then, encoding and the Transformer account for most of the remaining time. Faster action diffusion alone therefore cannot eliminate the main fixed costs.
Where the evidence stops. The caption generically states 100 diffusion steps, but explicit rows and Section X-D cover both 16 and 100. Follow those row labels. Flash-Attention totals are reported alternatives, not Table II's deployed configuration; no corresponding success comparison is supplied here.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| PushT-M multitask policy and joint-training ablation Varying target positions; best checkpoint; 50 simulated rollouts. | UVA 0.88. Average reward in V-A; caption calls it success rate; higher is better. | DP-C 0.68; UVA-action 0.46. The baseline gap is 0.20 score units. Removing video generation changes both architecture and supervision, so the ablation does not isolate a loss-only effect. sim-resultssim-protocolbaseline-settings |
| Libero10 multitask policy Ten language-conditioned tasks; main text specifies 50 environments per task. | UVA 0.90. Reported task score; V-A calls it average reward. | π0 0.85; π0-FAST 0.60; UVA-action 0.86. The 0.05 gap to π0 is not a matched pretraining/input comparison: π0 uses all Libero, wrist images, and proprioception. Appendix episode-count wording is inconsistent. sim-resultssim-protocolbaseline-settings |
| Single-task simulation policies Separate policies; best checkpoints; 50 rollouts per task. | PushT 0.98; Toolhang 0.88. Success rate. | DP-C 0.91 and 0.95, respectively. The advantage is task-dependent; UVA loses on Toolhang. sim-resultssim-protocol |
| Real-world UMI manipulation ARX X5; 20 rollouts per task; multitask evaluation totals 60 OOD rollouts. | Multitask Cup/Towel/Mouse: 0.65/0.70/0.80; single-task Cup: 0.85. Success rate. | DP-UMI: 0.50/0.70/0.40; single-task Cup: 0.95. Cup and Mouse improve by 15 and 40 percentage points; Towel ties. Single-task Cup declines by 10 points. Small test counts limit certainty. real-resultsreal-protocol |
| Video generation Libero10 and Cup Arrangement; 500 generated videos per method. | Eight autoregressive passes: 51.10/29.72; one pass: 89.36/51.34. FVD; lower is better. | UniPi: 56.55/71.37. Eight passes improve both datasets; one pass loses to UniPi on Libero10. FVD measures video-distribution similarity, not executed manipulation success. video-results |
| Block-pushing trajectory selection Four specified color assignments, ten trials each; DP-C candidates ranked by predicted outcomes. | UVA-guided 0.60. Average success rate. | DP-C alone 0.38; simulator-guided 0.75. Learned forward prediction improves execution but remains below simulator-guided selection. The simulator figure is an empirical reference limited by proposals and detection. planning |
| Inverse dynamics on UMI Cup Arrangement Unseen camera-pose sequences compared with motion capture. | 0.75 cm / 1.11°. Position/rotation L2 errors, as reported; lower is better. | UniPi 1.92 cm / 2.21°; visual-inertial SLAM 0.41 cm / 0.30°. UVA improves learned pose inference but does not match mapped SLAM; this test does not measure robot task completion. inverse |
| Policy inference latency RTX 3080; 16-action trajectory; 16 denoising iterations. | UVA 95 ms. Inference time; lower is better. | DP-UMI 70 ms; UVA Policy+Planner 195 ms. Skipping video avoids substantial decoding cost, yet UVA is slower than DP-UMI. Separate L40 simulation timings must not be mixed with this deployment comparison. runtimereal-results |
| Additional action-free human video Libero10; human-video pretraining followed by mixed-data fine-tuning; 500 tests. | UVA + Human Data 0.91. Reported policy score; higher is better. | UVA 0.90; 30-test setting separately reports 0.97 versus 0.93. The larger evaluation shows a small gain without uncertainty estimates; it does not establish web-scale transfer. human-video |
| Masking-strategy diagnostic on PushT-M Application-dependent versus independent input masks at varied ratios. | Independent 50%: 0.87 success, 200.08/203.59 FVD, 13.55 L2. Policy success; video and forward FVD; inverse L2 error. | Dependent 25%: 0.84 success, 127.09/129.77 FVD, 17.54 L2. The preferred mask differs by objective. Inverse-error units and uncertainty are unspecified. masking |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table IX. Masking choices trade policy and inverse performance against the quality of predicted videos. Original paper, p. 16 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row using its own arrow: policy success should increase, while FVD and inverse L2 error should decrease. The two column groups change how masks are chosen; the three columns within each group change the ratio. Section X-F describes application-dependent masking as respecting a task's input/output structure, and independent masking as randomly removing inputs regardless of task semantics. Compare within groups before comparing across them. The table's forward-dynamics metric is FVD on action-conditioned videos. It is a different evaluation from the block-pushing execution-success experiment and should not be substituted for that result. maskingtask-masksplanning
What it supports. Independent 50% masking gives the highest policy score, 0.87, and lowest inverse error, 13.55. Dependent 25% masking instead gives the lowest video and forward FVD values, 127.09 and 129.77. Reader interpretation: this diagnostic does not identify one universally best masking setting across the model's advertised functions.
Where the evidence stops. The appendix provides neither units for this inverse L2 error nor uncertainty estimates. These are additional masking experiments on PushT-M; the report cannot infer the full default training-mask distribution or extrapolate the ranking to other benchmarks.
7. Analysis & limitations
7.1 What the evidence leaves open
Reported gains are not universal: visual disturbances still cause failures, including matching object/background colors. Goal-color robustness improves, but UniPi scores higher with background objects. Figure 6 suggests history robustness without reporting error bars. limitationsrobustness
Protocol ambiguities remain: multitask rewards are also called success rates; X-B says ten Libero10 runs per task yet 500 total. Main-text and Table VIII support 50 per task. Tables omit uncertainty, and best-checkpoint reporting requires care about selection bias. sim-protocolsim-resultshuman-video
Section IX leaves large-scale actionless-video training to future work; X-E nevertheless includes a small human-video experiment. Sound/force extensions are proposals. Neither establishes broad generalization beyond the tested settings. limitationshuman-video
7.2 Questions for discussion
- Would a capacity-matched action-only control retain the joint model's advantage?
- When does improved video FVD improve candidate ranking and executed success?
8. Reproducibility audit
8.1 Requirements and known gaps
A faithful implementation needs kl-f16, modified MAR-B, action/video heads, CLIP conditioning, mask scheduling, and dataset-specific action/image alignment. The PDF does not provide a complete training configuration, frozen-module policy, software versions, or training-compute budget. architecturemasked-latentsdecoders-lossautoregression
Preserve seeds, checkpoint selection, UMI sampling, and evaluation success definitions. Match L40 simulation versus RTX 3080 deployment hardware and 100 versus 16 denoising iterations. Proposed checks should separate video-supervision benefits from decoder-capacity changes and time video-on/off inference under identical settings. sim-protocolreal-protocolbaseline-settingsruntime
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate the benefit of future-image supervision
Reader-proposed check: train matched PushT-M variants with identical data, initialization, history length, Transformer capacity, action head, training updates, and task-mask schedule. Retain the video branch in both variants but set its loss weight to zero in one; separately include the paper's branch-removal UVA-action control. Predeclare a validation-based checkpoint rule and evaluate the same held-out target configurations across multiple training seeds, reporting uncertainty. If the joint-loss model consistently outperforms the capacity-matched zero-video-loss model, that supports a supervision effect. If only the branch-removal comparison produces a gap, the original ablation cannot establish that explanation. No experiment has been run for this report. sim-resultssim-protocolbaseline-settingsdecoders-losstask-masks
Check 2: Test decoder bypass without changing the action computation
Reader-proposed check: on one RTX 3080, hold the checkpoint, histories, task mask, action-noise seed, action horizon, and action denoising count fixed at the reported 16-action/16-iteration configuration. Time synchronized inference with video decoding disabled and with the video head executed as an additional output, keeping its tokens out of action conditioning. Compare action arrays before any robot execution and report warmed-up median and tail latency, including the VAE and Transformer. Bypass should leave actions unchanged within numerical tolerance while removing video-decoder cost. A changed action sequence would reveal unintended coupling; failure to reduce latency would challenge the practical interpretation of Table VII. This check does not reproduce candidate-search planning. architecturedecoders-lossruntimereal-results
8.3 Reading coverage
Visual audit: Visually inspected the title/version block; all nine figures and nine tables; main method, losses, evaluation, limitations, and supplementary configuration pages. Read all seven text chunks, including acknowledgment, references, and Section X-A–F. Inspected all six final original crops individually. Figure 2's input concatenation, decoder branches, and noise outputs were cross-checked against Sections III-A–C; Figure 9's shared spatial filling against III-B and X-A. Table VII's caption/row mismatch and the Libero10 protocol inconsistency are disclosed. Reference-only pages 11–13 were read as text. External code, websites, datasets, earlier revisions, and separate supplements were outside the supplied visual material.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 14, 15, 16. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract; I. Introduction; II. Related Work
- III-A. Encode History; III-B. Masked Autoencoder for Observation Prediction; III-C. Decoupled Video and Action Diffusions; III-D. Masked Training with Flexible Objectives
- IV. Evaluation; V-A–D. UVA as Policy
- VI. UVA as a Video Generator; VII. UVA as a Forward Dynamics Model; VIII. UVA as an Inverse Dynamic Model; IX. Discussion
- Acknowledgment and References
- X-A–F. Supplementary Materials: autoregressive generation, simulation, real-world benchmarks, inference timing, action-free video, and masking strategies
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Version scope: the inspected title page identifies arXiv:2503.00200v3, 24 April 2025. Title and all four authors match the catalog; the catalog submission date is 28 February 2025. Earlier versions were not supplied, so revision changes were not compared.
- Text extraction does not reconstruct figure images; this limitation was addressed by inspecting the retained PDF figures, tables, and method pages.
- Separate supplemental material availability has not been fully verified. The supplementary Section X embedded in this PDF was read in full.
- Code, linked websites, datasets, and external supplements were not inspected; no experiments were reproduced. Reference-list text was read, but cited works were not independently reviewed.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title block and arXiv margin
The title is Unified Video Action Model; authors are Shuang Li, Yihuai Gao, Dorsa Sadigh, and Shuran Song, Stanford University. The margin identifies arXiv:2503.00200v3 [cs.RO], 24 Apr 2025.
Go to primary source ↓motivationPDF pp. 1–2, Abstract and Section I
The paper proposes joint video–action latent modeling with separately decoded outputs to combine visual supervision with fast action inference.
Go to primary source ↓architecturePDF p. 3, Figure 2, Section III problem statement and Section III-A; p. 4, Section III-B
Image observations are encoded with a pretrained kl-f16 VAE and projected into N tokens. Repeated action chunks and masked future observation tokens are concatenated channel-wise; temporal concatenation and a Transformer produce shared video–action latents.
Go to primary source ↓masked-latentsPDF p. 4, Section III-B
Future visual tokens are randomly masked at matching spatial positions across frames. CLIP language tokens are repeated and appended when language is used; the text also describes cross-attention. Inference starts without known future tokens.
Go to primary source ↓decoders-lossPDF p. 4, Section III-C and three unnumbered loss expressions
Video diffusion conditions each patch on its latent token. Action diffusion uses all tokens aggregated by a convolution and MLP. Both predict added noise; their losses are added per time step and summed across the horizon. Either decoder can be bypassed.
Go to primary source ↓task-masksPDF p. 1, Figure 1; pp. 4–5, Section III-D
Five application-specific input/output patterns support policy, video generation, forward dynamics, inverse dynamics, and policy-plus-planner operation. Unused components receive learned mask tokens and only applicable losses are used.
Go to primary source ↓sim-resultsPDF p. 5, Table I, all method rows and four task columns
UVA scores 0.98, 0.88, 0.88, and 0.90 on PushT, Toolhang, PushT-M, and Libero10. UVA-action scores 0.45, 0.62, 0.46, and 0.86. DP-C scores 0.91, 0.95, 0.68, and 0.53; π0 scores 0.85 on Libero10.
Go to primary source ↓sim-protocolPDF p. 5, Section V-A and Table I caption; p. 14, Section X-B; p. 15, Table VIII caption
PushT and Toolhang report best-checkpoint success over 50 rollouts. PushT-M reports best-checkpoint average reward over 50 rollouts. Main-text Libero10 uses 50 environments per task across 10 tasks; X-B instead says 10 runs per task while still claiming 500 episodes. Table VIII explicitly defines its 500-test setting as 50 per task. Simulation table captions call the task scores success rates, whereas V-A calls multitask scores average rewards.
Go to primary source ↓baseline-settingsPDF p. 6, Section V-C; p. 7, Section V-D, simulation comparison and inference speed
UVA-action removes video generation and trains only as a policy. UniPi uses an alternative pixel-video implementation. π0 variants use released checkpoints fine-tuned on all Libero, with third-person images, wrist images, and proprioception; UVA trains on Libero10 with third-person images. UVA, DP-C, and DP-T use 100 action denoising steps in simulation.
Go to primary source ↓real-resultsPDF p. 6, Table II and caption; p. 7, Section V-D
Single-task cup success is UVA 0.85 versus DP-UMI 0.95. Multitask Cup/Towel/Mouse success is 0.65/0.70/0.80 versus 0.50/0.70/0.40. Sixteen-action inference with 16 denoising steps takes 95 ms versus 70 ms.
Go to primary source ↓real-protocolPDF pp. 5–6, Section V-B and Figure 4; pp. 14–15, Section X-C
Public UMI demonstrations train policies deployed on an ARX X5 arm. Multitask training samples 500 episodes per task, totaling 1,500; evaluation has 20 rollouts per task, 60 total, with changed scenes, objects, and grippers. Single-task cup uses the same dataset for both methods and 20 rollouts. Cup success requires saucer placement and handle alignment within ±15 degrees.
Go to primary source ↓robustnessPDF p. 7, Figure 5 and Table III; p. 8, Figure 6 and Section V-D
Goal-color-shift success is UVA 0.64, UniPi 0.40, and OpenVLA 0.32. With background objects UVA scores 0.31 versus UniPi 0.36. Figure 6 shows UVA staying relatively stable as history length increases while DP-C declines; it reports no error bars.
Go to primary source ↓video-resultsPDF p. 8, Section VI and Table IV; p. 9, Figure 7
FVD is evaluated using 500 generated videos per method. On Libero10/Cup Arrangement, UVA with one autoregressive step scores 89.36/51.34, with eight steps 51.10/29.72, and UniPi 56.55/71.37. Figure 7 shows validation examples.
Go to primary source ↓planningPDF pp. 8–10, Section VII; p. 10, Figure 8 and Table V
Block-pushing tests four specified color assignments, ten trials each. DP-C proposes 100 trajectories of 16 actions; UVA predicts outcomes for image-based distance scoring. Six actions of the selected trajectory execute before replanning. Average success is DP-C 0.38, UVA-guided 0.60, and simulator-guided 0.75. Candidate quality and detection errors limit even simulator-guided selection.
Go to primary source ↓inversePDF p. 10, Section VIII and Table VI
On unseen UMI Cup Arrangement data, predicted camera-pose actions are compared to motion capture. Position/rotation errors are UVA 0.75 cm/1.11 degrees, UniPi inverse dynamics 1.92 cm/2.21 degrees, and visual-inertial SLAM 0.41 cm/0.30 degrees. UVA predicts 16 actions together; SLAM needs additional mapping.
Go to primary source ↓limitationsPDF p. 7, Section V-D, real-world discussion; p. 10, Section IX
The authors report failures when background and object colors match. They leave web-scale actionless-video pretraining and additional modalities such as sound and force to future work.
Go to primary source ↓autoregressionPDF p. 14, Section X-A and Figure 9; p. 4, Section III-B
UVA builds on pretrained MAR-B with substantial modifications. Continuous visual tokens are progressively filled across all future frames, starting with no known tokens; one autoregressive step fills the video in one pass.
Go to primary source ↓runtimePDF p. 15, Section X-D and Table VII; p. 5, Table I speed column; p. 6, Table II caption
Simulation timings use NVIDIA L40 GPUs; real-world timings use an RTX 3080. VAE encoding, Transformer attention, and 16-step action diffusion take 40, 40, and 15 ms. UVA totals 95 ms; with video decoding the 16-step Policy+Planner row is 195 ms. At 100 denoising steps UVA is 173 ms and Policy+Planner 798 ms. Flash-Attention rows report 85/163 ms totals. Table VII's generic caption mentions 100 diffusion steps although rows explicitly include both 16 and 100.
Go to primary source ↓human-videoPDF p. 15, Section X-E and Table VIII
An additional experiment pretrains video generation on 3,175 human-only videos, then fine-tunes on combined human videos and LIBERO-10 using masked training. Baseline/human-data scores are 0.93/0.97 over 30 tests and 0.90/0.91 over 500 tests.
Go to primary source ↓maskingPDF p. 15, Section X-F; p. 16, Table IX and continuation of Section X-F
PushT-M compares application-dependent and application-independent input masking at 25%, 50%, and 75%. Independent 50% masking gives policy success 0.87 and inverse L2 error 13.55. Dependent 25% masking gives video/forward FVD 127.09/129.77; independent 50% gives 200.08/203.59. No inverse-error units or uncertainty are specified here.
Go to primary source ↓8.5 Primary sources
Unified Video Action Model ↗
PDF · 11,400 extracted words
Source fingerprint
8f0e312622e926623a90e8f8dbe7d17fca3c007b3b8a9b936cd695c204860118