RepWAM: World Action Modeling with Representation Visual-Action Tokenizers
1. Paper overview
In one sentence: RepWAM makes visual states and latent actions share a semantic representation, improving the reported control ablations while leaving the robot adaptation interface incompletely specified. e02e03e04e05e08e09e11e13
| At a glance | What to know |
|---|---|
| Research problem | Author claim The authors argue that reconstruction-oriented video latents devote capacity to appearance while leaving manipulation semantics weakly represented. Separately, visual states and motor commands occupy disjoint spaces. RepWAM addresses both bottlenecks by learning visual semantics and action-like transitions before adapting to embodiment-specific control. e02 |
| Core mechanism | |
| A key reported result | RoboTwin 2.0: 50-task average: RepWAM-5B: 89.3 Easy / 88.4 Hard; 1.3B: 86.6 / 83.1. Success rate (%). Standard dual-arm simulation suite, official Easy/Hard randomization; Table 1. π0.5: 82.7 / 76.8; Motus: 88.7 / 87.0; Lingbot-VA: 92.9 / 91.6. 5B exceeds π0.5 and Motus but trails Lingbot-VA. Backbone pretraining differs; this is not an isolated tokenizer comparison. e09 |
| Reading caution |
Core contributions
Figure 1. Semantic visual tokens become the state space in which latent actions describe change. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow the upper path from video frames through the visual encoder, latent tokens and reconstruction decoder. The separate foundation-model path supplies feature alignment; Section 3.2 states that this teacher is frozen. The temporal-causal thumbnail corresponds to attention across frames, with full spatial attention inside a frame. Next follow the lower path: consecutive latent states enter inverse dynamics, whose compact output feeds forward dynamics. At the right, multiplication transports the current state using K_t, and addition introduces delta_t. Equation (3) makes the conditioning explicit: forward dynamics receives both the current state and the latent action. e03e04e05
What it supports. The architectural idea is to represent a transition using transport of existing semantic content plus a residual for changes that transport cannot explain. The inverse/forward pair is trained after freezing the visual tokenizer. This figure explains RepViTok’s representation learning; the language-conditioned causal WAM is specified separately in Section 3.3.
Where the evidence stops. The graphic labels its action token a_t; Equation (3) calls it ell_t and explicitly supplies z_t to forward dynamics, an input not drawn at that block. Preserve this notation/diagram gap: the token shown here is not yet an executable motor command.
2. Motivation
2.1 The problem and the proposed response
The authors argue that reconstruction-oriented video latents devote capacity to appearance while leaving manipulation semantics weakly represented. Separately, visual states and motor commands occupy disjoint spaces. RepWAM addresses both bottlenecks by learning visual semantics and action-like transitions before adapting to embodiment-specific control. e02
2.2 What this reading follows
A useful world model must preserve the scene information that an instruction and a robot action depend on. RepWAM tackles this at the tokenizer: it aligns visual latents with a frozen teacher, then learns compact actions as transformations between those states. A causal world/action model predicts both streams before adaptation to motor commands. Read the architecture first, then separate three evidence levels: physical task success, the main simulation comparison, and controlled ablations. The strongest lesson is that representation and training schedule both matter in these experiments. Neither attractive reconstructions nor improved open-loop action scores alone establish reliable execution. e02e03e04e05e08e09e11e13
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | Dual-system |
| Prediction paradigm | Joint prediction |
| Quadrant | Q3 · Dual-system × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
Dual-system × Joint prediction is supported by paired world/action experts with shared attention but separate FFNs and a joint flow objective. Tokenizer inverse dynamics does not make the WAM an inverse-dynamics-only predictor. Latent representation and action alignment fit; “JEPA” is broader than the actual reconstruction-plus-teacher-alignment formulation. e03e04e05e07
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Learn what a state preserves before learning how it changes
RepViTok’s two learning steps solve different problems. The video autoencoder reconstructs observations while matching temporally pooled features from a frozen visual teacher. That alignment supplies a semantic target beyond pixel appearance; reconstruction remains part of the objective. Afterward, the visual tokenizer is frozen and inverse dynamics compresses consecutive latent frames into a small transition variable. Forward dynamics must reconstruct the next state from the current state and that variable through transport plus residual change. Reader interpretation: this arrangement encourages the transition variable to explain change while reusing content already available in the current state. The paper’s small action bottleneck supports that design intent, but neither the equations nor the visualizations establish complete removal of irrelevant content. e03e04e07e14
5.2 Distinguish latent-action pretraining from executable control
The causal WAM operates on the tokenizer’s paired visual and action representations. A language embedding and initial visual context precede chunks, and a block-causal mask permits preceding context while excluding future chunks. Within a chunk, the visual and action experts share attention weights and use modality-specific feed-forward networks. Teacher forcing supplies preceding context during flow-matching training, which regresses both visual and action velocity targets. Robot adaptation then introduces demonstrations with continuous motor commands. The source describes an adapted action expert mapping transitions to executable commands and reports closed-loop robot tests. It does not fully specify that mapping or the timing of observation updates and motor execution. Reproducing the causal predictor therefore would not, by itself, reproduce the deployed robot interface. e05e06e07e08
5.3 Use the ablation sequence to locate the supported conclusion
Table 2 first changes the visual tokenizer. It shows improved generation and action metrics for RepViTok, but also a warning: ViTok beats WAN2.2 on open-loop action scores while doing worse on physical PickFruit. Table 4 then starts from the RepViTok visual baseline and changes latent-action training. Its two-stage recipe improves every reported metric and the physical task. Reader interpretation: taken together, these comparisons support choosing both semantic visual representations and staged action adaptation in this setting. They do not identify a single sufficient cause of robust control. The cross-model simulation comparison has different pretraining histories, and the physical evaluation is small. A stronger causal test would hold tokenizer architecture, training exposure and adaptation budget fixed while separately varying alignment and scheduling. e08e09e10e11e13
5.4 Training and inference
During training
RepViTok trains on Panda-70M. WAM pretraining uses roughly 100G AgiBot video/action latent tokens; adaptation uses roughly 300G tokens from AgiBot, RoboMIND, RoboCOIN and InternA1 with motor commands. WAM backbones train from scratch; visual and text teachers are pretrained. e06e07
Tokenizer encoder/decoder each have 12 layers and width 768. Visual/action latent dimensions are 96/4; dynamics MLPs have four layers of width 256. World experts have 30 layers and widths 1536/3072 at the named 1.3B/5B scales; the width-768 action expert adds approximately 350M parameters. e07
Muon uses peak learning rate 0.01, weight decay 0.01, cosine decay with warmup, bfloat16 and gradient clipping 2.0. The 1.3B/5B runs use 64/128 H20 GPUs and sequence limits 200K/160K tokens. e07
During inference
Condition on language and visual context, generate visual/action chunks, and use the adapted action expert for motor commands in closed-loop manipulation. Chunk size is sampled from [1,4] during training and fixed to 2 at inference. A complete action-decoding and feedback-timing algorithm is absent. e05e07e08
Video classifier-free guidance affects only video denoising. Scale 1.0 uses the conditional video branch without extra extrapolation; potential latency/memory savings are discussed but not measured. e15
5.5 Implementation flow
- Encode semantic visual states
The first frame uses 16×16 patches; later frames use 4×16×16 tubelets. Temporal-causal attention and full within-frame spatial attention produce visual latents. Reconstruction combines pixel L1, perceptual and adversarial losses; squared feature alignment pulls projected latents toward a frozen visual teacher. e03
- Learn transitions in the frozen visual space
Freeze the visual tokenizer. Inverse dynamics compresses consecutive states into a latent action; forward dynamics uses the current state and action to predict spatial transport plus residual change. Forward prediction and reversed-pair consistency supervise it. The bottleneck is intended to limit content leakage, rather than proving that nuisance information disappears. e04e07
- Jointly forecast paired chunks
Language tokens and the initial visual state prefix visual/action chunks. Block-causal attention excludes future chunks. Visual and action experts share attention weights but have separate feed-forward networks. Teacher-forced conditional flow matching regresses both streams from Gaussian-noise interpolants. e05e07
6. Experiments & results
RepWAM learns semantic visual tokens and compact transition tokens, jointly models their future evolution under language, then adapts to robot commands. Its central evidence is the tokenizer and training-stage ablations: better latent prediction accompanies better physical execution. The main simulation comparison remains below Lingbot-VA, and robot results use few trials.
6.1 Read the original evidence
Figure 2. Physical execution improves most visibly on the drawer and tube tasks at the larger model scale. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each task group using the four-color legend, keeping the two RepWAM sizes separate. The abbreviated labels refer to complete tasks: fruit placement into a plate, drawer manipulation followed by block placement, and test-tube insertion into a rack. Section 4.2 reports fifty demonstrations per task for fine-tuning and ten physical evaluation rollouts per task on a Franka dual-arm platform. The percentage axis therefore changes in ten-point increments for individual outcomes. Figure 3 on the following page shows selected successful executions, but the bars provide the actual reported comparison. e08e10
What it supports. RepWAM-5B records 60%, 80% and 60% across the three tasks, versus 50%, 70% and 40% for Lingbot-VA. RepWAM-1.3B ties the larger model on fruit but reaches 50% on drawer and 30% on tube. This is physical manipulation evidence, distinct from image or video prediction quality.
Where the evidence stops. With ten rollouts per task and no uncertainty bars, a ten-point difference is one successful trial. These results do not isolate the authors’ suggested capacity or perception bottlenecks, and should not be pooled with the shorter ablation runs.
Table 1. RepWAM-5B exceeds two baselines on the overall simulation average but remains below Lingbot-VA. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with the bottom row, which averages all fifty RoboTwin 2.0 tasks. Each model has paired Easy and Hard columns; compare like columns before interpreting overall rankings. The rows above summarize the paper’s horizon categories and should not be substituted for the full-suite average. The backbone-pretrained row is essential context: all three baselines have check marks, while the RepWAM models have crosses. Section 4.3 says evaluation follows official scene, lighting, camera and physics randomization. Table 3 on page 9 provides a more focused comparison of tokenizer replacements within a fixed 1.3B WAM. e09e12
What it supports. The 5B model reaches 89.3 Easy and 88.4 Hard, compared with Motus at 88.7 and 87.0 and π0.5 at 82.7 and 76.8. Lingbot-VA remains ahead at 92.9 and 91.6. The table supports competitive simulation performance, not an across-the-board lead over prior WAMs.
Where the evidence stops. Different backbone pretraining prevents attributing these gaps solely to tokenization. The authors’ explanation for Lingbot-VA’s advantage is a hypothesis. Table 3 changes the tokenizer, but does not experimentally isolate the value of Lingbot-VA’s backbone pretraining.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| RoboTwin 2.0: 50-task average Standard dual-arm simulation suite, official Easy/Hard randomization; Table 1. | RepWAM-5B: 89.3 Easy / 88.4 Hard; 1.3B: 86.6 / 83.1. Success rate (%) | π0.5: 82.7 / 76.8; Motus: 88.7 / 87.0; Lingbot-VA: 92.9 / 91.6. 5B exceeds π0.5 and Motus but trails Lingbot-VA. Backbone pretraining differs; this is not an isolated tokenizer comparison. e09 |
| Franka dual-arm manipulation: fruit, drawer and tube 50 demonstrations/task; 500 fine-tuning steps, learning rate 1e-5, sequence length 150K; 10 physical rollouts/task. | RepWAM-5B: 60 / 80 / 60; 1.3B: 60 / 50 / 30. Success rate (%) in fruit / drawer / tube order | π0.5: 10 / 50 / 10; Lingbot-VA: 50 / 70 / 40. Physical evidence favors 5B on drawer and tube, but ten trials cannot establish the authors’ perception-versus-capacity explanation. e08 |
| Tokenizer ablation: AgiBot prediction and PickFruit 1.3B, 40,000 steps on 32 H20 GPUs; seen versus held-out tasks; OLS threshold 0.03. | RepViTok: seen/unseen gFVD 61.01/72.91, OLS 18.82/14.15; PickFruit 30%. gFVD; open-loop score (OLS); physical success (%) | WAN2.2 VAE: 67.42/83.98, 13.68/11.21, 20%; ViTok: 69.23/80.14, 16.29/13.81, 10%. ViTok improves OLS over WAN but lowers physical success; open-loop gains alone do not guarantee execution gains. e10e11 |
| Tokenizer replacement on RoboTwin 2.0 Fixed 1.3B WAM; Easy/Hard, 50-task average; Table 3. | RepViTok: 86.6 / 83.1. Success rate (%) | WAN2.2 VAE: 78.0 / 76.0. Reader-calculated gains are 8.6/7.1 percentage points. This supports tokenizer choice, not a causal explanation for Lingbot-VA’s advantage. e12 |
| Latent-action training ablation AgiBot seen/unseen and physical PickFruit within the 1.3B ablation regime. | Two Stages: gFVD 48.23/58.83, OLS 19.87/16.98, PickFruit 50%. gFVD; OLS; success (%) | Without latent-action pretraining: 61.01/72.91, 18.82/14.15, 30%; Joint Pred: 94.25/98.77, 18.52/15.22, 20%. Two Stages leads all columns. Joint Pred is an appended-head alternative; its failure does not refute joint prediction generally. e10e13 |
| Video CFG sensitivity on RoboTwin 2.0 Figure 5: video CFG scales 1.0, 1.25 and 2.0. | RepWAM: 88.9, 88.0, 88.1. Average success rate (%) | Lingbot-VA: 90.8, 91.1, 91.2 within this figure. RepWAM’s best displayed average uses 1.0. Lingbot-VA values differ from Table 1; the source does not reconcile settings. e15e09 |
| Visual reconstruction on ImageNet and UCF101 Table 5: ImageNet 256; UCF101 256 resolution, 17 frames. | RepViTok: 0.80 / 28.90 / 0.89; 1.09. ImageNet rFID / PSNR / SSIM; UCF101 rFVD | WAN2.2 VAE: 0.50 / 28.16 / 0.87; 4.28. Better PSNR, SSIM and video rFVD coexist with worse image rFID. Reconstruction metrics do not measure robot success. e16 |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 2. Semantic tokenization improves the reported prediction and action scores, while open-loop and physical rankings can diverge. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read horizontally in three groups. Eval Seen covers tasks observed during training; Eval Unseen covers held-out tasks. In each group, gFVD evaluates generated video, PSNR and SSIM evaluate visual similarity, and OLS evaluates open-loop action accuracy at threshold 0.03. The final PickFruit column instead measures closed-loop physical success. Lower gFVD and higher PSNR, SSIM and OLS are favorable in the paper’s comparisons. These ablations use a 1.3B WAM trained for forty thousand steps. Compare ViTok against WAN2.2 as well as RepViTok: their different offline and physical rankings are an informative part of the table. e10e11e13
What it supports. RepViTok improves gFVD from WAN2.2’s 67.42/83.98 to 61.01/72.91 on seen/unseen tasks and raises OLS to 18.82/14.15. PickFruit rises from 20% to 30%. However, ViTok’s higher OLS than WAN2.2 accompanies only 10% physical success, illustrating that better offline action scores need not imply better execution.
Where the evidence stops. The table compares tokenizer designs without a complete factorial isolation of architecture, semantic alignment and training exposure. Its RepViTok row is the visual-tokenizer baseline reused as w/o in Table 4, not the final two-stage configuration.
Table 4. Pretraining on latent transitions before adapting to robot commands gives the strongest reported ablation results. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Resolve the row definitions on page 8 before reading the numbers. The w/o baseline already uses RepViTok visual tokens, but learns robot actions without latent-action pretraining. Joint Pred appends a head that predicts action latents alongside video and pads the final action slot with zero. Two Stages first pretrains with visual and latent-action tokens, then adapts to robot control. Compare both prediction columns and PickFruit: the proposed schedule improves visual quality, open-loop action accuracy and the physical task. The label Joint Pred names a particular ablation; the final two-stage model also jointly predicts visual and action streams during pretraining. e05e08e10e13
What it supports. Two Stages reaches seen/unseen gFVD 48.23/58.83 and OLS 19.87/16.98, leading the table. Physical PickFruit success is 50%, versus 30% without latent-action pretraining and 20% with the appended-head alternative. The pattern supports the staged recipe under the reported setup, extending beyond a purely visual prediction gain.
Where the evidence stops. The table changes training organization and prediction design together; it does not fully isolate their individual effects or document equal total exposure across variants. Its 50% physical result belongs to the ablation regime, separate from the main model’s Figure 2 results.
Figure 5. Extra video guidance does not improve RepWAM’s average across the tested settings. Original paper, p. 10 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the shared legend to distinguish RepWAM’s blue circles from Lingbot-VA’s red squares. Move left to right within each panel through video guidance scales 1.0, 1.25 and 2.0. The implementation changes only video denoising; the action prediction procedure is held unchanged. Inspect Easy and Hard before the Average panel: RepWAM’s Easy result peaks at 1.25, while its Hard result falls at that setting. Scale 1.0 means no additional guidance extrapolation. The plot thus tests an inference setting for the video branch, rather than removal of future prediction from the policy. e15e09
What it supports. RepWAM’s displayed averages are 88.9, 88.0 and 88.1 across the three scales; the best is 1.0. On Easy, 1.25 improves success to 90.6, but Hard drops to 85.5. This supports using the conditional video branch alone for the best average among these tested settings, not a universal guidance optimum.
Where the evidence stops. Lingbot-VA’s values here differ from Table 1, with no reconciliation of settings in the source; keep the comparisons separate. No error bars or latency/memory measurements are supplied. Potential savings from skipping the unconditional video branch remain an implementation-dependent claim.
7. Analysis & limitations
7.1 What the evidence leaves open
The principal comparisons provide no confidence intervals or repeated-seed uncertainty. Different pretraining histories complicate attribution; the authors hypothesize that WAN pretraining explains Lingbot-VA’s advantage. e08e09e10e13
Figure 4 shows lower robot-action decoding loss with frozen RepViTok latents than LAPA. This diagnostic and selected successful rollouts do not establish broad cross-embodiment transfer. e14e08
Extending WAM pretraining to internet and egocentric human video remains future work, despite internet-video training of the tokenizer. e17e06
7.2 Questions for discussion
- Does semantic alignment improve physical success when tokenizer architecture and pretraining budget are fully matched?
- How much of the two-stage gain survives identical data exposure and adaptation updates?
8. Reproducibility audit
8.1 Requirements and known gaps
Replication needs the teacher configuration, data mixing and exact AgiBot task split. Dimensions and major optimization settings are given, but split sizes, teacher feature selection, tokenizer schedule, motor-action parameterization, adaptation mapping, denoising step count and control frequency are unspecified. e03e04e05e06e07e10
Proposed checks should repeat tokenizer and latent-action ablations with matched data, update budgets and multiple seeds, tracking OLS and physical success. Test CFG separately with measured end-to-end latency. e10e11e13e15
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Isolate semantic alignment while measuring real execution
Reader-proposed check, not performed: train the same ViTok architecture on the same Panda-70M sample with identical initialization policy, reconstruction losses and update budget. Compare correct frozen-teacher alignment, no alignment, and an explicitly experimental shuffled-frame teacher-target control. Freeze each tokenizer and train the same 1.3B WAM under matched AgiBot data and adaptation budgets. Predefine the seen/held-out task split, repeat training seeds, and report gFVD, OLS at 0.03 and physical PickFruit success with trial counts and uncertainty. If correct alignment improves both held-out prediction and execution over both controls, it supports the semantic-target explanation. Improvement only in reconstruction or OLS would weaken the claim that alignment itself causes reliable control. e03e06e07e10e11
Check 2: Separate the two-stage schedule from extra data exposure
Reader-proposed check, not performed: freeze one RepViTok tokenizer and compare direct robot-action training, the appended-head Joint Pred configuration, and the proposed latent-action pretraining followed by robot adaptation. Match total token exposure and optimization compute, and include a direct-action control with the same final adaptation updates and replay exposure as Two Stages. Declare the motor-command parameterization and decoding interface, which the source leaves incomplete. Evaluate the same held-out AgiBot tasks and physical PickFruit starts across seeds, retaining OLS and success as separate endpoints. A persistent two-stage advantage would support the intermediate transition representation; a gap that disappears after budget matching would indicate that schedule or exposure explains much of Table 4. e04e05e06e07e10e13
8.3 Reading coverage
Visual audit: The title/author/version page and all scientific body pages were rendered and visually inspected. This includes Figure 1 with Eqs. (1)–(4), the causal and flow equations on p. 5, implementation and hardware on p. 6, robot tasks and executions on pp. 6–7, all five quantitative tables, the transfer diagnostic on p. 9, CFG on p. 10, and reconstructions/conclusion on p. 11. All six final original crops were separately viewed and retain the relevant labels, legends and table headers. Figure 1’s action notation and omitted state-conditioning arrow are disclosed; multiplication/addition agree with Eq. (3). Figure 5 and Table 1 baseline values are kept separate. Reference pages 12–13 were read as text and are outside the image pass. No separate supplement was supplied.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11. Appendix coverage: not present.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work
- 2.1 World Action Models
- 2.2 Latent Action Models
- 3 Method
- 3.1 Overview
- 3.2 Representation Visual-Action Tokenizer
- 3.3 Causal World Action Models
- 4 Experiments
- 4.1 Implementation Details
- 4.2 Real-World Experiments
- 4.3 Simulation Experiments
- 4.4 Ablation Studies
- 5 Conclusion
- References
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- Identity/version: the inspected title page states arXiv:2606.13674v2 [cs.CV], 13 Jun 2026. The title and all eight authors match the catalog after normalizing name order. The catalog specifies no revision; no earlier edition was supplied, so revision differences cannot be established.
- All four supplied text chunks were read, covering PDF pages 1–13. Pages 1–11 and Figures 1–6 / Tables 1–5 were visually inspected; reference pages 12–13 were read as text. The supplied PDF contains no appendix.
- The extraction limitation above was addressed by inspecting the original PDF pages and six final crops. Separate supplements remain unverified; none were supplied.
- Code and project links were not opened, code was not inspected, and experiments were not reproduced.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e01PDF p. 1, title, author/affiliation block and arXiv margin
The exact catalog title appears with Junke Wang, Qihang Zhang, Shuai Yang, Yiming Luo, Yujun Shen, Zuxuan Wu, Yu-Gang Jiang and Yinghao Xu; the margin identifies arXiv:2606.13674v2, 13 Jun 2026.
Go to primary source ↓e02PDF pp. 1–3, Abstract, Section 1 and Section 3.1
The motivation is weak semantic grounding in reconstruction-oriented visual latents and a gap between visual states and motor actions; RepWAM learns aligned tokens before robot adaptation.
Go to primary source ↓e03PDF p. 3, Figure 1 and caption; p. 4, Section 3.2, visual tokenization and Eqs. (1)–(2)
The video encoder uses temporal-causal/spatial attention and patch/tubelet tokenization. Reconstruction is combined with frozen-teacher alignment. Figure 1 sketches the tokenizer, rather than the entire language-conditioned WAM.
Go to primary source ↓e04PDF p. 4, Section 3.2, Eq. (3); p. 5, Eq. (4); p. 3, Figure 1
Inverse dynamics processes consecutive frozen visual states; forward dynamics takes current state and latent action to produce transport K and residual delta. Forward and reversed-pair losses train it. Figure 1 labels the latent action a_t, whereas Eq. (3) uses ell_t; the schematic omits an explicit current-state input to the forward-dynamics block.
Go to primary source ↓e05PDF p. 5, Section 3.3, Eqs. (5)–(7)
Paired chunks follow language/initial-state context under block-causal masking. Shared attention and separate FFNs serve both modalities. Teacher forcing and conditional flow matching supervise visual and latent-action velocities.
Go to primary source ↓e06PDF p. 5, Section 4.1, Training data
Panda-70M trains RepViTok; approximately 100G AgiBot tokens pretrain the WAM; approximately 300G tokens from the named adaptation corpus include aligned continuous motor commands.
Go to primary source ↓e07PDF p. 6, Section 4.1, Model and optimization
Specifies tokenizer/dynamics dimensions, 30-layer world/action experts, frozen PLM text encoder, Muon settings, action loss weight 1, uniform SNR sampling, chunk sizes, sequence limits and H20 GPU counts. The action expert adds about 350M parameters.
Go to primary source ↓e08PDF p. 6, Figure 2 and Section 4.2; p. 7, Section 4.2 continuation and Figure 3
Three Franka dual-arm tasks use 50 demos and 10 physical rollouts each. Figure 2 gives fruit/drawer/tube success: 5B 60/80/60%, 1.3B 60/50/30%, π0.5 10/50/10%, Lingbot-VA 50/70/40%. Figure 3 contains selected successful executions.
Go to primary source ↓e09PDF p. 7, Section 4.3; p. 8, Table 1, backbone-pretrained row and Average_50 Tasks row
Under official RoboTwin randomization, Easy/Hard averages are π0.5 82.7/76.8, Motus 88.7/87.0, Lingbot-VA 92.9/91.6, RepWAM-1.3B 86.6/83.1 and 5B 89.3/88.4. Baseline backbones are pretrained; RepWAM backbones are not.
Go to primary source ↓e10PDF p. 8, Section 4.4 opening paragraph
Ablations use 1.3B, 40,000 steps and 32 H20 GPUs; AgiBot has seen-task and held-out-task evaluation sets. Metrics include gFVD, PSNR, SSIM, OLS at threshold 0.03 and physical PickFruit success.
Go to primary source ↓e11PDF p. 8, Table 2, all tokenizer rows and seen/unseen/PickFruit columns
RepViTok gives gFVD 61.01/72.91, OLS 18.82/14.15 and 30% PickFruit. WAN2.2 VAE gives 67.42/83.98, 13.68/11.21 and 20%; ViTok gives 69.23/80.14, 16.29/13.81 and 10%.
Go to primary source ↓e12PDF p. 8, paragraph introducing Table 3; p. 9, Table 3, Average_50 Tasks row
With a fixed 1.3B WAM, WAN2.2 VAE versus RepViTok gives 78.0 versus 86.6 Easy and 76.0 versus 83.1 Hard success.
Go to primary source ↓e13PDF p. 8, World action modeling with latent actions; p. 9, Table 4 and following paragraph
The w/o row omits latent-action pretraining. Joint Pred adds a latent-action head with a zero-padded final slot. Two Stages pretrains on paired latents then adapts to control; gFVD 48.23/58.83, OLS 19.87/16.98 and PickFruit 50% lead the table.
Go to primary source ↓e14PDF p. 9, Figure 4 and Latent actions and transferring to robot control
Selected visualizations compare LAPA and RepViTok. The right plot trains the same IDM to decode robot actions from frozen latents and shows lower loss for RepViTok; exact endpoints are not tabulated.
Go to primary source ↓e15PDF p. 10, Figure 5 and The effects of video classifier-free guidance
Only the video path changes in the CFG sweep. At 1.0/1.25/2.0, RepWAM averages are 88.9/88.0/88.1; Lingbot-VA averages are 90.8/91.1/91.2. RepWAM Easy peaks at 1.25, but Hard and the average peak at 1.0. Latency/memory benefits are potential, not measured.
Go to primary source ↓e16PDF p. 10, Table 5, ImageNet 256 and UCF101 256×17 columns; p. 11, Figure 6
RepViTok versus WAN2.2 gives ImageNet rFID 0.80/0.50, PSNR 28.90/28.16, SSIM 0.89/0.87 and UCF101 rFVD 1.09/4.28. Figure 6 shows selected image/video reconstructions.
Go to primary source ↓e17PDF p. 11, Section 5, final paragraph
Scaling WAM pretraining beyond robotics video to large-scale internet and especially egocentric human video is proposed future work.
Go to primary source ↓8.5 Primary sources
RepWAM: World Action Modeling with Representation Visual-Action Tokenizers ↗
PDF · 5,396 extracted words
Source fingerprint
83e43b36df758323556ff5e7ff300f914bfc29ecbf8cdfef3929ef6e219773a1