Point Tracking Improves World Action Models
1. Paper overview
In one sentence: JOPAT couples future pixels, point tracks and robot actions in one denoiser, improving motion-sensitive manipulation while retaining the limits of sparse 2D supervision. identitystate-interfacearchitectureobjectivesmodalityvisibilityreal-resultsrobot-protocollimitations
| At a glance | What to know |
|---|---|
| Research problem | Author claim Pixel-based prediction mixes controllable motion with lighting, texture and background. The authors ask which future-state representation makes video learning useful for robot control. Their answer combines visual semantics with persistent point correspondences and visibility, so disappearance under occlusion need not be confused with lack of motion. This is a manipulation policy with executable actions, not merely a video predictor. state-interfaceobjectives |
| Core mechanism | Source description A single denoising transformer couples visual, track and action tokens during both training and sampling; visibility is supervised through a separate output head. architecturetracksobjectives |
| A key reported result | LIBERO four-suite manipulation: Reported average 97.8; Spatial 97.2, Object 98.9, Goal 98.4, Long 96.4. Success rate (%). Forty tasks; fifty demonstrations/task; LIBERO-90 pretraining for 100K steps, target-suite finetuning for 10K. CogVLA: average 97.4, Long 95.4; UWM: average 84.7, Long 77.6. Highest printed average; JOPAT does not win Spatial. Baseline training resources are not controlled here, and no uncertainty establishes the small lead. simulationtable1-arithmetic |
| Reading caution | Author claim The authors identify sparse-track detail loss, tracker supervision errors, static-camera dependence and a bottleneck above 20 Hz. Appendix failures involve reflective lids, insertion geometry, pushing localization and cluttered grasp locations. These diagnoses are qualitative; 2D temporal consistency does not supply missing 3D contact geometry. limitationsfailures |
Core contributions
- Source description
A single denoising transformer couples visual, track and action tokens during both training and sampling; visibility is supervised through a separate output head. architecturetracksobjectives
- Reader analysis
Controlled modality ablations support complementary predictive targets. Real-robot Joint/Latent-only/Track-only averages are 57.5/35.0/15.0 percent; these results support the design but do not isolate every architectural factor. modality
Figure 2. Point correspondences become a predicted modality inside the shared action-generating model. Original paper, p. 4 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start with panel (a): the latest observed frame anchors a regular query grid, and the sliding window supplies future coordinates and visibility. Panel (c) reshapes coordinates into a grid over time and uses (2,5,5) patchification; it is not a set of selected semantic object keypoints. In panel (b), the image, action and track encoders identify the three token families. Section 4.1 supplies the full attention and conditioning details that the schematic omits. Read its equation as noisy tokens entering the DiT and predicted modality noise leaving it. At deployment, action and future-state tokens start from noise, not known future actions or images. architecturetracksobjectiveshyperparameters
What it supports. The architecture provides an inference-time route for predicted motion to influence action denoising. Track coordinates share attention with visual and action tokens; visibility is a separately supervised output, never ground-truth input to the track encoder. This distinction supports the paper's joint-prediction classification and prevents interpreting visibility labels as privileged test-time observations.
Where the evidence stops. Figure 2(b) draws epsilon-labeled tokens upstream of the DiT, while Section 4.1 defines predicted noise as output. The crop preserves that discrepancy; the equations govern this explanation. Panel (c)'s repeated patchifier labels do not fully specify its internal computation.
2. Motivation
2.1 The problem and the proposed response
Pixel-based prediction mixes controllable motion with lighting, texture and background. The authors ask which future-state representation makes video learning useful for robot control. Their answer combines visual semantics with persistent point correspondences and visibility, so disappearance under occlusion need not be confused with lack of motion. This is a manipulation policy with executable actions, not merely a video predictor. state-interfaceobjectives
2.2 What this reading follows
A robot can recognize an object yet lose track of what happened when its arm blocked the camera. JOPAT addresses that gap by predicting both visual latents and persistent point correspondences, with a separate visibility output. These future variables share a diffusion transformer with executable action tokens, so the predicted world participates in control generation. Read the architecture first, then compare the headline benchmarks with the modality and visibility ablations. The evidence supports useful motion supervision and video transfer, while the small physical evaluation and weak insertion performance limit stronger claims. This edition follows the supplied arXiv v1, including its appendix and reporting discrepancies. identitystate-interfacearchitectureobjectivesmodalityvisibilityreal-resultsrobot-protocollimitations
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | WAMs |
| Architecture | One Model |
| Prediction paradigm | Joint prediction |
| Quadrant | Q1 · One Model × Joint prediction |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded One Model × Joint prediction classification is supported by shared bidirectional denoising across future-world and action tokens, including at inference. Modality-specific encoders and heads do not constitute separate world and policy models. Visibility is an auxiliary supervised output, while predicted track coordinates participate in action sampling. architecturetracksobjectives
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Make motion part of the state that actions are generated with
Start from the paper's state-interface question. A future image latent can preserve what an object looks like without explicitly identifying how the same physical point moves across time. JOPAT adds reference-frame point coordinates to that future representation and supervises whether each correspondence remains visible. The coordinates are arranged by their query-grid locations and encoded over time, while visual latents preserve appearance context. Both streams join action tokens in one bidirectional denoising sequence. During sampling, the transformer repeatedly updates an imagined pixel-track future together with a candidate action chunk. Only decoded actions reach the robot. This is why track prediction is more than a training diagnostic in the proposed implementation, even though visibility itself is an output-only supervised quantity. state-interfacetracksarchitectureobjectives
5.2 Transfer from video by retaining the prediction interface
The video-only objective removes action supervision but retains visual, coordinate and visibility losses. This lets action-free clips train the same future-state representation that later coexists with executable action tokens. Demonstrations then supervise all branches and ground that representation in the robot's controls. The source distinguishes several experiments: standard LIBERO results use LIBERO-90 pretraining, while the modality ablation uses DROID, and Table 6 compares DROID with generic OpenVid-1M video under limited demonstration budgets. Generic-video gains therefore show transfer after action-labeled finetuning, not an ability to execute actions learned directly from unlabeled video. At test time, ten DDIM updates generate the streams together, and the default eight-action prefix is executed before new observations trigger replanning. Feedback comes from that repeated observation cycle. objectivessimulationmodalityvideo-pretrainingimplementationrobot-protocol
5.3 Use ablations to narrow the claim, then inspect what they leave open
Reader analysis: the strongest mechanism evidence is the within-protocol Joint versus single-modality comparison, because it holds data and optimization more closely aligned than the main leaderboard. Its large gap supports combining the two future targets, but removing an entire stream also changes the model's supervision and computation. The visibility experiment narrows the question further, yet a missing head is not a matched-capacity control and task success does not measure visibility calibration. The physical task breakdown supplies another boundary: temporal consistency can help sequencing and pushing while leaving precise insertion mostly unsolved. Finally, the horizon plot shows that an almost stable mean can conceal opposing task changes. Together, these observations motivate tests of inference-time information flow and occlusion-specific supervision, rather than treating every gain as proof of object permanence. modalityvisibilityreal-resultsrobot-protocolhorizonfailures
5.4 Training and inference
During training
Independent diffusion timesteps corrupt each modality, using 100 training diffusion steps and the squaredcos_cap_v2 noise schedule. Video-only batches mask the action branch; demonstrations supervise all branches. Table 8 assigns every loss unit weight. Standard LIBERO evaluation uses LIBERO-90 pretraining, whereas the modality and video-source studies use separately stated DROID/OpenVid protocols. objectivessimulationmodalityvideo-pretraininghyperparameters
During inference
Initialize all three token streams with Gaussian noise and run ten DDIM steps. Visual and track variables continue interacting with action tokens during sampling. Decode actions, execute the first eight of nineteen, observe again and replan. No separate inverse-dynamics stage or candidate-scoring planner is described. objectivesimplementationrobot-protocol
5.5 Implementation flow
- Encode the observed context
An ImageNet-initialized ResNet-18 encodes two 224×224 RGB frames. Its global feature conditions every DiT block through AdaLN. A frozen SDXL VAE encodes future-image targets into 28×28×4 latents. The future window contains two frames at offset H=16. architectureimplementationhyperparameters
- Construct correspondence targets
CoTrackerV3 tracks 625 grid queries anchored on the latest observed frame. Coordinates are reshaped to a 25×25 spatial grid over time and patchified with a (2,5,5) 3D convolution. Visibility labels supervise an output head and never enter the track encoder. tracksimplementationhyperparameters
- Denoise one shared sequence
Noisy visual, action and track tokens join eight register tokens. Bidirectional full attention allows cross-modal interaction. Separate heads predict three modality noise streams and visibility logits. The backbone has twelve layers, width 768 and twelve heads; default action and track horizons are nineteen steps. architectureobjectiveshyperparameters
6. Experiments & results
JOPAT learns robot actions together with future image latents and 2D point tracks. A shared diffusion transformer lets explicit motion correspondences influence action sampling, while visibility supervision represents missing visual evidence. The paper reports strong LIBERO and SO-101 success rates, but precision insertion remains weak. Its central contribution is a richer predicted state for control, supported by modality ablations rather than by visual realism alone.
6.1 Read the original evidence
Table 1. The printed benchmark lead is small against the strongest listed baseline. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Each suite has two columns: SR is success rate, where higher is better, and RK is rank, where lower is better. Compare SR before reading boldface or ranks. JOPAT's highlighted row leads Object, Goal and Long, while CogVLA leads Spatial. The dagger means a result reproduced by the paper's authors; it does not imply that every baseline used matched training resources. Section 5.1 specifies forty tasks, fifty demonstrations per task, LIBERO-90 pretraining for 100K steps and 10K target-suite finetuning steps. This standard protocol differs from the DROID-pretrained modality ablation shown later, even though both print 96.4 for Long. simulationtable1-arithmeticmodality
What it supports. JOPAT reports 97.8% average SR against CogVLA's 97.4%, and 96.4% on Long against 95.4%. The larger gap to UWM, at 84.7% average and 77.6% Long, motivates examining representation design. The benchmark ranking alone does not demonstrate which component causes the improvement.
Where the evidence stops. Displayed JOPAT suite cells average 97.725%, versus printed 97.8%; pi0.5-KI cells average 94.3%, versus printed 96.0%. Unrounded data are unavailable. These source discrepancies remain visible, and absent uncertainty prevents treating the small leading margin as statistically established.
Table 2. Physical execution improves on average, but precision insertion remains unreliable. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read down each task column before comparing the average. Cook-Soup tests repeated interactions and sequencing; Insert-Peg demands alignment; Push-Tomato tests object displacement; Pick-Grocery tests unseen-object generalization. Appendix A.2 gives the missing denominator: ten evaluation rollouts per task per method. JOPAT is pretrained on DROID video and finetuned separately with fifty demonstrations per task. Thus a ten-percentage-point difference within a column represents one rollout in this evaluation. The nearby Figure 3 shows successful task stages, whereas this table measures how often success occurred. The average summarizes these four task rates and is not an independent larger sample for each task. real-resultsrobot-protocolfailures
What it supports. JOPAT's 57.5% average exceeds ACT's 40% and UWM's 32.5%. Its 100% Push-Tomato result accompanies only 10% Insert-Peg success. The practical message is therefore uneven competence: explicit temporal motion helps this task collection, while sparse image-plane correspondences leave substantial precision-contact difficulty unresolved.
Where the evidence stops. Ten trials per task provide limited precision, with no reported confidence intervals or seed variation. Successful snapshots are illustrative, and 100% on these pushing trials does not establish reliability over arbitrary starts, contact conditions or deployments.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| LIBERO four-suite manipulation Forty tasks; fifty demonstrations/task; LIBERO-90 pretraining for 100K steps, target-suite finetuning for 10K. | Reported average 97.8; Spatial 97.2, Object 98.9, Goal 98.4, Long 96.4. Success rate (%) | CogVLA: average 97.4, Long 95.4; UWM: average 84.7, Long 77.6. Highest printed average; JOPAT does not win Spatial. Baseline training resources are not controlled here, and no uncertainty establishes the small lead. simulationtable1-arithmetic |
| SO-101 physical manipulation Four tasks; DROID pretraining; fifty demonstrations/task; ten evaluation rollouts/task/method. | Cook-Soup 60; Insert-Peg 10; Push-Tomato 100; Pick-Grocery 60; average 57.5. Success rate (%) | ACT average 40; UWM 32.5: differences of 17.5 and 25.0 percentage points. Executed robot evidence, with severe insertion weakness and very small per-task samples. real-resultsrobot-protocol |
| LIBERO-Long predicted-modality ablation Ten tasks; identical DROID action-free pretraining, optimization and downstream protocol across variants. | Joint 96.4. Average success rate (%) | Latent-only 77.4; Track-only 26.2: gains of 19.0 and 70.2 points. Supports complementary targets. Removing a prediction stream changes more than its semantic content; this is not a matched-capacity test. modality |
| SO-101 visibility ablation Four physical tasks under Appendix A.2 protocol. | With visibility 57.5. Average success rate (%) | Without visibility 47.5; Cook-Soup improves 40→60; Pick-Grocery remains 60. Consistent with occlusion-related benefit; unchanged performance on one task does not rule out generic regularization. visibilityrobot-protocol |
| LIBERO-Long action-free pretraining 10/25/50 demonstration budgets; DROID or matched 10k-video OpenVid-1M subset. | DROID: 64.2/82.7/96.4; OpenVid-1M: 48.5/84.6/95.1. Success rate (%) | No pretraining: 11.9/31.6/66.1. Both sources help; robot-domain advantage is largest at ten demonstrations. This is distinct from Table 1's pretraining protocol. video-pretrainingsimulation |
| Modified LIBERO-Long distribution shift Five reported tasks; initialization expanded 5 cm, replaced background object and unseen foreground distractor. | 0.66, as printed. Average success fraction | Diffusion Policy 0.32; UWM 0.34; fine-tuned pi0.5 0.53. JOPAT wins the aggregate, while pi0.5 wins three tasks. Simultaneous perturbations do not isolate occlusion robustness. ood |
| SO-101 action-free pretraining Same fifty demonstrations/task; Appendix A.2 evaluation. | DROID pretraining 57.5. Average success rate (%) | Without pretraining 10.0. Figure 4 supports transfer under this data budget; it does not establish performance without action-labeled finetuning. robot-pretrainingrobot-protocol |
| SO-101 future-observation offset sensitivity H=16/64/128; Appendix A.2 protocol. | 57.5/57.5/60.0 in Figure 5. Average success rate (%) | Insert-Peg: 10/0/0; Cook-Soup: 60/60/80. Task tradeoffs matter. The prose's equal-average description overstates agreement with the longest offset; no uncertainty supports a saturation claim. horizonrobot-protocol |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 3. Predicting both visual latents and tracks outperforms either predictive stream alone. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The two halves continue one ten-task comparison; they are not different splits or experiments. Read Latent-only, Track-only and Joint within each task, then use the shaded average row. The caption explicitly fixes DROID action-free pretraining, and Section 5.3 states that optimization and downstream finetuning are shared. Latent-only removes future track prediction; Track-only removes future visual-latent prediction. These names describe predicted modalities, so Track-only should not be read as proving that the policy receives no RGB observation. Joint includes visibility along with coordinates, visual latents and actions. Keep this controlled representation comparison separate from the heterogeneous baseline leaderboard. modalityarchitecture
What it supports. Joint reaches 96.4% average SR, compared with 77.4% for Latent-only and 26.2% for Track-only. It exceeds both variants in every displayed task. The comparison supports complementary visual and correspondence targets, rather than the stronger proposition that tracks by themselves are an adequate control representation.
Where the evidence stops. Removing a modality also changes the token stream and supervision. The experiment does not separately match parameter count, token count or auxiliary-loss effects, nor does it isolate the contribution of track attention specifically during inference.
Table 5. Visibility supervision improves several tasks, with the largest observed gain on Cook-Soup. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare the two rows within each task. This ablation removes the visibility head, while the method separately diffuses 2D coordinates. Section 4.2 states that ground-truth visibility is not encoded as input, and Equation (6) trains a visibility output with binary cross entropy. The intended role is to distinguish missing evidence from visible correspondences during learning. Cook-Soup changes by twenty points, Insert-Peg and Push-Tomato by ten each, and Pick-Grocery does not change. Under the physical protocol, these differences correspond to two, one, one and zero additional successful trials, respectively. Read them as observed task outcomes, not direct measures of tracking calibration. tracksobjectivesvisibilityrobot-protocol
What it supports. The average increases from 47.5% to 57.5%. This is consistent with the authors' explanation that visibility supervision helps through self-occlusion and transient loss of sight. The gain is not universal across tasks, and the table measures downstream control rather than the accuracy of individual visibility predictions.
Where the evidence stops. An unchanged Pick-Grocery score cannot establish that the head's benefit is unrelated to capacity or regularization. The source does not vary occlusion independently or provide a matched-capacity control, so the proposed mechanism remains partly interpretive.
Figure 5. Longer future-image offsets change task tradeoffs more than the aggregate success rate. Original paper, p. 18 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. The horizontal variable is future-observation offset H, not the action-execution prefix and not automatically the track horizon. The default configuration lists action and track horizons separately as nineteen steps. Follow the black diamond markers for the four-task mean, then compare individual curves: Push-Tomato stays at 100%, Insert-Peg drops from 10% to zero, and Cook-Soup rises from 60% to 80% only at the longest offset. Pick-Grocery peaks at 70% in the middle. The gray saturation annotation is an author interpretation. Appendix A.3.2 discusses increased future uncertainty, but the plot itself records task success rather than an uncertainty estimate. horizonrobot-protocolhyperparameters
What it supports. The averages are 57.5%, 57.5% and 60.0%; improvements and regressions partly cancel. H=16 matches H=64 on this average and is close to H=128, while retaining the only nonzero insertion result. Choosing an offset from the mean alone can therefore hide important differences between sequencing and precision tasks.
Where the evidence stops. Appendix A.3.2 says H=16 matches longer offsets on average, but Figure 5 prints 60.0% at H=128. Preserve that mismatch. Ten rollouts per task and absent error bars do not establish saturation, and changes to other horizons in this sweep are not explicitly documented.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors identify sparse-track detail loss, tracker supervision errors, static-camera dependence and a bottleneck above 20 Hz. Appendix failures involve reflective lids, insertion geometry, pushing localization and cluttered grasp locations. These diagnoses are qualitative; 2D temporal consistency does not supply missing 3D contact geometry. limitationsfailures
Simulation rollout counts, seeds and uncertainty are not supplied. Physical comparisons use ten trials per task. In Table 1, displayed JOPAT suite values average 97.725%, versus printed 97.8%; pi0.5-KI cells average 94.3%, versus printed 96.0%. Original values remain reported, without silently repairing aggregate columns. simulationrobot-protocoltable1-arithmetic
7.2 Questions for discussion
- How much benefit requires track-to-action attention during sampling, beyond training regularization?
- Would calibrated 3D or contact sensing improve insertion while preserving the motion-transfer gains?
8. Reproducibility audit
8.1 Requirements and known gaps
Use the stated encoders, tracker and data protocols. Table 8 specifies AdamW at 1e-4, weight decay 1e-6, batches 144/72, 1000 warmup steps and constant/cosine schedules. Reported resources are four H200 GPUs for about five days of pretraining, one H200 for about one day of finetuning, and RTX 4090 inference near 10 Hz. hyperparametersimplementation
Resolve the track indexing difference between construction at t+1:t+Hp and sequence/loss indices at t:t+Hp-1. Temporal padding for nineteen steps with patch size two, coordinate normalization, tracker configuration, video subset IDs, software versions and detailed success/reset rules remain unspecified. These are implementation questions, not measured failures. tracksarchitectureobjectiveshyperparametersvideo-pretrainingrobot-protocol
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Does the track stream help through inference-time attention?
Reader-proposed check, not performed: reproduce the DROID-pretrained LIBERO-Long Joint setting and compare full attention with a mask preventing every non-track query, including action, visual and register tokens, from reading track keys. Keep all tokens, heads, targets, parameter counts and training budgets fixed. Evaluate the full-trained model with and without this mask using paired starts and diffusion seeds, then train a matched masked model to distinguish test-time distribution shift from a persistent information-flow effect. An inference-only drop that persists against the retrained control would support a useful track-to-control pathway. Recovery after masked training would instead weaken the claim that access to track tokens during sampling is essential. Report task rates and variation across repeated training seeds. architectureobjectivesmodalitysimulationhyperparameters
Check 2: Separate visibility semantics from generic extra supervision
Reader-proposed check, not performed: compare correct visibility targets with visibility labels shuffled within clips, retaining the same head, loss weight, coordinate targets and optimization. Use held-out SO-101 Cook-Soup and Pick-Grocery resets with predefined, matched levels of arm occlusion; increase trial counts beyond the paper's ten and repeat training seeds. Keep ground-truth visibility out of all model inputs. Measure task success and visibility calibration against manually checked labels, separating visible and occluded intervals. A correct-label advantage that grows with occlusion would support missing-evidence supervision. Similar gains from shuffled labels, or no interaction with occlusion, would suggest a broader regularization effect and weaken that specific explanation. tracksobjectivesvisibilityrobot-protocolimplementationhyperparameters
8.3 Reading coverage
Visual audit: The title/authors/version page, all five original figures and all eight tables were visually read. Pages 3–5 verify notation, attention, objectives and execution distinctions; pages 6–10 verify evaluations and limitations; pages 16–19 verify the robot protocol, compute, hyperparameters, supplementary ablations and failure diagnoses. All six final original crops were inspected at their final bounds; the two table crops with short caption notes retain protocol and dagger definitions. Figure 2's upstream epsilon labels were checked against the output equations, and Figure 5 against Appendix A.3.2. Printed Table 1 aggregate discrepancies remain disclosed. Reference-only pages 11–15 were read in the complete text chunks but not visually rendered. No separate supplement, external code or rollout video was inspected.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 16, 17, 18, 19. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related work
- 3 Preliminaries: world-action models
- 4 Methods, including 4.1–4.3
- 5 Experiments, including 5.1–5.6 and 5.5 action-free pretraining
- 6 Conclusion and discussion
- Acknowledgments
- References
- A Supplementary material: A.1, A.2, A.3.1, A.3.2 and A.4
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- All six text chunks were read. The retained PDF was visually inspected for all five figures and eight tables; reference-only pages 11–15 were read as text.
- Code and external resources were not inspected; experiments were not reproduced. No separate supplement was supplied.
- Version scope: the inspected title page identifies arXiv:2605.23856v1, 22 May 2026. Title and six authors match the catalog; no revision discrepancy was observed. Other editions were not supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title, author/affiliation block and arXiv margin
The title is Point Tracking Improves World Action Models. Authors, in order: Jiarui Guan, Wenshuai Zhao, Yue Pei, Ziliang Chen, Arno Solin, Juho Kannala. The margin identifies arXiv:2605.23856v1 [cs.RO], 22 May 2026. Six institutions are listed; Wenshuai Zhao is corresponding author.
Go to primary source ↓state-interfacePDF pp. 1–3, Abstract, Section 1, Figure 1 and Section 3
JOPAT augments future visual latents with point correspondences and visibility. Section 3 defines RGB observations, end-effector pose deltas and gripper commands, and the policy as the action marginal of a joint future-state/action model. Figure 1 distinguishes visible and invisible query points qualitatively.
Go to primary source ↓architecturePDF p. 4, Figure 2 and Section 4.1, displayed denoising equations
Two RGB frames condition a shared DiT via AdaLN. Noisy action, future visual and track tokens plus registers receive bidirectional full attention. Separate outputs predict modality noise and visibility. Default future-image offset is 16; action and track horizons are 19. Figure 2(b) places epsilon labels on a strip entering the DiT, whereas the equations define hatted epsilon as transformer outputs; the schematic is not a literal complete computational graph.
Go to primary source ↓tracksPDF p. 5, Section 4.2, Eqs. (2)–(4); p. 4, Figure 2(a,c)
Tracks from a 25 by 25 reference-frame grid give 625 points, reshaped into a two-coordinate spatiotemporal grid and patchified in 3D. Coordinates and visibility have separate decoders. Ground-truth visibility is never supplied to the track encoder. Construction is indexed t+1 through t+Hp, unlike the t through t+Hp-1 indices elsewhere.
Go to primary source ↓objectivesPDF p. 5, Section 4.3, Eqs. (5)–(7) and inference paragraph
Modality diffusion timesteps are sampled independently. Squared noise-prediction losses supervise actions, visual latents and track coordinates; a separate BCE supervises visibility. Video-only training omits the action loss. At inference all three token streams are jointly denoised from Gaussian noise, with only actions executed.
Go to primary source ↓simulationPDF p. 6, Section 5 experiment design, Section 5.1 and Table 1, JOPAT/CogVLA/UWM rows
Each of four LIBERO suites has ten tasks and fifty demonstrations per task. JOPAT uses 100K LIBERO-90 pretraining steps and 10K target-suite finetuning steps. JOPAT SRs are 97.2, 98.9, 98.4 and 96.4, with printed average 97.8; CogVLA average is 97.4 and Long 95.4; UWM average is 84.7 and Long 77.6. Dagger marks the authors' reproduced PD-VLA result. No simulation rollout count, seed count or uncertainty is given here.
Go to primary source ↓table1-arithmeticPDF p. 6, Table 1, JOPAT and pi0.5-KI suite/average columns
The JOPAT suite cells are 97.2, 98.9, 98.4 and 96.4, with average printed 97.8. The pi0.5-KI cells are 98.0, 97.8, 95.6 and 85.8, with average printed 96.0. Reader arithmetic gives 97.725 and 94.3, respectively; underlying unrounded observations are not supplied.
Go to primary source ↓real-resultsPDF p. 7, Section 5.2, Figure 3 and Table 2
On SO-101, JOPAT obtains 60/10/100/60 percent on Cook-Soup/Insert-Peg/Push-Tomato/Pick-Grocery, average 57.5. ACT obtains 40/0/70/50, average 40; UWM obtains 10/0/80/40, average 32.5. Figure 3 shows start, middle and successful end states; it is not an estimate of success frequency.
Go to primary source ↓modalityPDF p. 8, Section 5.3 and Tables 3–4
Under identical DROID action-free pretraining, optimization and finetuning settings, LIBERO-Long average SR is 77.4 for Latent-only, 26.2 for Track-only and 96.4 for Joint. Table 3 lists ten task rows; Joint exceeds both variants on each task. Table 4 real-robot averages are 35.0, 15.0 and 57.5. The variants remove predicted modalities, not necessarily current RGB conditioning.
Go to primary source ↓visibilityPDF p. 8, Section 5.4; p. 9, Table 5
Without versus with visibility, Cook-Soup is 40 versus 60, Insert-Peg 0 versus 10, Push-Tomato 90 versus 100 and Pick-Grocery 60 versus 60; average SR is 47.5 versus 57.5. The authors attribute the benefit to missing-evidence supervision. No matched-capacity or independently varied occlusion control is reported.
Go to primary source ↓video-pretrainingPDF p. 6, Section 5 experiment design; pp. 8–9, Section 5.5 and Table 6
At 10/25/50 demonstrations, LIBERO-Long SR is 11.9/31.6/66.1 without pretraining, 64.2/82.7/96.4 with DROID and 48.5/84.6/95.1 with OpenVid-1M. The generic-video comparison uses a matched 10k-video subset; video-only pretraining masks actions. The supplied text does not identify the sampled video IDs.
Go to primary source ↓oodPDF p. 9, Section 5.6 and Table 7
The modified LIBERO-Long setting expands initialization by 5 cm, replaces a background object and adds an unseen foreground distractor. Five task rows have JOPAT SR 0.88/0.78/0.68/0.48/0.50; printed average 0.66 versus 0.32 Diffusion Policy, 0.53 fine-tuned pi0.5 and 0.34 UWM. Pi0.5 wins three individual rows. Table 7 uses fractions, not percentage-valued cells.
Go to primary source ↓limitationsPDF p. 10, Section 6, Limitations
The authors identify sparse grid coverage of fine deformation, dependence on tracker quality, a high-frequency control bottleneck above 20 Hz, and a static-camera design that limits mobile manipulation with ego-motion.
Go to primary source ↓implementationPDF p. 16, Appendix A.1, Architecture, Training data, Inference and Compute resources
The future visual encoder is a frozen SDXL VAE; targets use CoTrackerV3. The first eight decoded actions execute before replanning. Reported compute is four NVIDIA H200 GPUs for approximately five days of action-free pretraining, one H200 for approximately one day of task finetuning, and an RTX 4090 for approximately 10 Hz inference.
Go to primary source ↓robot-protocolPDF p. 16, Appendix A.2, Training data and finetuning; Inference and evaluation
Each SO-101 task uses DROID action-free pretraining followed by separate finetuning on fifty action-labeled demonstrations. Evaluation uses ten rollouts per task per method. Defaults are two RGB frames, future-image offset 16, action/track horizon 19, ten DDIM steps and an eight-action execution prefix. Exact reset distributions, success tolerances and seeds are not enumerated.
Go to primary source ↓hyperparametersPDF p. 17, Table 8, all parameter groups
Inputs are 224 by 224; SDXL latents are 28 by 28 by 4. Conditioning uses ImageNet-initialized ResNet-18. DiT has twelve layers, width 768, twelve heads, MLP ratio four and eight registers. Track patch size is (2,5,5). AdamW uses learning rate 1e-4, weight decay 1e-6, betas (0.9,0.999), batches 144/72, constant/cosine schedules and 1000 warmup steps. All loss weights are one. Diffusion uses squaredcos_cap_v2, 100 training steps and ten DDIM inference steps. Table 8 does not specify temporal padding, coordinate normalization or software versions.
Go to primary source ↓robot-pretrainingPDF p. 17, Appendix A.3.1; p. 18, Figure 4
With fifty real-robot demonstrations per task, DROID pretraining changes average SR from 10.0 to 57.5 percent. Without/with pretraining: Push-Tomato 10/100, Cook-Soup 0/60, Pick-Grocery 30/60 and Insert-Peg 0/10.
Go to primary source ↓horizonPDF p. 17, Appendix A.3.2; p. 18, Figure 5, axes, legend and points
Figure 5 varies future-observation offset H over 16, 64 and 128. Black average markers read 57.5, 57.5 and 60.0 percent. Insert-Peg is 10/0/0, Push-Tomato 100/100/100, Cook-Soup 60/60/80 and Pick-Grocery 60/70/60. The prose says H=16 matches longer offsets on average, which is not literally true of the H=128 plotted value. The figure labels saturation at H=16 without error bars.
Go to primary source ↓failuresPDF pp. 18–19, Appendix A.4, qualitative behavior and task-level failure modes
The authors describe temporally coherent tracks through occlusion and boundary motion. Remaining failures involve specular pot lids and unusual orientations, millimeter-scale insertion geometry, compounding localization/approach errors during pushing, and grasp-location overfitting in clutter. These are qualitative diagnoses, not quantified failure frequencies.
Go to primary source ↓8.5 Primary sources
Point Tracking Improves World Action Models ↗
PDF · 8,427 extracted words
Source fingerprint
37f0c636baa16b7003c80ce209149c50ce48dbca56ec8e37c56c285f9ea983c3