Point Tracking Improves World Action Models
Classification
View four quadrants- Major category
- WAMs
- Architecture
- One Model
- Prediction paradigm
- Joint prediction
- Subcategories
- Joint video-action modelingMemory & long-horizon modelingGeneralization & action alignment
- Source review status
- Verified from primary sources
Contribution
We propose JOPAT, a JOint Pixel-And-Track World-Action Model that predicts latent visual observations, 2D point tracks with visibility, and actions in a single denoising diffusion transformer.
Abstract
Robot policy learning benefits from world-action models that capture environment dynamics, but pixel-level prediction entangles dynamics with nuisance factors such as lighting and texture, making learned representations vulnerable to task-irrelevant visual variation. We propose JOPAT, a JOint Pixel-And-Track World-Action Model that predicts latent visual observations, 2D point tracks with visibility, and actions in a single denoising diffusion transformer. The key insight is that tracks provide an explicit representation of motion that captures long-horizon dynamics and remains robust under occlusion or partial out-of-frame motion, offering greater utility than modeling pixel appearance alone. On LIBERO and real-world LeRobot tasks, JOPAT improves over pixel-based baselines, with the largest gains on long-horizon tasks involving occlusion, object interaction, and off-screen motion.
Affiliations
Aalto University; ELLIS Institute Finland; University of Oulu; Sun Yat-sen University; Peng Cheng Laboratory; Beihang University