EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video
1. Paper overview
In one sentence: EgoDex makes dexterous human motion a large-scale supervised learning target, but its strongest sampling results measure offline trajectory coverage rather than executable robot control. e-datasete-modalitiese-comparisone-goale-usecases
| At a glance | What to know |
|---|---|
| Research problem | Reader analysis Dexterous imitation learning needs both diverse manipulation and usable motion labels. Robot teleoperation ties collection to hardware and operator effort; ordinary egocentric video often lacks detailed hand poses. EgoDex records human manipulation and pose estimates together, making annotation part of capture. The authors call this passively scalable, although the reported collection still uses organized sessions and collector-entered metadata. e-probleme-collectione-modalities |
| Core mechanism | Source description The dataset contains 338,000 episodes, 90 million frames, 829 hours and 194 tabletop tasks, occupying 2.0 TB compressed. Figure 1 instead displays rounded 300K episodes, 800 hours and 200 tasks; the precise text totals are retained here. e-datasete-identity |
| A key reported result | Dexterous trajectory prediction: EncDec+FM: average 0.051/0.041/0.038 and final 0.070/0.047/0.041 for K=1/5/10. Best-of-K average 3D keypoint distance and reported final distance, meters; lower is better.. Fixed within-task test set; 2-second horizon; six architecture/representation combinations. EncDec+BC: 0.044 average and 0.060 final at every K. Dec+BC: 0.045 and 0.062. BC wins at K=1; FM wins at K=5 and 10. Average distance aggregates 12 wrist/fingertip positions across time. This positional metric does not directly score wrist orientation or manipulation success; no uncertainty estimates are supplied. e-comparisone-evaluatione-action |
| Reading caution | Reader analysis The authors acknowledge limited scene/background diversity and pose errors under occlusion or fast motion. Tracked labels are model predictions, so agreement with them is not independent motion-capture validation. e-limitationse-modalities |
Core contributions
- Source description
The dataset contains 338,000 episodes, 90 million frames, 829 hours and 194 tabletop tasks, occupying 2.0 TB compressed. Figure 1 instead displays rounded 300K episodes, 800 hours and 200 tasks; the precise text totals are retained here. e-datasete-identity
- Source description
Two offline benchmarks predict dexterous trajectories, with or without an observed endpoint image. Baselines compare two Transformer architectures and three policy representations, plus horizon, data-size, model-size and goal-conditioning variants. e-benchmarkse-models
Figure 3. The learning targets begin as synchronized human pose annotations. Original paper, p. 5 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Start at the skeleton on the left: it connects the camera and upper body to the hands, whose internal joints appear in an enlarged inset. Then scan the nine examples across the right panel. They include opening a bag, removing a book and screw, folding a shirt, decluttering, opening a case, unscrewing a cap, tying shoelaces and washing a cup. The caption defines the colored fingertip trails as the preceding 0.5 seconds of motion. They are recorded pose annotations, not future model predictions. Section 4.1 later reduces the available skeleton to wrist poses and fingertip positions for the benchmark action representation. e-skeletalvisuale-actione-modalitiese-jointse-limitations
What it supports. The figure shows why wrist-only motion would miss part of this dataset's purpose: the tasks involve coordinated finger behavior across varied objects. EgoDex records a richer skeleton than its 48-dimensional benchmark action target uses, leaving room for alternative representations. The illustrated examples establish modality and behavioral scope rather than prediction accuracy.
Where the evidence stops. The annotations are themselves tracking-model outputs. Occlusion and rapid movement can corrupt them, and Appendix A.3 warns that finger confidence must be interpreted relative to wrist detection. Some joint labels overlap in the original inset; the appendix provides the readable joint inventory.
2. Motivation
2.1 The problem and the proposed response
Dexterous imitation learning needs both diverse manipulation and usable motion labels. Robot teleoperation ties collection to hardware and operator effort; ordinary egocentric video often lacks detailed hand poses. EgoDex records human manipulation and pose estimates together, making annotation part of capture. The authors call this passively scalable, although the reported collection still uses organized sessions and collector-entered metadata. e-probleme-collectione-modalities
2.2 What this reading follows
Imagine predicting how both hands will move while someone unscrews a cap or ties a shoelace. EgoDex supplies synchronized egocentric images, language and tracked skeletal poses for this problem, with 829 hours spanning 194 tasks. The interesting result is not a single winning policy: behavior cloning gives the better single prediction, while flow matching gives better coverage when several predictions can be compared with the recorded future. An additional endpoint image changes the problem again by revealing where the motion should finish. Read the visuals as a progression from annotation to prediction to evaluation, keeping the human–robot embodiment gap in view. e-datasete-modalitiese-comparisone-goale-usecases
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Datasets |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
3.1 Evidence-based assessment
Supports the recorded classification
The recorded dataset classification fits the principal contribution: human egocentric video, dexterous hand behavior and 3D pose annotations. Architecture and prediction quadrants are not applicable to the resource itself. Its inverse-dynamics baseline does not turn EgoDex into a joint future-image/action world model; world modeling is proposed as a downstream use. e-datasete-modalitiese-benchmarkse-usecases
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Turn captured hand motion into a prediction target
EgoDex's first design choice is to collect tracking signals at the same time as video. Vision Pro and ARKit provide camera information and skeletal poses while the person manipulates objects naturally; language is assembled from collector metadata. The stored skeleton is richer than the benchmark action. For each hand, the benchmark keeps the wrist's position and orientation plus the five fingertip positions, producing 48 coordinates per timestep. The current camera frame supplies the coordinate reference, and a chunk represents relative motion. This distinction matters: learning all stored joints and learning the benchmark target are different tasks. A reader's inference is that confidence handling can affect both supervision and evaluation, because Appendix A.3 makes finger reliability conditional on detecting the wrist. The paper does not specify a filtering threshold for resolving that issue. e-collectione-modalitiese-actione-jointse-training
Figure 7. The baselines vary architecture and policy representation independently. Original paper, p. 20 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read upward from the image, language instruction and proprioceptive observation. ResNet, CLIP and an MLP turn these inputs into conditioning features. The middle tier offers two alternatives: the left decoder mixes conditioning and action tokens, while the right encoder processes observations and supplies the decoder through the rightward cross-attention arrow. The top tier lists three alternative policy representations, not three consecutive stages. Appendix A.4 identifies CLIP as frozen and the image encoder as pretrained. It also restricts these experiments to the current image and state, even though the general benchmark equations allow an observation history. e-modelse-traininge-benchmarkse-action
What it supports. This is a controlled baseline design space for predicting human hand trajectories. Switching from decoder-only to encoder-decoder architecture changes how observations condition actions; switching BC, DDPM or FM changes the policy representation. The figure contains no future-image prediction branch, so it does not establish a joint world/action model.
Where the evidence stops. The diagram is schematic: it does not specify attention masks, the optional goal-image insertion path or loss formulas. The surrounding text delegates other hyperparameters to X-IL. No stronger token-level implementation claim is inferred from the colored blocks or output arrows.
5.2 Separate prediction quality from sample coverage
Given an image, current skeletal state and language, the baselines predict a future action chunk. The benchmark definition permits history, but Appendix A.4 says the implemented models use only the current observation and state. An encoder-decoder architecture changes how conditioning reaches the action decoder; BC, diffusion and flow matching provide alternative policy representations. The evaluation then introduces another choice: how many trajectories may be sampled before the recorded future is used to pick the closest one. Deterministic BC receives no benefit from increasing K, while the stochastic models gain opportunities to cover the demonstrated motion. Table 2 therefore supports two conclusions at once: BC is stronger at K=1, and flow matching is stronger under best-of-many scoring. The reader should keep that scoring choice attached to every claim of superiority. e-benchmarkse-modelse-traininge-evaluatione-comparison
5.3 Ask what the model knows about the endpoint
Inverse dynamics adds the image observed at the end of the prediction horizon. That image supplies information about where the hands and scene should finish; the model estimates the intervening trajectory. Table 4 shows a larger relative improvement in final distance than in average distance, consistent with the authors' endpoint-anchoring explanation. This is distinct from a world model generating a future observation and a controller acting to realize it. The paper proposes robotics and world-model applications, but evaluates offline human trajectories. The same boundary applies to transfer: Appendix A.1 reports mixed results on tasks designated OOD, while the complete inventory leaves their exclusion details open. A reader's inference is that usable downstream control requires both a verified generalization protocol and an additional mechanism for embodiment transfer and action execution. e-benchmarkse-goale-usecasese-oode-taskinventory
5.4 Training and inference
During training
The reported recipe uses Adam at 10^-4, 50,000 gradient steps, batch size 2,048 and eight A100 80GB GPUs on one node with 48 physical/96 logical CPUs. A run takes approximately 72 hours. Images become 224×224. DDPM and flow matching use 16 sampling steps. Remaining hyperparameters are delegated to X-IL defaults; explicit loss formulas and ResNet freezing status are not given. e-training
A random 1% subset of each task forms the fixed test set; the remaining 99% permits a user-chosen training/validation split. The main test therefore measures within-task prediction. Appendix A.1 separately evaluates six tasks designated OOD. e-evaluatione-ood
During inference
The implemented policy receives the current observation and state plus language, optionally adding the endpoint image, and predicts a future hand-motion chunk. Evaluation samples stochastic policies K times and selects the trajectory closest to the recorded target. That target-based selection is an evaluation oracle, not an executable decision rule or feedback controller. e-benchmarkse-evaluatione-training
5.5 Implementation flow
- Capture synchronized demonstrations
Apple Vision Pro running visionOS 2 and ARKit records bare-hand demonstrations in roughly 10–15-minute sessions. Pause/resume marks episode boundaries. Reversible task pairs and reset-free tasks improve collection yield; reset periods are excluded. e-collectione-taskdesign
- Retain rich observations and confidence
RGB at 1920×1080, camera calibration, joint poses and confidence are recorded at 30 Hz. GPT-4 consolidates collector metadata into language descriptions. Poses use SE(3) matrices. Finger confidence is relative to the wrist: high fingertip confidence cannot rescue an undetected hand. e-modalitiese-joints
- Construct the action target
Each hand contributes wrist position, 6D wrist orientation and five 3D fingertip positions: 2 × (3 + 6 + 15) = 48 dimensions. Action chunks are relative trajectories expressed in the current camera frame, rather than the complete tracked skeleton. e-action
- Encode and predict
A pretrained ResNet encodes the resized image, frozen CLIP encodes language, and an MLP embeds proprioception. Figure 7 contrasts a shared decoder token stream with an observation encoder supplying cross-attention to an action decoder; BC, DDPM and flow matching are alternative policy representations. e-modelse-training
6. Experiments & results
EgoDex pairs egocentric video with tracked human hand poses to support dexterous imitation learning. Its 829 hours and 194 tasks underpin offline hand-trajectory benchmarks: behavior cloning leads for one prediction, while flow matching leads under best-of-many scoring. Robot transfer and world modeling remain proposed uses.
6.1 Read the original evidence
Table 2. The winner changes when evaluation can choose among several predictions. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Fix the prediction horizon at two seconds and read the two blocks separately: average distance on the left and reported final distance on the right, both in meters. Within each block, K is the number of sampled trajectories offered to the best-of-K evaluator. Compare EncDec+BC with EncDec+FM first. BC repeats the same score because it is deterministic. FM improves as more samples give the evaluator more chances to match the recorded motion. Then compare Dec and EncDec rows within the same policy representation and K. Section 4.3 defines average error over twelve wrist/fingertip positions and the predicted timesteps. e-comparisone-evaluatione-action
What it supports. EncDec+BC achieves 0.044 m average error at K=1, better than EncDec+FM's 0.051 m. At K=10, EncDec+FM reaches 0.038 m average and 0.041 m final distance, compared with BC's unchanged 0.044 m and 0.060 m. This supports a sampling-versus-single-prediction tradeoff, not unconditional FM superiority.
Where the evidence stops. Best-of-K selection uses the recorded future. A deployed controller would need another way to select a trajectory. The table reports neither uncertainty nor task success, and its positional distances do not directly test the wrist-orientation components of the action target.
Table 5. The appendix reports markedly different transfer errors across tasks. Original paper, p. 15 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use the first row as the in-distribution reference, then compare each named task with that row within the same metric column. Appendix A.1 identifies the model as Dec+BC and places the six designated OOD tasks in a separate extra folder. Jigsaw Puzzle stays close to the reference, while the other rows show progressively larger average and final errors. These rows are individual task results, not confidence intervals or a single aggregate OOD score. The authors associate the variation with task similarity, but the table itself supplies no independent measure of similarity or per-task sample counts. e-oode-taskinventory
What it supports. Jigsaw Puzzle reports 0.047 m average and 0.065 m final error, close to the in-distribution 0.045 m and 0.062 m. Stamp Paper reports 0.099 m and 0.162 m. Thus the reported transfer behavior is heterogeneous; a favorable task cannot summarize all six tasks or establish general dexterous competence.
Where the evidence stops. Appendix A.2 also lists task names corresponding to these OOD rows. This may reflect a release-wide inventory, but the PDF does not resolve exact training membership. Preserve the authors' designation while treating exclusion verification as open; the overlap alone does not prove leakage.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Dexterous trajectory prediction Fixed within-task test set; 2-second horizon; six architecture/representation combinations. | EncDec+FM: average 0.051/0.041/0.038 and final 0.070/0.047/0.041 for K=1/5/10. Best-of-K average 3D keypoint distance and reported final distance, meters; lower is better. | EncDec+BC: 0.044 average and 0.060 final at every K. Dec+BC: 0.045 and 0.062. BC wins at K=1; FM wins at K=5 and 10. Average distance aggregates 12 wrist/fingertip positions across time. This positional metric does not directly score wrist orientation or manipulation success; no uncertainty estimates are supplied. e-comparisone-evaluatione-action |
| Goal-conditioned inverse dynamics Default Dec+BC comparison with versus without an observed endpoint image. | With goal image: 0.035 / 0.029. Average / final distance, meters. | Without goal image: 0.045 / 0.062; reported relative reductions are 22% / 53%. Endpoint information strongly helps endpoint accuracy. This changes the information available to the model and does not establish future prediction without a goal. e-goale-benchmarks |
| Prediction horizon sensitivity Separately trained Dec+BC models at H=30/60/90; Appendix EncDec+FM comparison at K=10. | Dec+BC: 0.031/0.049, 0.045/0.062, 0.053/0.069 for 1/2/3 seconds. Average / final distance, meters. | EncDec+FM average distance at K=10 also rises: 0.026, 0.038, 0.047. Longer horizons worsen positional prediction under both settings; sampling does not remove horizon sensitivity. e-horizon |
| Dataset and model scaling Dec+BC data subsets of 5%, 10%, 25%, 50%, 100%; separate 200M-versus-500M parameter comparison. | Both plotted errors fall with more data; 500M model: 0.045 / 0.062. Average / final distance, meters. | The 200M model reports the same 0.045 / 0.062. More data helps this baseline, whereas the tested capacity increase does not. The plot supplies no error bars or fitted scaling law. e-scaling |
| Out-of-distribution task prediction Appendix A.1 Dec+BC evaluation on six tasks labeled OOD; per-task sample counts and horizon are not restated. | Jigsaw Puzzle: 0.047 / 0.065; Stamp Paper: 0.099 / 0.162. Average / final distance, meters. | In-distribution average: 0.045 / 0.062. Transfer varies substantially across tasks. The complete task inventory also includes corresponding task names; split membership needs verification before treating the OOD designation as independently established. e-oode-taskinventory |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table 4. An observed endpoint image particularly improves endpoint accuracy. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Compare downward within each metric column. Both rows use decoder-only behavior cloning; the second adds the goal image. To understand what changed, return to the two estimators in Section 4.2: inverse dynamics receives the image at t+H in addition to the observation, state and language inputs. The model is therefore asked to recover a trajectory between observations rather than infer the future solely from the present. The final-distance column improves more sharply than the average-distance column. The authors interpret this as the endpoint image anchoring the motion and reducing ambiguity about its destination. e-goale-benchmarkse-usecases
What it supports. Average distance falls from 0.045 to 0.035 m and final distance from 0.062 to 0.029 m. The paper reports these as 22% and 53% relative reductions. They establish a useful inverse-dynamics baseline and show that endpoint information addresses a substantial source of prediction ambiguity.
Where the evidence stops. This comparison changes the information supplied to the predictor. The goal is an observed future image, not a generated one. The result does not show that the model can obtain that endpoint autonomously or execute the predicted motion on a robot.
Figure 5. Additional data helps the tested decoder-only behavior-cloning baseline. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Follow either colored curve from left to right, using the legend to distinguish average distance in green from final distance in blue. The vertical axis is positional error in meters, so lower is better. The horizontal axis shows five fractions of the training dataset and uses a logarithmic scale: equal-looking horizontal intervals do not mean equal numbers of added demonstrations. Section 5 fixes these experiments to Dec+BC. Read the trend alongside the separate model-capacity comparison on page 9, where increasing from 200 million to 500 million parameters leaves both reported errors unchanged. Those are different interventions, not a joint data-and-model scaling sweep. e-scalinge-usecases
What it supports. Both error measures improve across the tested data fractions, supporting the practical value of EgoDex's scale for this baseline. Together with the unchanged capacity result, the experiment motivates studying data quality and coverage before assuming that a larger network alone will improve the benchmark.
Where the evidence stops. The figure provides no uncertainty bands, fitted law or extrapolation beyond the available dataset. The source does not give a detailed subset-construction protocol here. A monotonic curve for this model does not establish the same gains for robot transfer or other architectures.
7. Analysis & limitations
7.1 What the evidence leaves open
The authors acknowledge limited scene/background diversity and pose errors under occlusion or fast motion. Tracked labels are model predictions, so agreement with them is not independent motion-capture validation. e-limitationse-modalities
Best-of-K rewards coverage of a recorded trajectory, without testing physical feasibility, contact quality or an autonomous sample-selection mechanism. The described metric omits orientation error; the PDF does not fully specify final-distance sample selection separately from average-distance selection. e-evaluatione-comparisone-action
The paper does not establish participant or session separation for the main random split. Its task inventory/OOD naming overlap leaves exclusion details unresolved, but is not itself proof of leakage. Robot embodiment transfer and egocentric world models are research proposals. e-evaluatione-oode-taskinventorye-usecases
7.2 Questions for discussion
- How much of the best-of-K advantage survives when a trajectory must be selected without the recorded future?
- How sensitive are results to wrist-aware confidence filtering and whole-session holdout?
- What additional evidence would connect human trajectory accuracy to successful dexterous robot execution?
8. Reproducibility audit
8.1 Requirements and known gaps
Reproduction requires the compressed dataset, synchronized calibration/pose interpretation, the fixed test membership, and a pinned X-IL configuration. The PDF names torchcodec for sampled-frame decoding but supplies no exact package versions, confidence-filtering rule, random seeds or dataset license terms. e-collectione-modalitiese-jointse-evaluatione-traininge-identity
Proposed checks: repeat BC-versus-FM comparisons with identical inputs and explicit K, then repeat the task split with whole-session isolation and documented exclusion of the six extra tasks. Report uncertainty and per-task results; these checks have not been run. e-comparisone-evaluatione-collectione-oode-taskinventory
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Test whether the FM advantage depends on oracle selection
Reader-proposed experiment, not performed: train EncDec+BC and EncDec+FM using the same current-observation inputs, training split and two-second horizon. Keep the reported optimization budget and FM sampling-step count fixed. Evaluate average keypoint distance at K=1, 5 and 10, using identical test examples and documenting the rule for final-distance sample selection. Report both the mean error across all sampled trajectories and the oracle best-of-K error, with repeated-seed uncertainty and inference cost. Table 2 predicts BC leading at K=1 but FM leading under best-of-10. If FM's advantage disappears when all samples rather than only the closest are scored, the benefit is specifically coverage of the recorded motion; if it also improves the non-oracle score, the reproduced advantage is broader than that explanation. e-comparisone-evaluatione-traininge-benchmarks
Check 2: Separate held-out tasks from repeated collection sessions
Reader-proposed experiment, not performed: first map the six extra-task labels to the complete task inventory and verify their episode membership against the training manifest. Then compare Dec+BC trained on the described random within-task split with a split holding out entire collection sessions, matching training volume and task proportions. Exclude the six extra tasks from both training conditions and evaluate on common held-out episodes where feasible, reporting all six OOD rows separately. Preserve identical preprocessing, horizon and optimization settings, and disclose whether session identifiers are available. A material error increase under session isolation would weaken an interpretation of the original split as broad generalization. Similar errors with confirmed task exclusion would support the narrower transfer claim without establishing new-scene or robot-execution success. e-collectione-evaluatione-oode-taskinventorye-traininge-usecases
8.3 Reading coverage
Visual audit: Visually inspected the title/authors/version on page 1; dataset context, distribution and collection on pages 2–5; action representation and benchmark equations on page 6; prediction examples, split and metrics on page 7; all main results and scaling evidence on pages 8–9; limitations on page 10; appendix OOD and horizon tables on page 15; complete task inventory on pages 15–18; joint representations/confidence on pages 18–19; and hardware, preprocessing and architecture on pages 19–20. All seven figures and six tables were visually read. All six final original crops were separately viewed, including the corrected Figure 3 crop. Figure 7's encoder-to-decoder cross-attention arrow and alternative policy branches agree with the surrounding description; masks and goal-image routing are not specified. Reference-only pages 11–14 were read as text. Separate supplements, dataset files, code and robot execution remain outside this inspection.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 15, 16, 17, 18, 19, 20. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Abstract
- 1 Introduction
- 2 Related Work (2.1–2.3)
- 3 EgoDex Dataset (3.1–3.4)
- 4 EgoDex Benchmarks (4.1–4.3)
- 5 Experiments
- 6 Research Use Cases
- 7 Conclusion
- References
- A.1 Additional Experiments
- A.2 Complete List of Tasks
- A.3 Complete List of Skeletal Joints
- A.4 Training Details
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The supplied 20-page PDF was read completely; all seven figures and six tables were also visually inspected. Reference-only pages 11–14 were read as text, without a separate visual pass.
- Identity/version note: the inspected title page identifies arXiv:2505.11709v3, 9 March 2026, published at ICLR 2026. Title and all five authors match the catalog; the PDF capitalizes Mouli Sivapurapu and lists Apple. Ryan Hoque, Peide Huang and David J. Yoon have equal-contribution marks. Earlier revisions were not supplied, so revision-to-revision scientific changes were not compared.
- Dataset files, external repositories and linked resources were not inspected; no training, inference or robot experiments were reproduced. The PDF's download statement is not independent verification of availability or licensing.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
e-identityPDF p. 1, title/author block, arXiv margin, Figure 1 and abstract
Exact title and five authors are visible; affiliation Apple; first three authors share equal-contribution marks. The artifact is v3 dated 9 March 2026 and labeled ICLR 2026. Figure 1 uses rounded dataset totals. The abstract gives a public download link, without license terms.
Go to primary source ↓e-problemPDF pp. 2–3, Sections 1–2.3
The paper motivates paired egocentric video and detailed hand tracking through limitations of teleoperation scale and unannotated human video; passive scalability is an author claim.
Go to primary source ↓e-datasetPDF p. 2, Section 1; p. 3, Table 1, EgoDex row; p. 4, Section 3
Text and Table 1 report 338,000 episodes, 194 tasks and 90 million frames; Sections 1 and 3 give 829 hours; Section 3 gives 2.0 TB storage.
Go to primary source ↓e-collectionPDF pp. 4–5, Section 3.1
Collection uses Vision Pro, visionOS 2 and ARKit; sessions last approximately 10–15 minutes; pause/resume marks episodes. Video is compressed, and torchcodec decodes sampled training frames.
Go to primary source ↓e-modalitiesPDF p. 5, Section 3.2
Modalities include 1920×1080 RGB, camera intrinsics/extrinsics, joint position/orientation and confidence at 30 Hz, plus language. GPT-4 consolidates collector metadata. Zero confidence indicates full occlusion.
Go to primary source ↓e-taskdesignPDF pp. 5–6, Sections 3.3–3.4
Task types are reversible pairs, reset-free and reset-required. Resets are not recorded. Diversity emphasizes tabletop manipulation behaviors rather than broad scene coverage.
Go to primary source ↓e-skeletalvisualPDF p. 5, Figure 3 and caption
The left panel shows tracked joints; the right grid shows nine dexterous behaviors with colored fingertip trails from the preceding 0.5 seconds. These trails depict annotations rather than policy forecasts.
Go to primary source ↓e-actionPDF p. 6, Section 4.1
The 48-dimensional action includes both wrists' 3D positions and 6D orientations plus five 3D fingertips per hand. Poses use the current camera frame and chunks are relative trajectories.
Go to primary source ↓e-benchmarksPDF p. 6, Section 4.2, two unnumbered estimator equations
Trajectory prediction conditions on images, skeletal observations and language; inverse dynamics adds the endpoint image. H=30 and H=90 denote one and three seconds.
Go to primary source ↓e-evaluationPDF p. 7, Sections 4.2–4.3 and 5
Test data are a fixed random 1% of each task; the remainder can be split for training/validation. Best-of-K selects the closest sampled trajectory using Euclidean distances averaged over time and 12 wrist/fingertip positions, then the test set. Main testing is within-task.
Go to primary source ↓e-modelsPDF p. 7, Section 5; pp. 19–20, Appendix A.4 and Figure 7
Baselines combine decoder-only or encoder-decoder Transformers with BC, DDPM or FM. Figure 7 maps image/language/proprioception through ResNet/CLIP/MLP and depicts encoder-to-decoder cross-attention.
Go to primary source ↓e-trainingPDF pp. 19–20, Appendix A.4
Training uses Adam at 1e-4, 50,000 steps, batch 2,048, eight A100 GPUs with 80GB each and 48 physical/96 logical CPUs; approximately 72 hours. Images are 224×224; ResNet is pretrained and CLIP frozen. Baselines use current observations only. DDPM/FM use 16 sampling steps; other hyperparameters defer to X-IL.
Go to primary source ↓e-comparisonPDF p. 8, Table 2, all six model rows and K=1/5/10 columns
At two seconds, EncDec+FM average distances are 0.051/0.041/0.038 and final distances 0.070/0.047/0.041; EncDec+BC has 0.044/0.060 at all K; Dec+BC has 0.045/0.062. No uncertainty accompanies the table.
Go to primary source ↓e-horizonPDF p. 8, Table 3; p. 15, Table 6
Dec+BC average/final errors are 0.031/0.049, 0.045/0.062 and 0.053/0.069 at H=30/60/90. EncDec+FM best-of-10 average errors are 0.026/0.038/0.047.
Go to primary source ↓e-goalPDF p. 8, Section 5 goal-conditioning paragraph; p. 9, Table 4
Dec+BC with goal image reports average/final 0.035/0.029 versus 0.045/0.062 without it; the authors state 22% and 53% relative reductions.
Go to primary source ↓e-scalingPDF p. 8, Figure 5 and Section 5; p. 9, model-capacity and data-scaling paragraphs
Dec+BC errors decrease over 5%, 10%, 25%, 50% and 100% training-data subsets on a log x-axis. The 500M-parameter variant matches the 200M baseline at average 0.045 and final 0.062. The plot has no uncertainty bands.
Go to primary source ↓e-oodPDF p. 15, Appendix A.1 and Table 5
Six tasks in a separate extra folder are called OOD. Average/final errors: Jigsaw Puzzle 0.047/0.065, Tetra Board 0.060/0.082, Knit Scarf 0.064/0.093, Play Reversi 0.065/0.096, Blowdry Hair 0.083/0.118, Stamp Paper 0.099/0.162; ID average 0.045/0.062.
Go to primary source ↓e-taskinventoryPDF pp. 15–18, Appendix A.2; specifically p. 17 reversible list and p. 18 reset-free/reset lists
The stated inventory totals 76 reversible pairs, 28 reset-free and 14 reset tasks. It includes assemble_disassemble_jigsaw_puzzle, stack_unstack_tetra_board, knit_scarf, play_reversi, blowdry_hair and stamp_paper, corresponding to the separately designated OOD tasks; training membership is not resolved by this list.
Go to primary source ↓e-jointsPDF pp. 18–19, Appendix A.3
Joint poses are SE(3), stored as 4×4 transformations. leftHand/rightHand mean wrists; wrist confidence describes hand detection and finger confidence is relative to the wrist.
Go to primary source ↓e-limitationsPDF p. 10, continuation of Section 7
The authors acknowledge limited scene/background diversity and imperfect model-predicted labels during heavy occlusion or rapid motion. More diverse collection and background randomization are future work.
Go to primary source ↓e-usecasesPDF p. 9, Section 6
Robotics applications must bridge the human–robot embodiment gap. Perception, video generation and egocentric world models are suggested research uses, rather than evaluated deployments in this paper.
Go to primary source ↓8.5 Primary sources
EgoDex: Learning Dexterous Manipulation from Large-Scale Egocentric Video ↗
PDF · 8,463 extracted words
Source fingerprint
7ea269e5885392e65a7e206c2d6b5792022ec499367a070c2c269cbcf187e2b8