Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations
1. Paper overview
In one sentence: AINA transfers human demonstrations to a dexterous robot through aligned 3D point trajectories, reducing dependence on robot training data while retaining scene-calibration and contact-control constraints. overviewcollectionperceptionalignmentarchitecturecontroldata-ablationheight-transferobject-transfer
| At a glance | What to know |
|---|---|
| Research problem | Source description Human video and robot control differ in viewpoint, embodiment and available motion labels. AINA asks whether rich sensing from portable Aria Gen 2 glasses can supply enough geometry to learn closed-loop dexterous policies using human demonstrations alone. The setup still requires calibrated robot cameras and a human demonstration in the deployment scene. overviewcollectioncontrol |
| Core mechanism | Source description The framework combines stereo-derived object tracks, on-board hand poses, an in-scene alignment anchor and point-based imitation learning for a multi-fingered robot. perceptionalignmentarchitecture |
| A key reported result | Oven Opening and Drawer Opening: RGB comparison: AINA: 12/15 and 11/15. Successful task completions / 15 trials. Table II: same human dataset, similar deployment scenarios, 15 trials per method and task. Masked BAKU: 6/15 and 1/15; Masked BAKU with History: 0/15 and 0/15. The point-based pipeline wins these two comparisons. Attributing the entire difference to geometry alone is not isolated because encoders, representation and alignment also differ. rgb-comparisonarchitecture |
| Reading caution | Source description Authors identify missing force information, RGB/SLAM shutter mismatch under rapid head motion, and a residual Aria-to-RealSense sensing gap. Collectors are instructed to avoid rapid head movement; FoundationStereo's real-time depth difficulty prevents direct Aria streaming at deployment. limitations |
Core contributions
- Source description
The framework combines stereo-derived object tracks, on-board hand poses, an in-scene alignment anchor and point-based imitation learning for a multi-fingered robot. perceptionalignmentarchitecture
- Author claim
The authors claim the first multi-fingered policy framework requiring no robot training data, simulation, online corrections or reinforcement learning. This is their priority claim; the supplied comparison is not an independent exhaustive verification. overview
- Source description
Real-robot evaluation separates task execution, data-source ablations, image-input baselines, changed workspace heights and novel-object transfer. rolloutsdata-ablationrgb-comparisonheight-transferobject-transfer
Figure 4. Shared point coordinates connect human sensing to robot joint commands. Original paper, p. 3 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the left panel as two processing branches that meet in 3D: the upper branch tracks object pixels, while the lower branch estimates stereo depth from rectified SLAM frames. Cyan hand points come from Aria's hand-pose output. On the right, red object histories and cyan fingertip histories enter the Vector Neuron MLP. The learned positional-encoding addition is placed under the cyan tokens, consistent with Section III-C1. Follow the transformer and output MLP down to the fingertip bracket feeding IK, then to robot execution. The missing alignment step is described separately in Section III-B3; this overview does not show every preprocessing operation. architectureperceptionalignmentcontrol
What it supports. The shared representation is the bridge between embodiments: object geometry conditions a learned fingertip trajectory, while IK handles the robot's joint configuration. Equations (4)–(5) establish supervised future-fingertip prediction. The crop therefore supports an action-policy interpretation, with a separate kinematic execution stage, rather than establishing learned prediction of the future visual world.
Where the evidence stops. The drawing repeats red object-like traces below the output MLP, but the text and loss define only predicted fingertips. No object-output objective is specified. Also, equivariant Vector Neuron layers do not establish equivariance of the complete transformer policy.
2. Motivation
2.1 The problem and the proposed response
Human video and robot control differ in viewpoint, embodiment and available motion labels. AINA asks whether rich sensing from portable Aria Gen 2 glasses can supply enough geometry to learn closed-loop dexterous policies using human demonstrations alone. The setup still requires calibrated robot cameras and a human demonstration in the deployment scene. overviewcollectioncontrol
2.2 What this reading follows
A human hand and a robot hand look different, but both can be described through fingertip positions relative to objects. AINA builds its transfer pipeline around that shared geometry. Smart glasses provide human hand poses and stereo imagery; a short human demonstration in the robot workspace anchors the coordinate system and contributes deployment-relevant training data. A transformer learns future fingertip motion, and inverse kinematics turns those predictions into robot joint commands. The figures below connect that mechanism to actual execution, separate the anchor's two roles, and show why success on a new height or object requires a carefully stated evaluation protocol. overviewcollectionperceptionalignmentarchitecturecontroldata-ablationheight-transferobject-transfer
3. Research context
We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.
| Catalog dimension | Recorded classification |
|---|---|
| Major category | Foundational work |
| Architecture | Not applicable |
| Prediction paradigm | Not applicable |
| Quadrant | Not applicable |
This table preserves the labels recorded at reading time. The current major category is Related resources. View the current classification.
3.1 Evidence-based assessment
Supports the recorded classification
The recorded foundational-work classification fits the human-data collection and action-policy contribution. AINA predicts fingertip actions from observed geometry; it does not define a learned future-world predictor jointly generating observations and actions. Its IK module converts desired kinematics to joints, rather than inferring actions from predicted world states. The world-action architecture, prediction-paradigm and quadrant fields therefore remain not applicable. overviewarchitecturecontrol
This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.
4. Problem formulation
4.1 Inputs and outputs
| Inputs | Outputs |
|---|---|
|
|
4.2 Equations and their role
5. Method in detail
5.1 Recover geometry before asking the policy to generalize
The glasses do not directly deliver the object depth consumed by the policy. Grounded-SAM first locates task objects through a language prompt, and CoTracker follows their image points. FoundationStereo estimates disparity from rectified front SLAM-camera images; focal length and camera separation turn disparity into depth. The tracked pixels can then be unprojected into 3D. Aria supplies human hand poses separately. In the robot scene, two RGB-D cameras supply observations, and HaMeR estimates from their views are triangulated for the human demonstration. These routes produce comparable geometric inputs, but their errors need not match. The authors explicitly identify shutter timing and sensor differences as remaining problems. Thus the representation removes much irrelevant appearance while still depending on accurate correspondence between pixels, depth and hand coordinates. collectionperceptionlimitations
5.2 Treat the single in-scene demonstration as two interventions
The anchor demonstration first supplies a target coordinate frame: object centroids determine translation, while initial fingertips supply a gravity-axis rotation through Kabsch alignment. The same demonstration then enters supervised policy training. These roles answer different questions: where should a human trajectory be placed, and what do observations and actions look like under deployment sensing? This distinction is a reader interpretation of the mechanism, supported by Table I's separate transformation-only and training-only conditions. Neither reaches the complete recipe's reported success. It also explains why anchor quality matters: at the highest workspace level, the authors describe an unusual demonstration that brought the toy too close to the bowl. One demonstration is therefore a strong constraint on transfer, rather than a neutral coordinate marker. alignmentdata-ablationheight-transfer
5.3 Follow the prediction all the way to physical execution
The learned policy sees histories of object points and fingertips, compresses each history with Vector Neuron MLPs, and processes the resulting tokens with a transformer. Learned fingertip positional encodings distinguish the hand tokens; an output MLP predicts their future trajectory. Training compares those predictions with demonstrated fingertips using mean squared error. Deployment adds two operations outside that prediction loss: forward kinematics reconstructs the observed robot fingertips, and full arm–hand inverse kinematics converts desired fingertips into joint targets. A separate distance-triggered closure rule compensates for absent force labels during grasping. Consequently, a geometrically plausible prediction does not by itself establish a successful manipulation. Figure 6 makes that distinction visible through orange markers for reaching and grasping without completing the task. architecturetrainingcontrolrollouts
5.4 Training and inference
During training
The policy uses observation horizon 10 and prediction horizon 30, supervised by fingertip mean squared error. Off-the-shelf vision models provide preprocessing; the paper does not describe jointly fine-tuning them. Training runs 2,000 epochs, reportedly about two hours per task, without identifying the training accelerator. perceptionarchitecturetraining
Apply the same sampled translation (−30 to 30 cm), scale (0.8–1.2) and gravity-axis rotation (−60° to 60°) to inputs and targets. Input fingertips additionally receive what the source calls Gaussian noise in the range −2 to 2 cm; its standard deviation and truncation convention are unspecified. training
During inference
Segment and track deployment objects in 3D; obtain robot fingertips by forward kinematics. Feed these histories to the policy and convert desired fingertips plus current joints through full arm–hand IK. The hardware is a seven-DOF Kinova Gen3 and six-DOF Psyonic Ability Hand, observed by two RealSense cameras. control
If predicted thumb-to-any-other-finger distance is below 5 cm, a heuristic moves fingers closer for grasping. This substitutes a closure rule for missing force information. The paper describes closed-loop operation but does not specify how many predicted steps execute before replanning. control
5.5 Implementation flow
- Acquire geometry
Aria supplies head and hand poses in a gravity-aligned world frame. In-scene hand keypoints are estimated with HaMeR from two camera views and triangulated. Grounded-SAM initializes object segmentation; CoTracker tracks 2D points. FoundationStereo estimates depth from rectified front SLAM-camera pairs, enabling 3D unprojection. collectionperception
- Align demonstrations
Use the initial object-centroid difference for translation and Kabsch alignment of initial fingertips to extract a gravity-axis rotation. Apply the transform to both object and fingertip trajectories. The text fixes N at 500 in the object-point matrix but calls N the number of objects; that wording is ambiguous. alignment
- Predict fingertips
Vector Neuron MLPs encode each point's temporal history with SO(3)-equivariant activations. Flattened features become transformer tokens; only fingertip tokens receive learned positional encodings. An MLP predicts future fingertips. Equations (4)–(5) specify fingertip prediction and supervision, despite red object-like traces also drawn beneath the output MLP in Figure 4. architecture
6. Experiments & results
AINA converts smart-glass human demonstrations into aligned 3D object and fingertip tracks, learns a future-fingertip policy, and executes it through arm–hand inverse kinematics. Fifty in-the-wild demonstrations and one human demonstration in the deployment scene support task-specific training without robot interaction data. Real-robot transfer is strongest on several simple tasks and substantially weaker on pouring, stowing and knob rotation.
6.1 Read the original evidence
Figure 6. Reaching an object and completing its manipulation are different outcomes. Original paper, p. 6 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read each row from the initial-position map at left to the illustrated rollout frames at right. The counts label task completion across trials, not the number of successful frames. Green ticks mean completion; orange circles mean the robot reached and grasped but did not finish; red crosses mean failure to reach or grasp. Dotted marks indicate object orientation when it differs from the displayed rollout. Compare the strong Toaster Press and Toy Picking rows with Cup Pouring and Stowing, where completing the interaction is harder. Appendix A defines the tasks: in particular, Wiping needs only a stable grasp and wiping some part of the board. rolloutstasks
What it supports. AINA completes Toaster Press and Toy Picking in 13/15 trials each, Oven Opening in 12/15, Drawer Opening in 11/15, and Wiping in 9/15. Completion drops to 5/15 for Planar Reorientation, 3/15 each for Cup Pouring and Stowing, and 1/15 for Oven Turning. These are real execution outcomes with substantial task dependence.
Where the evidence stops. The snapshots illustrate selected rollouts, not every trial. The figure supplies no uncertainty estimates or comparative baseline. Its Oven Turning label corresponds to the appendix's Knob Rotating task; the appendix specifies a 90° target.
Table II. The point-based pipeline outperforms two masked-image baselines on opening tasks. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Keep the two task columns separate. All three methods use the same human dataset, and each cell has fifteen trials. Masked BAKU receives an object-masked RGB frame plus fingertip history; its history variant adds multiple image frames. AINA instead consumes 3D object and fingertip histories. Figure 8 and Section IV-C explain the baseline's image and fingertip encoders. The table is especially useful for checking whether adding image history necessarily helps transfer: here, both history-baseline entries are zero. The authors associate that failure with moving human-head views during collection versus fixed cameras during deployment. rgb-comparisonarchitecture
What it supports. AINA achieves 12/15 Oven Opening completions and 11/15 Drawer Opening completions. Masked BAKU obtains 6/15 and 1/15, while its image-history version obtains 0/15 on both. The observed advantage applies to these two physical tasks and this common training dataset under the paper's deployment scenarios.
Where the evidence stops. The comparison changes representation, encoders and geometric alignment together. It does not isolate point coordinates as the sole cause of improvement. The explanation based on camera-motion distribution shift is the authors' interpretation, not a separately controlled causal experiment.
Table III. Height transfer reuses the wild demonstrations but adds a new in-scene anchor. Original paper, p. 8 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Use Figure 9 on the same source page to decode the rows: Heights 1, 2 and 3 correspond to 7, 14 and 17.5 cm above the desk. Each cell represents ten trials. Read down the Toy Picking column and then down Wiping; the trends differ. Section IV-D specifies an additional human demonstration at each height for alignment, while reusing the original in-the-wild demonstrations. The experiment therefore changes both workspace geometry and the anchor demonstration. That protocol matters when deciding whether these entries measure simple invariance of a fixed deployed policy or transfer supported by fresh human information. height-transfertasks
What it supports. Toy Picking succeeds in 5/10, 6/10 and 2/10 trials as height increases, while Wiping succeeds in 5/10, 5/10 and 8/10. The authors link the highest-level Toy Picking failures to an unusual anchor that brought the toy too close to the bowl, suggesting sensitivity to demonstration content as well as height.
Where the evidence stops. These are not unchanged-policy, zero-shot height tests. Height and anchor identity change together, and the small samples do not isolate their effects. Wiping's permissive success definition also differs from Toy Picking's grasp-and-drop requirement.
Figure 10. Keeping policy weights fixed does not guarantee transfer across object changes. Original paper, p. 9 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the quoted words as prompts for the perception front end, not newly learned instructions to the policy. The top strip shows Toaster Press and two Wiping replacements; the bottom strip shows two Toy Picking replacements paired with a bowl. Each photograph includes its own completion count. Section IV-E states that no new policy is trained for these tests, but Grounded-SAM receives updated task keywords. This distinguishes the experiment from the height study, which collects an additional in-scene demonstration. Preserve the prompt labels when associating images with numbers; they are more precise provenance than inferring an object's category solely from its appearance. object-transfertasks
What it supports. The fixed policies achieve 6/10 on the new toaster, 7/10 and 5/10 on the two wiping examples, and only 1/10 and 2/10 on the picking replacements. The result supports some object transfer while showing that a shared point representation does not remove all consequences of changing the manipulated object.
Where the evidence stops. Shape, weight and perception prompts vary across examples without independent controls. Their individual causal effects remain unresolved. Wiping still uses the appendix's partial-board criterion, so success rates across task families should not be treated as equivalent difficulty.
6.2 Results and evaluation conditions
| Task & protocol | Reported result | Comparison & interpretation |
|---|---|---|
| Nine-task robot execution Physical robot; spatially varied initial configurations, 15 trials per task in Figure 6; task definitions in Appendix A. | Toaster Press 13/15; Toy Picking 13/15; Oven Opening 12/15; Drawer Opening 11/15; Wiping 9/15; Planar Reorientation 5/15; Cup Pouring 3/15; Stowing 3/15; Oven Turning 1/15. Successful task completions / trials | No baseline in Figure 6. Appendix A calls the last task Knob Rotating and specifies 90° rotation. Execution spans nine skills but reliability is uneven. Wiping requires a stable grasp and wiping some portion of the board, not complete cleaning. rolloutstasks |
| Toaster Press and Toy Picking: data-source ablation Table I: similar deployment scenarios, minimum ten trials per method; exact denominators are not tabulated. | AINA: 86% / 86%, in task order. Reported success percentage | In-Scene Only 30% / 10%; In-The-Wild Only 0% / 0%; In-Scene Transform and In-The-Wild 0% / 10%; In-Scene Training and In-The-Wild 60% / 20%. The strongest recipe uses the anchor for both alignment and training. These are adapted baselines, not complete reproductions of the cited frameworks. Retain the table's 86% separately from Figure 6's 13/15. data-ablationrollouts |
| Oven Opening and Drawer Opening: RGB comparison Table II: same human dataset, similar deployment scenarios, 15 trials per method and task. | AINA: 12/15 and 11/15. Successful task completions / 15 trials | Masked BAKU: 6/15 and 1/15; Masked BAKU with History: 0/15 and 0/15. The point-based pipeline wins these two comparisons. Attributing the entire difference to geometry alone is not isolated because encoders, representation and alignment also differ. rgb-comparisonarchitecture |
| Toy Picking and Wiping: height transfer Table III: workspace raised 7, 14 and 17.5 cm; a new in-scene demonstration per height, reusing the original in-the-wild data. | Toy Picking: 5/10, 6/10, 2/10; Wiping: 5/10, 5/10, 8/10. Successful task completions / 10 trials | Heights 1, 2 and 3 respectively; no competing baseline. This is transfer with a new anchor, not unchanged-policy zero-shot height generalization. Authors attribute the hardest Toy Picking condition to an atypical anchor trajectory pushing the bowl. height-transfer |
| Novel-object transfer Existing policies deployed without new policy training; Grounded-SAM prompts changed for the new objects. | Toaster 6/10; wiping object prompted Sponge 7/10; Board Eraser 5/10; Package, Bowl 1/10; Toy, Bowl 2/10. Successful task completions / 10 trials | Five novel-object conditions across three tasks in Figure 10. Transfer varies with the replacement object. Shape and weight are not independently controlled, so their causal effects cannot be separated. object-transfer |
6.3 Ablations and diagnostic examples
Read component removals and qualitative examples within their stated evaluation conditions.
Table I. The in-scene demonstration contributes both geometric alignment and training supervision. Original paper, p. 7 ↗
Excerpt from the authors’ paper; cropped without altering the figure or table.
How to read it. Read the first two rows as single-source policies: one in-scene demonstration alone, or in-the-wild demonstrations alone. The third row adds the in-scene trajectory only as a transformation reference; it does not train on that trajectory. The fourth row adds it to training but uses the baseline's simpler geometric transfer. The final row combines both uses with the in-the-wild data. Compare within each task column before comparing tasks. This is an ablation of how human data are used, not of robot-data fine-tuning. The caption states similar deployment scenarios and a minimum of ten trials for every method. data-ablationalignmentrollouts
What it supports. AINA reports 86% success on both tasks, compared with 60% and 20% when the anchor is used for training but not the full alignment, and 0% and 10% when it supplies alignment but is excluded from training. The paired contrasts support complementary roles for coordinate transfer and deployment-scene supervision.
Where the evidence stops. Exact per-row trial counts, seed variation and in-scene sampling weights are absent. The cited baselines are modified implementations. Preserve the printed 86% values; the separate rollout figure reports 13/15 rather than an exact 86% fraction.
7. Analysis & limitations
7.1 What the evidence leaves open
Authors identify missing force information, RGB/SLAM shutter mismatch under rapid head motion, and a residual Aria-to-RealSense sensing gap. Collectors are instructed to avoid rapid head movement; FoundationStereo's real-time depth difficulty prevents direct Aria streaming at deployment. limitations
Small trial sets lack reported confidence intervals or seed variability. Height adaptation depends on a new demonstration, and object transfer changes segmentation prompts. These tests support conditional transfer, not universal robustness. data-ablationrgb-comparisonheight-transferobject-transfer
7.2 Questions for discussion
- How much does performance depend on which single in-scene demonstration is chosen?
- Does resolving rotation–translation composition improve transfer independently of co-training?
- Which failures arise from predicted motion, IK feasibility or the grasp-closure heuristic?
8. Reproducibility audit
8.1 Requirements and known gaps
Required components include Aria recordings, synchronized camera/pose processing, hand–eye calibration, two RGB-D views, the segmentation/tracking/depth stack and full arm–hand IK. Appendix A provides task prompts and placement areas, including 30 × 30 cm Toy Picking and Wiping regions. collectionperceptioncontroltasks
The supplied paper leaves optimizer, learning rate, batch size, transformer size, in-scene sampling weight, train/validation split, software versions, accelerator model, IK objective and execution cadence unspecified. Resolve alignment composition and fingertip-noise conventions before attributing reproduction differences to learning. alignmentarchitecturetrainingcontrol
Proposed checks: test whether the printed transform actually aligns centroids under nonzero yaw; then repeat the alignment-versus-training anchor ablation with fixed evaluation placements, multiple anchors and seeds. Neither check was performed here. alignmentdata-ablationheight-transfer
8.2 Proposed reproduction checks
The following checks are proposals motivated by the paper. They have not been run as part of this reading.
Check 1: Check the printed alignment transform against its stated invariant
Reader-proposed offline check: select supplied-style trajectory pairs with different nonzero initial yaw and object centroids. Apply Equations (1)–(2) literally, using the pre-rotation centroid offset described in Section III-B3. Compare with a control that recomputes translation after rotating the initial object centroid. Measure initial centroid residuals and verify that both transformations preserve fingertip-to-object distances within each trajectory. If the printed expression leaves systematic centroid offsets while the control removes them, the claimed alignment needs an unstated pivot or translation convention. Resolve that ambiguity before robot trials; a geometric correction alone would not demonstrate better task success. alignmentperception
Check 2: Repeat the anchor's alignment and training roles with controlled sampling
Reader-proposed robot check: on Toaster Press and Toy Picking, hold the in-the-wild data, architecture, augmentation budget, training updates and evaluation placements fixed. Cross whether the in-scene demonstration supplies alignment with whether it enters policy training, matching the four relevant Table I conditions. Repeat with several independently collected anchors and random seeds; record the otherwise unspecified in-scene sampling weight. Report completions, grasp-only failures and uncertainty across matched trials. A consistent benefit from both roles would support the proposed mechanism. If performance instead changes primarily with anchor choice, the result would narrow the claim to demonstration-sensitive transfer. collectionarchitecturetrainingdata-ablationrolloutsheight-transfertasks
8.3 Reading coverage
Visual audit: All ten PDF pages were rendered and visually inspected, alongside complete individual reading of the five supplied text chunks. The visual pass covered the title and author block; Figures 1 and 3–10; Tables I–III; the stereo, alignment, policy-loss and IK formulations; training and deployment details; all evaluation protocols; limitations; and Appendix A task definitions. Six final original crops were separately viewed with all labels and legends retained. Figure 4's red traces below its output MLP are not treated as evidence of an object-prediction objective: Equations (4)–(5) supervise fingertips only. Separate supplements, project videos and the externally linked alignment illustration were outside the supplied material.
PDF pages inspected for this edition: 1, 2, 3, 4, 5, 6, 7, 8, 9, 10. Appendix coverage: reviewed.
Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.
Text reading scope & known omissions
- Title, author block, abstract and Figure 1 (p. 1)
- I. Introduction (pp. 1–2)
- II. Related Works (pp. 2–3)
- III. Method, A–C, including collection, geometry, alignment, training and deployment (pp. 3–6)
- IV. Experimental Evaluation, A–E (pp. 6–9)
- V. Discussion, Limitations, Conclusion (p. 9)
- Acknowledgements and references (pp. 9–10)
- Appendix A. Task Descriptions, a–i (p. 10)
Outside the original text pass
- Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
- Separate supplemental material availability has not been fully verified.
- The extraction limitation was addressed by visually inspecting all ten PDF pages and all six final crops. All five supplied text chunks were individually read.
- Project-site videos and the externally referenced alignment illustration were not supplied or inspected. Code was not inspected and experiments were not reproduced.
- Edition verified on p. 1: arXiv:2511.16661v1 [cs.RO], 20 November 2025. The observed title and all 13 authors match the catalog; no revision discrepancy was found. No other edition was supplied or compared.
The visual audit above records the subsequent illustrated pass.
8.4 Traceable evidence
identityPDF p. 1, title/author block and arXiv margin stamp
Title: Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations. Thirteen named authors match the supplied catalog, in order. Affiliations are New York University and Meta. The stamp identifies arXiv:2511.16661v1 [cs.RO], 20 Nov 2025.
Go to primary source ↓overviewPDF p. 1, abstract and Figure 1; p. 2, Introduction and contribution list
AINA learns point-based multi-fingered policies from in-the-wild human demonstrations and a single in-scene human demonstration, claiming no robot training data, simulation, online corrections or reinforcement learning. Figure 1 reports about 15 minutes average video-collection effort.
Go to primary source ↓collectionPDF p. 4, Section III-A(a) and III-B1
Hand–eye calibration takes approximately 5–10 minutes. Per task, 50 Aria demonstrations are recorded at 10 Hz, described as about ten minutes total; the in-scene demonstration takes under one minute. Aria poses use a gravity-aligned world frame. In-scene hand estimates from two RGB-D camera views use HaMeR and triangulation.
Go to primary source ↓perceptionPDF p. 4, Section III-B2 and unnumbered stereo-depth equation
Grounded-SAM initializes object segmentation; CoTracker supplies 2D tracks. FoundationStereo uses rectified front SLAM-camera streams and baseline B to estimate disparity d. Left-camera depth is Z=fB/d. Tracks are unprojected into 3D and transformed to world or robot frames.
Go to primary source ↓alignmentPDF p. 4, Section III-B3; p. 5, Section III-B3 continuation, Equations (1)–(3)
The text defines object-point and five-fingertip trajectories, fixing N=500 while calling N the number of objects. Translation is the first-frame object-centroid difference. Kabsch on initial fingertips provides a gravity-axis rotation. Printed equations apply R_z to original points and then add the earlier offset; rotation pivot and translation recomputation are not clarified.
Go to primary source ↓architecturePDF p. 3, Figure 4 and caption; p. 5, Section III-C1, Equations (4)–(5)
Vector Neuron MLPs encode histories; a transformer and output MLP predict fingertips, using observation horizon 10 and prediction horizon 30. Learned positional encoding applies only to fingertip tokens. MSE supervises fingertips. Figure 4 also draws red object-like traces below the output MLP, but only the cyan fingertips feed its IK bracket; the equations specify no object-prediction target.
Go to primary source ↓trainingPDF p. 5, Section III-C1, augmentation and training paragraphs
Training uses shared input/target translation ±30 cm, scaling 0.8–1.2 and gravity-axis rotation ±60°. Additional input-fingertip noise is described as Gaussian within ±2 cm. The policy trains 2,000 epochs, typically two hours per task. Optimizer, batch size, accelerator and detailed architecture configuration are not supplied here.
Go to primary source ↓controlPDF p. 5, Figure 5 and Section III-C2(a–c); p. 6, opening continuation of III-C2(c)
Deployment uses Kinova Gen3 with seven DOFs, Psyonic Ability Hand with six DOFs, and two RealSense RGB-D cameras. Forward kinematics provides observed fingertips; full arm–hand IK maps desired fingertips and 13 current joint angles to next joints. Predicted thumb-to-other-finger distance below 5 cm triggers closer finger closure.
Go to primary source ↓tasksPDF p. 10, Appendix A, task descriptions (a)–(i)
Appendix A defines nine tasks, placement areas and segmentation prompts. Toy Picking and Wiping use 30 × 30 cm areas. Wiping success means stable sponge grasp and wiping some portion of the board. Stowing combines bowl placement inside an oven with door closure. Knob Rotating targets 90°.
Go to primary source ↓rolloutsPDF p. 6, Figure 6, nine labeled task rows, counts, legend and caption
Counts are 13/15 Toaster Press, 13/15 Toy Picking, 12/15 Oven Opening, 11/15 Drawer Opening, 9/15 Wiping, 5/15 Planar Reorientation, 3/15 Cup Pouring, 3/15 Stowing and 1/15 Oven Turning. Green ticks denote completion; orange circles denote reaching/grasping without completion; red crosses denote failure to reach/grasp.
Go to primary source ↓data-ablationPDF p. 7, Section IV-B, baseline definitions, Table I and accompanying interpretation
For Toaster Press/Toy Picking, success is 30%/10% in-scene only, 0%/0% in-the-wild only, 0%/10% using in-scene transformation only, 60%/20% using in-scene training only, and 86%/86% for AINA. Caption specifies similar deployment scenarios and minimum ten trials. Baselines are adaptations, with explicit differences from cited methods.
Go to primary source ↓rgb-comparisonPDF p. 8, Section IV-C, Figure 8 and Table II
The same human dataset trains AINA and masked-image BAKU variants. For Oven Opening/Drawer Opening, Masked BAKU obtains 6/15 and 1/15; its image-history variant 0/15 and 0/15; AINA 12/15 and 11/15. BAKU uses ImageNet-pretrained ResNet-18 image encoders, MLP fingertip encoders and a zero action token. The authors attribute history failure to moving-head versus fixed-camera distribution shift.
Go to primary source ↓height-transferPDF p. 8, Section IV-D, Figure 9 and Table III
Heights 1–3 are 7, 14 and 17.5 cm above the desk. Each uses an additional in-scene demonstration and the original in-the-wild data. Toy Picking obtains 5/10, 6/10, 2/10; Wiping 5/10, 5/10, 8/10. The authors attribute poor highest-level Toy Picking to an unusual anchor trajectory that brings the toy too close to the bowl.
Go to primary source ↓object-transferPDF p. 8, Section IV-E; p. 9, Figure 10, caption and continuation of IV-E
Existing policies are deployed on new objects without retraining, with changed Grounded-SAM prompts. Figure 10 pairs Toaster with 6/10, Sponge with 7/10, Board Eraser with 5/10, Package, Bowl with 1/10 and Toy, Bowl with 2/10. Authors discuss shape and weight mismatch as limitations.
Go to primary source ↓limitationsPDF p. 9, Section V, Discussion, Limitations, Conclusion
Authors identify unavailable force feedback, RGB/SLAM shutter mismatch during rapid head motion and remaining Aria-to-RealSense keypoint differences. They ask collectors to avoid rapid head movement and cite difficulty computing FoundationStereo depth in real time as preventing direct Aria deployment streaming.
Go to primary source ↓8.5 Primary sources
Dexterity from Smart Lenses: Multi-Fingered Robot Manipulation with In-the-Wild Human Demonstrations ↗
PDF · 7,660 extracted words
Source fingerprint
8999196e91d2f29c9071026b6befb22651108219629f19d420e209932797ecfd