FMB: A functional manipulation benchmark for generalizable robotic learning

The observed title is FMB: a Functional Manipulation Benchmark for Generalizable Robotic Learning. Authors are Jianlan Luo, Charles Xu, Fangchen Liu, Liam Tan, Zipeng Lin, Jeffrey Wu, Pieter Abbeel and Sergey Levine. Luo and Xu contributed equally. The title block says July 2024; the stamp identifies arXiv:2401.08553v3 [cs.RO], 3 September 2024. The affiliation is printed as Department of Electric Engineering and Computer Sciences, University of California, Berkeley, USA.

FMB combines object generalization with contact-rich functional grasping, reorientation and insertion in single-object and interlocking multi-object assemblies. It is presented as a modular physical benchmark and imitation-learning toolkit.

The object set comprises 66 objects: 54 for single-object tasks, spanning nine shapes and six sizes, with one of eight colors assigned to each; the remaining objects form three multi-object assemblies. Five additional test objects vary shape, size and color. Mating tolerances are 1–2 mm.

Grasps must enable subsequent manipulation. A fixture supports placing and regrasping, rotation adjusts orientation, and insertion must align the part with its slot. Figure 4 arrows run from initial to resulting states within each skill. Repositioning data collection is partly scripted. Inter-stage pose shifts and assembly order complicate complete tasks.

The introduction says 22,500 demonstrations, while Section 3.5 and Table 1 give 22,550 primitive trajectories: 15,350 single-object and 7,200 multi-object. Single-object data include 2,700 complete trajectories plus 4,050 extra insertion demonstrations. There are 150 complete demonstrations per multi-object assembly. Primitive segments average about five seconds; complete single-object demonstrations last 20–40 seconds and multi-object demonstrations can exceed 100 seconds.

The setup uses a Franka Panda, a SpaceMouse issuing six-dimensional end-effector twist at 10 Hz, and a low-level impedance controller at 1 kHz. Four RealSense D405 RGB/depth cameras include two wrist and two side views. White curtains limit background distractions. Force/torque is estimated by the robot model, not directly measured with an added force/torque sensor.

Single-object grasp positions vary within 20 × 20 cm; insertion boards vary within 35 × 35 cm and up to 15 degrees rotation. Multi-object components vary within 20 × 30 cm while the board is fixed. Evaluation specifies initial poses and five multi-object arrangements; novel objects and held-out-object training are described as generalization protocols. Detailed procedures are referred to the project website.

The ResNet policy encodes camera inputs with weight-shared ResNet-34 backbones, combines visual features with projected proprioception and optional one-hot object/skill IDs, and uses an MLP Gaussian policy head. Outputs are six-dimensional end-effector twist and a binary gripper command. Modalities vary by experiment.

The decoder-only Transformer uses ResNet-34 image tokens, MLP proprioceptive tokens, sinusoidal position embeddings and self-attention with four heads and four MLP layers. The text describes FiLM conditioning on object or primitive IDs. The figure explicitly draws the object-ID-to-FiLM path, but only a token path for primitive ID. Each continuous action dimension is Gaussian-quantized into 256 bins during training and decoded to continuous commands at runtime.

Hierarchical execution uses a human oracle to choose skills, optionally object IDs, and retry failed or unsuitable outcomes. The low level is one conditioned policy or five separate skill policies. Execution stops at an unrecoverable state or a preset trial-step maximum; its numerical cap is not stated. Learning the high-level selector is left to future work.

Grasping evaluates five seen and five unseen objects, five trials each. With all training data, ResNet RGB obtains 27/50, ResNet RGBD 31/50 and Transformer RGB 28/50. ResNet RGB totals at 20%, 50%, 80% and 100% data are 12, 12, 18 and 27; RGBD totals are 15, 16, 26 and 31. At full data, seen/unseen counts are 13/14 for ResNet RGB, 17/14 for ResNet RGBD and 19/9 for Transformer RGB, each subgroup out of 25.

Across five seen and five unseen objects, each repositioning policy receives 50 trials per skill. Rotate/place-on-fixture/regrasp successes are 49/38/42 for ResNet without action chunking, 33/44/44 with chunk length 3, and 47/46/49 for Transformer without chunking. Failure modes include mistimed gripper commands and wrong rotation direction.

The rectangular insertion modality study tests ResNet policies over five object sizes and 25 trials. Three RGB views plus pose/velocity obtain 2/25; RGBD plus pose/velocity also 2/25; adding force/torque to RGB gives 11/25, and adding force/torque to RGBD gives 5/25. Views are two wrist and one side. The explanation involving noisy depth and precise end-effector pose is proposed in the text, not independently isolated.

Shape-specific ResNet insertion policies receive 25 trials across five sizes. Rectangle, hexagon, circle-square, three-prong and star successes are 11, 25, 5, 0 and 2 respectively. The text describes incorrect slot localization, getting stuck near holes and orientation-dependent action ambiguity.

Policies trained on all insertion data are tested five times per each of nine object shapes. Unconditioned ResNet obtains 7/45, shape-ID-conditioned ResNet 14/45, and shape-ID-conditioned Transformer 27/45. The attention/FiLM explanation and qualitative absence of wrong-hole reaches for the Transformer are author interpretations and observations, not an isolated attention ablation.

Single-object multi-stage evaluation has ten trials each on hexagon, circle-square and three-prong. Unconditioned Diffusion ResNet, ResNet and Transformer all total 0/30. Oracle-selected five-policy ResNet gives 9/8/1, total 18/30; one conditioned ResNet gives 0/30. Oracle-selected five-policy Transformer gives 9/8/2, total 19/30; one conditioned Transformer gives 7/6/2, total 15/30.

Table 5 reports ten trials on Assembly Board One, the red board. Human-oracle hierarchical ResNet achieves 5/10 and Transformer 7/10; unconditioned ResNet, Transformer and Diffusion ResNet each achieve 0/10. Hierarchical policies receive primitive and object IDs. The prose mentions ten trials per initial configuration, but the table reports a ten-trial board-level denominator without a per-configuration breakdown.

Five separate ResNet policies receive task-tailored sensor sets, whereas conditioned policies receive all available sensory inputs. The authors attribute ResNet degradation partly to irrelevant inputs, including a fourth view under board randomization, and say Transformers can ignore such inputs. The comparison also changes low-level parameter sharing and conditioning.

The PDF points to external setup and evaluation instructions and describes policy architectures without an explicit optimization equation, optimizer, learning-rate schedule, batch size, encoder initialization/freezing declaration or complete training-compute configuration. The diffusion baseline schedule is not given. Acknowledgments name computing support but provide no accelerator count or model. Section 5.4 does not give the oracle step cap numerically.

The authors emphasize task complexity, object generalization, reproducible components and prospective expansion through new printed objects and procedurally structured assembly boards. This discussion does not demonstrate cross-lab replication or broad generalization outside the task family.