Awesome-WAMs
GitHub
PAPER REPORTENAll readings ↗

Flex-ππ: A Multi-Stream World-Action Model with Compute Flexibility

English reading report: Method, equations, original figures, experiments and reproducibility.

Authors: Ge Yan, Jinghao Liu, Yuzhi Fan, Lei Cai, Minwen Liao, Jesse Zhang, Dieter Fox

Affiliations: University of Washington; Allen Institute for AI

Source: 2608.10860 ↗ · Project page ↗ · Catalog record

Reading: 8 / 558 · 5 original figures & tables · ~23 min ·

1. Paper overview

In one sentence: FLEX-π trains one action model to read optional RGB, semantic and geometric futures, letting inference spend more computation on imagined context while preserving an action-only route. E02E03E04E05E14E15E19E22

At a glanceWhat to know
Research problem
Author claim

The authors argue that RGB reconstruction targets underemphasize manipulation-relevant geometry and object semantics. They seek richer predictive supervision while reusing existing visual encoders and allowing deployment to omit expensive streams. E02

Core mechanism
Source description

The model reuses a frozen video VAE for both RGB and image-shaped pointmaps, while separately projecting folded DINO features into the visual transformer. The claimed accurate pointmap reconstruction motivates this reuse; it does not establish accurate physical dynamics. E03

A key reported resultEffect of inference-time future generation: Reported action-only: 40.2% at approximately 60 ms. Adding RGB futures: 60.4%. Generating RGB, DINO, and pointmap futures: 63.8% at approximately 193 ms.

Average success and approximate latency per policy call. Five domain-randomized RoboTwin tasks, 50 demonstrations per task, five epochs from scratch and RGB-only conditioning; output streams varied on one checkpoint without further training.

Full visual generation exceeds action-only by 23.6 percentage points. Analysis: Holding the checkpoint fixed supports a direct deployment benefit from future generation in this setting. These five-task results must not be conflated with the full-data RoboTwin averages. E19

Reading caution
Author claim

The authors report slower convergence from additional modalities and cross-modality forcing, requiring at least ten fine-tuning epochs on their real-world tasks. Full joint generation remains slower than parameter-comparable VLAs. E24

Core contributions

  • Source description

    The model reuses a frozen video VAE for both RGB and image-shaped pointmaps, while separately projecting folded DINO features into the visual transformer. The claimed accurate pointmap reconstruction motivates this reuse; it does not establish accurate physical dynamics. E03

  • Source description

    A shared visual transformer and a separately parameterized action expert jointly denoise visual futures and action chunks, with asymmetric attention allowing actions to read visual predictions. E04

  • Source description

    Independent input-presence and output-attention masks enable one flexible checkpoint to support different deployment configurations. Crucially, output masking changes attention during training, not which prediction losses are computed. E05E06

Figure 2. Three visual representations supply context to a separate action expert. Original paper, p. 4 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Follow the left-hand rows by representation. RGB and pointmaps enter Wan VAE encoders with shared weights, while DINOv3 extracts a different feature stream from RGB. Learned adapters bring those representations into the visual transformer. The vertical input mask marks which current streams are available; the horizontal output mask selects which future streams actions may use. The conditioning box supplies proprioceptive state and language. On the right, three future representations align with a continuous action chunk. The arrow from the visual trunk into the action expert reflects asymmetric attention: actions can read visual information, but visual streams do not read action tokens. E03E04E05E07E08E26

What it supports. The model does not simply concatenate every sensor into a conventional action head. It trains distinct future representations around a separate action expert, with explicit control over their visibility. This creates a route for richer imagined context to influence actions and a cheaper route that generates only the action stream.

Where the evidence stops. Current pointmaps have different provenance across experiments: pre-training uses estimated depth, while the main real-robot setup uses sensed stereo depth. An action-only output setting also does not imply RGB-only observations or removal of the current visual encoders.

2. Motivation

2.1 The problem and the proposed response

Author claim

The authors argue that RGB reconstruction targets underemphasize manipulation-relevant geometry and object semantics. They seek richer predictive supervision while reusing existing visual encoders and allowing deployment to omit expensive streams. E02

2.2 What this reading follows

Predicting several kinds of future is expensive, and a controller may not need all of them on every call. FLEX-π turns that tension into an architectural choice: one trained model can generate actions alone or let the action stream read selected visual futures. RGB carries appearance, DINO features provide semantic information, and pointmaps describe geometry. The key is to distinguish missing current observations, attention among future streams, and computation actually removed at inference. The diagrams below make those distinctions explicit. RoboTwin results show what the flexible checkpoint achieves at scale; the compute ablation and real gripper-repair experiment help locate when extra future prediction is useful and what it costs. E02E03E04E05E14E15E19E22

3. Research context

We place the paper in the collection through its world–action interface. The catalog labels and the reading’s assessment are shown separately.

Catalog dimensionRecorded classification
Major categoryWAMs
ArchitectureDual-system
Prediction paradigmJoint prediction
QuadrantQ3 · Dual-system × Joint prediction

3.1 Evidence-based assessment

Supports the recorded classification

Reader analysis

The recorded Dual-system × Joint prediction classification is supported by architectural separation: the visual trunk and action expert have distinct parameters and interact through selected attention layers. Joint prediction follows from simultaneous visual/action denoising, rather than joint training alone or a separate inverse-dynamics stage. Action-only deployment does not erase this training architecture. The 3D multiview label should be understood as multiple camera-frame pointmaps in a composite, not demonstrated reconstruction of an extrinsically registered global scene. E04E05E06E07E08

This is the collection’s architectural analysis, not a new related-work survey. Benchmark comparisons and their protocols appear in Section 6.

4. Problem formulation

4.1 Inputs and outputs

InputsOutputs
  • A current RGB observation, potentially arranged as a three-camera composite.
  • Optional current camera-frame pointmaps and DINO features.
  • Current proprioception in a canonical 32-dimensional layout.
  • A language instruction.
  • A continuous chunk of 32 actions in the canonical action layout.
  • Optional future RGB latents, pointmap latents, and folded DINO features spanning the action horizon.

4.2 Equations and their role

zτ=τz+(1−τ)ϵ,ϵ∼N(0,I),LFM(z)=E ⁣[∥vθ(zτ∣τ,l,z≤t)−(z−ϵ)∥22].\begin{aligned}z^\tau&=\tau z+(1-\tau)\epsilon,\qquad\epsilon\sim\mathcal N(0,I),\\\mathcal L_{\mathrm{FM}}(z)&=\mathbb E\!\left[\lVert v_\theta(z^\tau\mid\tau,l,z_{\le t})-(z-\epsilon)\rVert_2^2\right].\end{aligned}
Equation (1) trains velocity along a straight noise-to-data path. Sampling integrates the learned velocity from noise toward the target with Euler steps. E06
L(θ)=λaLaFM(at)+∑i∈{o,d,p}λiLiFM(it+1),λa=λo=λd=λp=1.\begin{aligned}\mathcal L(\theta)&=\lambda_a\mathcal L_a^{\mathrm{FM}}(a_t)+\sum_{i\in\{o,d,p\}}\lambda_i\mathcal L_i^{\mathrm{FM}}(i_{t+1}),\\\lambda_a&=\lambda_o=\lambda_d=\lambda_p=1.\end{aligned}
Equation (3) includes action, RGB, DINO, and pointmap targets on every training sample. The output-attention mask does not gate this sum. E06
v^=d^t+1−dt+1τ1−τ.\hat v=\frac{\hat d_{t+1}-d^\tau_{t+1}}{1-\tau}.
Equation (4) converts the DINO head's clean-feature prediction into velocity, preserving the flow-matching loss and integration procedure. The authors use this parameterization because folding makes DINO targets high-dimensional. E06

5. Method in detail

5.1 Align appearance, semantics and geometry before fusion

Reader analysis

FLEX-π begins with three representations that carry different information about the same scene. RGB and pointmaps share a frozen Wan VAE, while frozen DINOv3 produces semantic patch features. A two-by-two spatial fold combines four DINO tokens into a wider token without discarding entries; learned projection then matches the transformer width. The visual trunk shares parameters across streams, but the action expert has its own attention, feedforward and normalization weights. Fusion occurs in the middle sixteen of thirty blocks. The important directional constraint is that actions read current observations and selected futures, while visual tokens never read actions. My interpretation is that this preserves a reusable visual prediction pathway while letting control benefit from it, although the ablations do not independently isolate every representation and architectural choice. E03E04E05E26

Figure 4. The output mask changes information flow during training and work during inference. Original paper, p. 6 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Read each row from the two masks toward the final column. The input mask controls present observations; the output mask controls which future group the action tokens read. Solid links show mutually visible future streams, arrows point into actions, and magenta crosses block attention. In the action-only row, future visual streams are unread by the action expert and omitted at inference. During training, however, all future targets still receive their losses. The highlighted last row illustrates cross-modality forcing: the current pointmap is absent, but its predicted future remains available to actions. The model must infer that missing modality through the remaining observations. E05E06E27

What it supports. The training objective teaches one checkpoint several information patterns rather than training a separate model for each output mode. Cross-modality forcing also distinguishes reconstructing an absent input modality from merely tolerating its absence. The potential computation saving appears at inference, when unneeded visual futures are no longer denoised.

Where the evidence stops. Dashed future nodes do not mean their training loss is removed. Confusing attention masking with loss masking would implement a different objective and could invalidate the reported ability to switch generation modes without retraining.

5.2 Train every target while varying what actions are allowed to read

Source description

Two sampled masks serve different purposes. The input mask removes current visual streams, with sampling constrained to retain at least one. The output mask partitions future attention and determines which visual futures the action expert may read. It does not switch off their training losses: RGB, DINO, pointmap and action targets are all supervised on each sample with equal loss weights. This is why a future pointmap can be useful even when no current pointmap was supplied. At inference, the unread future streams can finally be omitted from computation, turning the learned attention pattern into a practical mode switch. Actions still condition on available current observations. Calling this action-only generation describes the outputs computed, not the full set of sensors, encoders or conditioning features required by the controller. E05E06E22E27

5.3 Relate the mode switch to the actual control cycle

Reader analysis

A lower model latency matters only within the controller that consumes its action chunk. The main real-robot setup executes thirty-two commands at 30 Hz, about 1.07 seconds, then holds position while synchronously computing the next chunk. Joint prediction can improve action quality while increasing that pause. The optimized measurements use caching and compilation as well as the architecture; the paper reports approximately 60 ms for action-only and 193 ms for full joint under its default four-step setting. My reading is that a deployment decision should compare success, observation age and total control-cycle timing together. The main RoboTwin averages suggest little aggregate benefit from joint prediction at high data scale, whereas restricted-input and gripper-repair diagnostics show larger gains. Those experiments motivate a task-dependent choice, not a universal preference for either mode. E08E09E14E15E19E22E23

5.4 Training and inference

During training

Source description

Robot pre-training uses approximately 500 hours from 100 AGIBOT World-Beta tasks, selecting up to the first 285 episodes per task and splitting episodes at instruction boundaries. Domain-specific fine-tuning follows. Reusing visual priors therefore does not eliminate robot pre-training. E07

Source description

The VAE, umT5, and DINOv3 remain frozen; the visual trunk, action expert, adapters, and prediction heads are trained. The narrower action expert is initialized by interpolating Wan weights. Appendix A.3 explicitly identifies randomly initialized DINO projections as well as action input/output layers. E03E04E11

Source description

All four stream losses have weight one. Shared settings include AdamW, learning rate 10⁻⁴, cosine scheduling with 5% warmup, bfloat16 mixed precision, and ZeRO stage 1. Reported flow-matching shifts are six for visual streams and one for actions. E06E11

During inference

Source description

The chosen output mask now removes unneeded visual futures from computation. Four Euler steps are the default; action-only generation denoises action tokens against cached observation features. The main optimized action-only latency measurement uses full current visual input. E05E22E23

Source description

On the real robot, all 32 actions execute open-loop at 30 Hz, covering approximately 1.07 seconds. The synchronous controller then holds position while computing the next chunk. LIBERO executes ten predicted actions before obtaining feedback and replanning. Generated visual futures are internal policy features, not commands executed by the robot. E08E09E10

5.5 Implementation flow

  1. Construct complementary observations

    AGIBOT pointmaps are produced offline using Depth Anything 3 and camera intrinsics. Main real-robot experiments instead use sensed stereo depth, while LIBERO uses rendered depth. The views are tiled into a composite; pointmaps remain camera-frame representations and require no extrinsic registration. E07E08E10

  2. Encode visual and task information

    RGB and pointmaps share the frozen Wan-2.2 VAE. DINOv3 patch features undergo an invertible 2×2 spatial fold, converting four 768-dimensional tokens into one 3072-dimensional token, followed by normalization and learned projection. DINO therefore shares transformer width, not the raw VAE representation. A linear proprioception encoder adds one token to the frozen umT5 language sequence. E03E09

  3. Exchange information between visual and action experts

    The approximately 5B visual trunk shares weights across visual streams, with modality-specific adapters. The approximately 1B action expert has its own attention, feedforward, and normalization parameters. Cross-stream attention occurs in the middle 16 of 30 blocks. Actions read observations and selected visual futures; visual tokens never read actions. E04E05

  4. Train with missing conditioning and selective future visibility

    Input and output mask entries are independently sampled with probability 0.5, rejecting input masks that remove every visual modality. Missing current-stream tokens are zeroed. Future streams communicate within equal-output-mask groups, and actions read only the selected group. Nevertheless, every future stream is predicted and supervised, including modalities absent from the input. E05E06

  5. Predict continuous actions over an aligned horizon

    Each action timestep is one continuous token. A 33-timestep sample supplies 32 actions and nine visual frames sampled at stride four; VAE compression yields one current and two future latent frames. Real-robot action poses and joint displacements are anchored to the state at the start of the chunk, while gripper commands remain absolute. E09

6. Experiments & results

FLEX-π trains a 6B manipulation policy to predict future appearance, geometry, semantic features, and actions. Its distinctive mechanism separates the supervision available during training from the visual futures computed during deployment: every target stream trains on every sample, while attention masks teach the action expert to operate with different available streams. Action-only deployment can consequently retain benefits from geometric and semantic training without generating those futures. Joint generation remains useful on difficult manipulation tasks, but costs additional computation. The fastest reported configuration still consumes current multimodal observations; action-only does not mean RGB-only.

6.1 Read the original evidence

Table 1. One flexible checkpoint reaches the same reported average in both output modes. Original paper, p. 9 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. The Clean and Rand. columns correspond to the two RoboTwin evaluation settings; Avg. summarizes them. Read the two green FLEX-π rows together despite the visual split into VLA and WAM blocks. They are inference modes of the same flexible checkpoint, not separately trained category-specific models. Action-only reports 94.5 and 94.6; full joint reports 94.3 and 94.8. Both average to 94.6 at the table’s precision. The baseline rows supply task context, but the most controlled contrast in this table is the mode switch within FLEX-π. The experiment covers 50 tasks and uses the larger demonstration setting described in the protocol. E15

What it supports. At this data scale, joint future generation has nearly the same aggregate success as action-only inference: the table reports 94.6% average for both. The result supports retaining a low-compute mode without an obvious aggregate penalty in this setting, while leaving room for task-specific differences hidden by the mean.

Where the evidence stops. An equal rounded average is not a statistical equivalence result. The table does not quantify uncertainty for the within-checkpoint mode difference, and aggregate results can conceal tasks where additional future streams help or hurt.

Figure 19. Partial task credit and full completion tell different stories in precision assembly. Original paper, p. 33 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. Start with the rubric panel: each stage earns a different amount of the three available points, so completing an early easy step is not equivalent to completing an insertion or fastening step. The green stages identify the precision operations. Then read the performance panel in its two groups. Normalized score measures accumulated partial credit; full-task success requires completion of the entire sequence. The two FLEX-π modes improve both measures, but a 76.0 normalized score does not mean 76% of trials finish. For full completion, action-only reaches 45% and joint generation reaches 55%. The appendix evaluates each method over 20 rollouts. E13E14E29

What it supports. Joint generation achieves a 76.0 normalized score and 55% full-task success, compared with 66.9 and 45% for action-only. These are distinct outcomes: the latter difference corresponds to two additional complete trials out of twenty. The result provides task-level evidence that extra predicted context can help beyond an already strong action-only policy.

Where the evidence stops. Twenty rollouts give a limited estimate of the mode difference, and the figure shows no uncertainty intervals. The training uses demonstrations plus corrective data; comparisons should retain that data protocol rather than interpreting the result as zero-shot precision manipulation.

6.2 Results and evaluation conditions

Task & protocolReported resultComparison & interpretation
Real-world Self-Repair Gripper

Eight sequential stages; 20 physical rollouts per method; identical task data comprising 802 demonstration episodes and 570 correction episodes.

Reported full joint: 55% success and 76.0% normalized score. Action-only: 45% and 66.9%.

Complete-task success and normalized partial-credit score

ManiFlow: 5% success and 33.3% score; π0.5: 0% and 26.3%.

Analysis: This is executed robot performance. Joint generation improves complete success by 10 percentage points over action-only in this small evaluation. Partial credit must remain distinct from finishing the assembly. E12E13E14

RoboTwin demonstration efficiency

50 domain-randomized tasks with 50 demonstrations per task; baselines retrained at this budget.

Reported full joint: 78.8%.

Average task success

Fast-WAM: 41.9%; π0.5: 31.4%; LingBot-VA: 17.2%.

Analysis: The strongest listed baseline trails by 36.9 percentage points, approximately a 1.88-fold success ratio. This establishes an advantage in this simulation training regime, not equivalent real-world data efficiency. E16

Full-data RoboTwin

50 tasks; training uses 2,500 clean and 25,000 randomized demonstrations; one FLEX-π checkpoint serves both modes.

Reported action-only: 94.5%, 94.6%, 94.6%. Full joint: 94.3%, 94.8%, 94.6%.

Clean, randomized, and combined average success

Published combined averages: Qwen-RobotManip-Context 93.9%; LingBot-VA 2.0 93.6%.

Analysis: Future generation provides no advantage in the rounded aggregate here. Its benefit is task- and data-regime-dependent. E15

Standard LIBERO

Fine-tuning across 40 tasks in four suites, 50 demonstrations per task; evaluation comprises 50 rollouts per task, totaling 2,000 episodes.

Reported flexible checkpoint: 98.4% action-only and 98.5% full joint. Fixed-mode FLEX-π*: 98.7% and 99.2%, respectively.

Average success

Qwen-RobotManip-Context reports 99.2%; LingBot-VA reports 98.5%.

Analysis: The 99.2% result ties the strongest listed baseline and belongs to fixed-mode fine-tuning without stream dropout. It is not the performance of the flexible checkpoint. The authors characterize this benchmark as action fitting rather than held-out-task generalization. E10E17

LIBERO-Plus robustness

10,030 perturbed tasks, one rollout each; official task-count-weighted total and corrected instruction extraction.

Reported action-only and full joint: 88.6% each.

Weighted success

Under the same corrected pipeline: π0.5 85.7%, Fast-WAM 70.8%. Published Qwen totals are 89.0% and 91.4%, but their instruction handling was not verified.

Analysis: Comparisons with the two reevaluated baselines are directly aligned on instruction handling. Ranking against the remaining published rows carries an unresolved protocol difference. E18

Effect of inference-time future generation

Five domain-randomized RoboTwin tasks, 50 demonstrations per task, five epochs from scratch and RGB-only conditioning; output streams varied on one checkpoint without further training.

Reported action-only: 40.2% at approximately 60 ms. Adding RGB futures: 60.4%. Generating RGB, DINO, and pointmap futures: 63.8% at approximately 193 ms.

Average success and approximate latency per policy call

Full visual generation exceeds action-only by 23.6 percentage points.

Analysis: Holding the checkpoint fixed supports a direct deployment benefit from future generation in this setting. These five-task results must not be conflated with the full-data RoboTwin averages. E19

Deployment computation

RTX 5090, batch size one, four Euler steps, full current visual input, best reported implementation for each path.

Reported optimized action-only: 60.3 ms; optimized full joint: 193.3 ms.

Measured mean latency per action-chunk call

Corresponding eager implementations: 131.8 ms and 447.5 ms.

Analysis: Future generation remains substantially more expensive after optimization. These are chunk latencies, not control-step durations, and comparisons with other policies do not hold engineering effort fixed. E13E22

6.3 Ablations and diagnostic examples

Read component removals and qualitative examples within their stated evaluation conditions.

Figure 11. Input richness and output computation are two different ablations. Original paper, p. 11 ↗

Excerpt from the authors’ paper; cropped without altering the figure or table.

How to read it. The left panel changes the available representations cumulatively: video, video plus DINO, then all three visual streams. Its vertical axis starts at zero. The right panel instead fixes RGB-only input and changes generated future streams from the same checkpoint. Moving right spends more inference time while success rises from action-only through video, DINO and pointmaps. Notice the break in the right vertical axis between the low baseline point and the FLEX-π points; its distances cannot be compared directly with the left panel. The green curve is the relevant within-model tradeoff. The orange compiled baseline also changes architecture and implementation. E19E28

What it supports. The paper reports 40.2% success for action-only generation, 60.4% after adding video futures, and 63.8% with all future streams in this diagnostic. The visible curve therefore supports a gain from imagined context under this restricted-input setting, with most of the displayed improvement arriving when video prediction is enabled.

Where the evidence stops. Section 4.5 and Appendix G.5 specify five tasks, 50 demonstrations per task and five epochs from scratch. Figure 11b varies outputs from that checkpoint without further training. Its restricted protocol must remain separate from the 50-task RoboTwin comparison.

7. Analysis & limitations

7.1 What the evidence leaves open

Author claim

The authors report slower convergence from additional modalities and cross-modality forcing, requiring at least ten fine-tuning epochs on their real-world tasks. Full joint generation remains slower than parameter-comparable VLAs. E24

Reader analysis

Sensor-free deployment is narrower than the main experimental configuration: real-robot pointmaps normally use sensed depth. On the reported plate-task ablation, withholding depth changes completion from 95.0% to 91.7%. Without accompanying uncertainty, this does not establish statistical equivalence despite the authors' negligible-cost wording. E08E21

Reader analysis

Real-world evaluations use only 10 or 20 rollouts per task. Baselines also differ in camera representation and chunk length, and Fast-WAM is absent from the two hardest tasks. Identical demonstrations do not isolate architecture from the entire training and deployment recipe. E12E13

Reader analysis

The cumulative input-stream ablation also changes which futures are predicted, so it does not isolate conditioning from supervision. The text describes its gains and the cross-modality-forcing gain using percentages without sufficiently explicit absolute-versus-relative interpretation; their numerical magnitudes are therefore omitted here. E20

Source description

The Figure 11 output sweep uses a single checkpoint from the five-task, 50-demonstration-per-task, five-epoch from-scratch recipe specified in Section 4.5 and Appendix G.5, with no further training between output modes. This diagnostic has a different task and training scope from the main 50-task RoboTwin comparison. E19

Reader analysis

Because visual predictions never attend to action tokens, this architecture does not demonstrate an action-conditioned transition model for evaluating arbitrary candidate controls. Successful joint action prediction should not be interpreted as validated counterfactual physical simulation. E04E05

7.2 Questions for discussion

  1. How much of the real-world advantage survives when both current inputs and future outputs are restricted to RGB and actions, with total deployment latency measured consistently? [E08, E21, E22]
  2. Does cross-modality forcing improve action representations beyond simply adding geometric and semantic target losses when conditioning and training budgets are held fixed? [E05, E20]
  3. Would shorter feedback intervals change the advantage of joint generation on precise assembly, given that the current robot executes entire 32-step chunks open-loop? [E08, E14]

8. Reproducibility audit

8.1 Requirements and known gaps

Source description

Reconstruction requires the Wan-2.2-5B components, frozen DINOv3 and umT5, AGIBOT sub-episode selection, depth-derived pointmaps, composite camera layout, canonical state/action normalization, and the specified attention masks. Real-world reproduction additionally needs the shared task demonstrations and correction episodes. E03E05E07E08E09E13

Open question

The supplied text does not provide the promised per-domain batch sizes and epoch counts. Other details needing clarification include numerical pointmap scaling before VAE encoding, the flow-time shift implementation, endpoint handling for Equation (4), and the exact masking/loss rule used to disable cross-modality forcing. E03E06E11E20E25

Source description

Latency reproduction should use the reported PyTorch 2.7.1, CUDA 12.8, TensorRT 10.16, RTX 5090, three warmup calls, and twenty synchronized timed calls. Observations must rotate between calls: reusing a fixed observation can incorrectly avoid the approximately 20 ms prefill cost. Task language encoding is cached outside the timed region. E22

Reader analysis

A bounded mechanism test would repeat the five-task output sweep using one checkpoint, identical RGB inputs, fixed Euler steps, and matched rollout seeds. A subsequent training comparison should separately control extra conditioning, extra target losses, and cross-modality forcing to identify which produces the benefit. E19E20

Source description

Only the supplied primary text was reviewed. The title-page project link was not followed; code, released weights, installation, and experimental reproduction were not inspected or verified. E01

8.2 Proposed reproduction checks

The following checks are proposals motivated by the paper. They have not been run as part of this reading.

Check 1: Is the output-mode gain stable across training seeds?

Reader-proposed experiment: repeat Appendix G.5’s five-task, 50-demonstration-per-task, five-epoch from-scratch recipe across training seeds, keeping initialization policy and optimization settings fixed. Evaluate identical episodes with RGB-only input under action-only, video-plus-action and full-joint outputs from each checkpoint. Record paired successes and inference latency with the same hardware and optimizations. A repeatable gain after enabling futures would support the intended computation-quality tradeoff; a gain that disappears across seeds would narrow that claim. E15E19E22E28

Check 2: Does a faster mode improve the closed-loop system or only the model call?

Reader-proposed experiment: on a fixed real task, compare both output modes with identical observations, action-chunk length, Euler steps and compilation. Log acquisition-to-command delay, hold time, partial rubric score and full completion. Add a control that delays action-only execution to match the joint-mode cycle. If joint mode still improves completion under matched timing, richer future context is a plausible explanation; if differences track delay instead, controller scheduling is an essential mediator. This is a proposed test, not a new result. E08E09E14E22E23

8.3 Reading coverage

Visual audit: Visually inspected Figures 2, 4, 11 and 19 and Table 1 in the source PDF, then inspected all five final crops. Table 1 was additionally rendered at 400 DPI for a legible enlargement of its narrow column, following the pilot’s explicit resolution adjustment; other pages use 200 DPI. Figure 11 retains its broken vertical axis. Section 4.5 and Appendix G.5 specify the five-task from-scratch recipe, and G.5 clarifies that Figure 11b switches outputs from one checkpoint without further training. No experiment, model code, or external implementation was run or inspected in this visual pass.

PDF pages inspected for this edition: 4, 6, 9, 11, 33. Appendix coverage: reviewed.

Original figures and tables remain the work of the source’s authors. Extractions preserve their scientific content; any HTML wrapper layout is disclosed with each figure. The surrounding reading notes are our own.

Text reading scope & known omissions
  • Abstract
  • 1 Introduction
  • 2 Related Work
  • 3 Problem Statement and Flow Matching
  • 3.1 Input Visual Streams and Model Backbone
  • 3.2 Flexible Training via Visual Stream Dropout and Cross-Modality Forcing
  • 3.3 Training Objectives, Data, and Inference
  • 4.1–4.5 Experimental Setup, Real-World Evaluation, Generalization, Simulation Comparisons, and Ablations
  • 5 Conclusion and Limitation
  • Acknowledgments and References
  • A.1–A.5 Model Architecture Details
  • B Pre-training Data
  • C Real-World Robot Platform
  • D Real World Experiment Criteria
  • E LIBERO Setup and Evaluation Protocol
  • F.1–F.3 Baseline Implementation and Latency Protocol
  • G.1–G.7 Additional Experiments
  • H Per-Task RoboTwin Results
  • I Inference Optimization and I.1 Denoising Steps
  • J Training Hyperparameters

Outside the original text pass

  • The catalog spells the model prefix Flex-ππ; the supplied title page spells it FLEX-π. The exact arXiv identifier, all seven authors, and the subtitle agree, establishing work identity while leaving this catalog spelling discrepancy recorded. The observed source is arXiv:2608.10860v3, dated 29 August 2026; the catalog was not changed.
  • Only the supplied extraction was inspected. Figure captions and legible extracted labels were read, but no figure images or original PDF layouts were visually inspected.
  • Source hash and acquisition time were copied from the supplied provenance, not independently recomputed.
  • Separate supplemental material availability was not verified.
  • Section J promises per-domain training settings, but the supplied Table 12 contains no domain-specific epoch or batch-size group.
  • Text extraction does not reconstruct figure images; inspect the retained PDF for figures and equation/table layout.
  • Separate supplemental material availability has not been fully verified.

The visual audit above records the subsequent illustrated pass.

8.4 Traceable evidence

E01PDF p. 1, title, author block, project URL, and arXiv footerInspect

The title is FLEX-π with the supplied subtitle; all seven catalog authors appear. The footer identifies arXiv:2608.10860v3, dated 29 August 2026.

Go to primary source ↓
E02PDF pp. 1–3, Abstract, Section 1, and Section 3 Problem StatementInspect

The paper motivates semantic and geometric targets beyond RGB reconstruction and formulates flexible prediction of action chunks and visual futures conditioned on observations, state, and language.

Go to primary source ↓
E03PDF pp. 4–5, Section 3.1 Visual Input Encoding; p. 23, Section A.3 Per-modality adapters and DINO feature foldingInspect

RGB and pointmaps use the frozen Wan VAE. DINO features use invertible 2×2 folding from 768 to 3072 dimensions, LayerNorm, and Xavier-initialized linear projections.

Go to primary source ↓
E04PDF p. 5, Section 3.1 Model Backbone and Causal Joint Generation; p. 23, Section A.3 Action expert initializationInspect

The visual trunk has approximately 5B parameters and the separate action expert approximately 1B. Fusion occupies the middle 16 of 30 blocks; visual tokens do not attend to actions. Action weights are initialized by resampling Wan tensors.

Go to primary source ↓
E05PDF p. 5, Section 3.2; p. 24, Section A.5 Sampling, Attention rules, and Training versus inferenceInspect

Independent Bernoulli masks retain at least one observed modality. All targets train; future streams communicate within equal-mask groups. At inference, unread futures can be removed from computation.

Go to primary source ↓
E06PDF p. 4, Equation (1); p. 6, Equation (3); pp. 23–24, Section A.4 and Equation (4)Inspect

Flow matching regresses the noise-to-data velocity. Four equally weighted stream losses are always applied. DINO clean-feature predictions are converted to velocity by division by one minus flow time.

Go to primary source ↓
E07PDF p. 6, Section 3.3 Pre-training and Fine-tuning; p. 24, Appendix BInspect

Pre-training uses approximately 500 hours from 100 AGIBOT tasks, selecting up to 285 episodes each and resegmenting by language annotations. All three views receive offline estimated depth and intrinsics-based pointmaps.

Go to primary source ↓
E08PDF pp. 24 and 26, Appendix C Hardware, Observations and actions, and DeploymentInspect

The real robot uses three stereo cameras, sensed depth, a composite canvas, and current observations only. It executes 32 commands at 30 Hz before synchronous replanning and holds position during inference.

Go to primary source ↓
E09PDF pp. 22–23, Sections A.1 and A.2Inspect

State and actions use canonical 32-dimensional slots. Proprioception becomes one language-conditioning token. Action chunks contain 32 continuous tokens; real-robot targets share a chunk-start anchor, and visual targets cover the same horizon.

Go to primary source ↓
E10PDF p. 26, Appendix E Camera layout, Depth and pointmaps, States and actions, and EvaluationInspect

LIBERO depth is rendered by replaying demonstrations; pointmaps use camera intrinsics. One checkpoint trains across 40 tasks and is evaluated over 2,000 episodes, executing ten actions between replans.

Go to primary source ↓
E11PDF pp. 40–41, Appendix J and Table 12Inspect

The text identifies frozen encoders and trained model components. Table 12 specifies AdamW, learning rate 10⁻⁴, cosine scheduling, 5% warmup, bfloat16, ZeRO stage 1, equal loss weights, and modality-dependent flow shifts.

Go to primary source ↓
E12PDF p. 7, Section 4.2; p. 27, Table 3; p. 29, Section G.1 ProtocolInspect

Real-world scoring separates normalized rubric credit from complete success. Tasks receive 20 rollouts except Sort Utensils, which receives ten; methods are interleaved and placements randomized.

Go to primary source ↓
E13PDF pp. 28–29, Sections F.2–F.3 and Tables 4–5Inspect

Methods share task datasets but retain different observation layouts and chunk lengths. Fast-WAM omits the hardest two tasks. Self-Repair data include 802 demonstrations and 570 corrections. Latency comparisons use deployed, unequally optimized configurations.

Go to primary source ↓
E14PDF p. 33, Section G.3 and Figure 19(b) legible extracted metric labelsInspect

The extracted score/success pairs are 26.3/0 for π0.5, 33.3/5 for ManiFlow, 66.9/45 for action-only, and 76.0/55 for full joint. The prose confirms 20 rollouts, one ManiFlow completion, no π0.5 completion, and a ten-point joint-generation success improvement.

Go to primary source ↓
E15PDF p. 9, Table 1, FLEX-π, Qwen-RobotManip-Context, and LingBot-VA 2.0 rowsInspect

FLEX-π action-only reports 94.5/94.6/94.6 and full joint 94.3/94.8/94.6 for clean/randomized/average success. The named comparison averages are 93.9 and 93.6.

Go to primary source ↓
E16PDF p. 34, Section G.6; p. 28, Section F.1 RoboTwin data-scaling paragraphInspect

At 50 demonstrations per task over 50 randomized tasks, full joint reaches 78.8%, compared with 31.4% for π0.5, 41.9% for Fast-WAM, and 17.2% for LingBot-VA. Baselines are retrained at reduced budgets.

Go to primary source ↓
E17PDF p. 10, Section 4.4 LIBERO and Table 2; p. 35, Table 6 caption and Avg columnInspect

Flexible deployment scores 98.4/98.5, while fixed-mode FLEX-π* scores 98.7/99.2. The starred variant removes stream dropout during fine-tuning. Qwen-RobotManip-Context reports 99.2.

Go to primary source ↓
E18PDF pp. 26–28, Appendix E LIBERO-Plus and Section F.1; p. 36, Table 7 caption and Total columnInspect

Evaluation uses 10,030 tasks and task-count weighting. Corrected instruction handling yields 88.6 for both FLEX-π modes, 85.7 for π0.5, and 70.8 for Fast-WAM. Other baselines were not rerun and their instruction handling is unknown.

Go to primary source ↓
E19PDF pp. 10–11, Section 4.5 opening and Output Modalities; p. 34, Section G.5Inspect

With RGB-only input, success rises from 40.2 to 60.4 to 63.8 as futures are added; endpoint latencies are approximately 60 and 193 ms. Section 4.5 and Appendix G.5 specify five domain-randomized tasks, 50 demonstrations per task and five epochs from scratch. G.5 names lift pot, place shoe, pick diverse bottles, place object basket and stack bowls two. It explicitly states that Figure 11b adds no further training and varies only the outputs generated by a single checkpoint from that recipe. This clarifies the main text’s loose description of the checkpoint as pre-trained.

Go to primary source ↓
E20PDF pp. 10–11, Section 4.5 Input Modalities and Cross-Modality Forcing, and Figure 11 caption; p. 34, Section G.5Inspect

The cumulative modality study observes and predicts all available streams. The text reports improvements when adding modalities and deterioration without cross-modality forcing, but expresses their magnitudes using percentage notation without explicit relative-versus-point definitions.

Go to primary source ↓
E21PDF p. 32, Section G.2 Real-World Depth Input AblationInspect

On Put Plate on Rack under full joint generation, completion is 95.0% with depth input and 91.7% without. No uncertainty accompanies these values in the supplied subsection.

Go to primary source ↓
E22PDF pp. 38–39, Appendix I Protocol and Prefill/decode split; Table 9 rows L0/L6 and Table 10 rows A0/A4, K=4 columnInspect

Full-input joint latency is 447.5 ms eager and 193.3 ms optimized; action-only is 131.8 and 60.3 ms. Measurements use an RTX 5090, specified software versions, three warmups, and twenty synchronized calls. Static observations can hide approximately 20 ms of prefill cost.

Go to primary source ↓
E23PDF p. 40, Section I.1 and Table 11Inspect

The action-only sweep peaks at four Euler steps, with 94.5% clean and 94.6% randomized success. Two steps remain close, while one step substantially reduces success. Four steps are used in both main deployment modes.

Go to primary source ↓
E24PDF p. 11, Section 5 LimitationsInspect

The authors state that richer modalities and cross-modality forcing slow convergence, requiring at least ten real-world fine-tuning epochs, and that full joint generation is slower than comparable VLAs.

Go to primary source ↓
E25PDF p. 40, Appendix J introductory paragraph; p. 41, complete supplied Table 12Inspect

Appendix J says per-domain epoch and batch-size values appear in a final group, but the supplied table ends after shared stream-dropout settings and contains no such group.

Go to primary source ↓
E26PDF p. 4, Figure 2; original architecture diagram visually inspectedInspect

RGB and pointmap rows share Wan VAE encoder weights; DINOv3 provides a separate representation. Input stream presence and output generation masks surround the visual MoT and separate action expert. The visual-to-action arrow is directional; outputs include three future visual representations and an action chunk.

Go to primary source ↓
E27PDF p. 6, Figure 4; original attention diagrams visually inspectedInspect

The four regimes distinguish input presence, output visibility, future attention and inference computation. Solid, arrow, dashed, dotted and crossed edges have an explicit legend. The starred cross-modality-forcing row omits the current pointmap but retains a future pointmap read by actions; loss masking is not depicted or implied.

Go to primary source ↓
E28PDF p. 11, Figure 11; original plots visually inspectedInspect

Panel (a) cumulatively adds input representations. Panel (b) plots success against inference latency for an RGB-only input setting with additional output streams, and has a broken vertical axis between the low Fast-WAM point and FLEX-π points. No exact intermediate DINO success label is printed on the curve.

Go to primary source ↓
E29PDF p. 33, Figure 19; original rubric and result bars visually inspectedInspect

Panel (a) allocates three points across eight sequential stages, with insertion and fastening stages highlighted. Panel (b) distinguishes normalized partial score from full-task success. FLEX-π action-only/full-joint bars read 66.9/76.0 for score and 45/55 for complete success; π0.5 reads 26.3/0 and ManiFlow 33.3/5.

Go to primary source ↓

8.5 Primary sources

Scroll across the image to inspect details. Press Esc to close.