Driving · 2026-10-15
Action Jets: Learning the Local Shape of the Action-conditioned Data Manifold
Anton Arapin, Chahyon Ku, Bernadette Bucher
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Action Jets estimate the local shape of action-conditioned data manifolds from generator-measured tangents plus higher-order corrections fit to recorded transitions. The resulting Taylor-polynomial “jet” can generate predicted futures for many candidate actions with one extraction per state, avoiding per-candidate generator inference for short-horizon planning. The authors report large speedups and improved held-out short-horizon accuracy on multiple benchmarks.
Why it matters
Planning and control with action-conditioned world models is costly because evaluators must score many candidate actions with repeated generator rollouts. By approximating a local manifold chart with a Taylor polynomial (tangent from the generator; higher orders from data), the authors reduce per-state inference to a fixed cost and allow many candidate trajectories via arithmetic, while improving short-horizon prediction accuracy and enabling closed-form validity radii.
Method
- Probing frozen action-conditioned generators: for each state and action axis, render at ±δ along the axis, encode both, and finite-difference them to estimate per-token action Jacobian fields \"\widehat{J}_i\".
- Train a geometry encoder to emit the shared tangent frame and per-axis Jacobian in one forward pass, then use a correction head trained on recorded transitions to fit second/third-order Taylor terms (a 3-jet).
- Compute a closed-form radius of validity from the Taylor coefficients and traverse the local Taylor approximation to score many candidate actions after a single per-state extraction.
Limitation
The authors state that the Taylor polynomial is no longer accurate for actions beyond its computed radius of validity, requiring a new chart approximation for larger state transitions.
Abstract (from arXiv)
World models are a powerful means of reasoning and planning, but many of these action-conditioned predictive models are high dimensional and computationally expensive. These challenges limit the number of action candidates that can be evaluated by a planner. We show that this cost is avoidable for planners over short horizons, which only require local information with respect to the current state. For a fixed state, the frames reachable by varying a low-dimensional action (a camera movement or an end-effector command) lie on a low-dimensional manifold. We estimate the local shape of that manifold directly via a network trained to estimate how each image region moves per unit of each action axis supervised by data collected from samples by the generator. A correction head fit to trajectory data estimates the shape of this manifold region up to a third-order Taylor polynomial. We call this network estimating the Taylor polynomial describing the manifold shape an Action Jet and compute its radius of validity in closed form. By traversing this approximation of the local manifold shape, we only have to perform inference once per state for any number of candidate actions. Scoring sixteen candidates from a single extraction is over 300x faster than with either source generator, and this gap widens with the number of candidates a planner evaluates. We measure against state-of-the-art action-conditioned generators, IRASim and WorldGym on BridgeData V2, a real 7-DoF manipulation benchmark, and FrameCrafter on DL3DV scenes, where we chart from the novel-view generators DepthSplat and LagerNVS. Because only the tangent comes from the generator and every other term from the data, the Action Jet also predicts held-out frames more accurately than IRASim, the generator that supplied its tangent, at short horizons and at over 100x less compute.
Related papers
- ForeAct: Geometry-Guided Action Learning from World Representations
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
- Beckmann World Models: Direct Terminal-Map Learning for One-Step Action-Conditioned Prediction
- E0.5: From Reconstructive to Future-Predictive 3D Foundations for Robotic World-Action Modeling
- WorldFlow: Joint Video and 3D Motion Prediction via Large-Scale Pretraining
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-15 UTC
Action Jets: Learning the Local Shape of the Action-conditioned Data Manifold
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.19007
Explicit source versions (1)
- v1 · 2026-10-13
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World models are a powerful means of reasoning and planning, but many of these action-conditioned predictive models are high dimensional and computationally expensive. These challenges limit the number of action candidates that can be evaluated by a planner. We show that this cost is avoidable for planners over short horizons, which only require local information with respect to the current state. For a fixed state, the frames reachable by varying a low-dimensional action (a camera movement or an end-effector command) lie on a low-dimensional manifold. We estimate the local shape of that manifold directly via a network trained to estimate how each image region moves per unit of each action axis supervised by data collected from samples by the generator. A correction head fit to trajectory data estimates the shape of this manifold region up to a third-order Taylor polynomial. We call this network estimating the Taylor polynomial describing the manifold shape an Action Jet and compute its radius of validity in closed form. By traversing this approximation of the local manifold shape, we only have to perform inference once per state for any number of candidate actions. Scoring sixteen candidates from a single extraction is over 300x faster than with either source generator, and this gap widens with the number of candidates a planner evaluates. We measure against state-of-the-art action-conditioned generators, IRASim and WorldGym on BridgeData V2, a real 7-DoF manipulation benchmark, and FrameCrafter on DL3DV scenes, where we chart from the novel-view generators DepthSplat and LagerNVS. Because only the tangent comes from the generator and every other term from the data, the Action Jet also predicts held-out frames more accurately than IRASim, the generator that supplied its tangent, at short horizons and at over 100x less compute.
