Driving · 2026-10-05

Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models

Junyi Hu, Zhewen He, Zhenhua Li, Yi Fang

arXiv:2610.02398PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors propose PEWAM, an actor-free “effect program” world-action model compiled from a demonstration’s object keypoint trajectories, contact points, and terminal state. PEWAM generates effect, robot execution, action, and terminal state as four streams with independent flow-matching times; clamping a program and sampling the execution turns inference into programming. Re-solving from the live scene absorbs pushes and enables editing and multi-arm transfer.

Why it matters

This paper separates what happens in the world (editable effect coordinates) from how a specific arm moves, aiming to make single-demonstration programming work across layouts, disturbances, and robot bodies without parameter updates. The program interface (terminal state + contact points) is presented as both readable/editable by people and robust when replaying fails due to pushes or changed robot embodiment.

Method

  • Compile a demonstration into an actor-free effect program: 3D object keypoint trajectories, two contact points, and a terminal state with the demonstrator removed.
  • Use PEWAM (71.5M parameters) that generates effect, end-effector execution, action, and terminal state as four streams with separate flow-matching times; clamping a program makes inference act like programming.
  • Run closed-loop “re-solving” by matching against live keypoints and using an engagement gate so progress is counted only for effects caused by the robot, not by pushes.

Limitation

On the five-class mean on Meta-World, “PEWAM is below DAMI” and “though not on the five-class mean.”

Abstract (from arXiv)

A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-05 UTC

preprint

Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models

Junyi Hu, Zhewen He, Zhenhua Li, Yi Fang

Preprint posted: 2026-10-01 · Original record ↗

arxiv
2610.02398
Explicit source versions (1)
  • v1 · 2026-10-01
  1. Preprint available

    Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.