Driving · 2026-10-05
Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models
Junyi Hu, Zhewen He, Zhenhua Li, Yi Fang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose PEWAM, an actor-free “effect program” world-action model compiled from a demonstration’s object keypoint trajectories, contact points, and terminal state. PEWAM generates effect, robot execution, action, and terminal state as four streams with independent flow-matching times; clamping a program and sampling the execution turns inference into programming. Re-solving from the live scene absorbs pushes and enables editing and multi-arm transfer.
Why it matters
This paper separates what happens in the world (editable effect coordinates) from how a specific arm moves, aiming to make single-demonstration programming work across layouts, disturbances, and robot bodies without parameter updates. The program interface (terminal state + contact points) is presented as both readable/editable by people and robust when replaying fails due to pushes or changed robot embodiment.
Method
- Compile a demonstration into an actor-free effect program: 3D object keypoint trajectories, two contact points, and a terminal state with the demonstrator removed.
- Use PEWAM (71.5M parameters) that generates effect, end-effector execution, action, and terminal state as four streams with separate flow-matching times; clamping a program makes inference act like programming.
- Run closed-loop “re-solving” by matching against live keypoints and using an engagement gate so progress is counted only for effects caused by the robot, not by pushes.
Limitation
On the five-class mean on Meta-World, “PEWAM is below DAMI” and “though not on the five-class mean.”
Abstract (from arXiv)
A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.
Related papers
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- OpenWAM: An Open Framework for Composable World-Action Models
- PrimeSim: A Generalist World Model Simulator from Large-Scale Cross-Embodiment Demonstrations
- EVO-WAM: Evolving World Action Models through Video-Action Verification
- RobotWorld: Benchmarking Multimodal Agents for Robot Use Across Diverse Tasks and Embodiments
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-01 · Indexed by Pub-AI: 2026-10-05 UTC
Keep the Effect, Drop the Actor: Programmable Effect-to-Execution World-Action Models
Preprint posted: 2026-10-01 · Original record ↗
- arxiv
- 2610.02398
Explicit source versions (1)
- v1 · 2026-10-01
- Preprint available
Source date: 2026-10-01 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
A robot demonstration records two things in the same frames: what happened to the objects, and how one particular arm made it happen. We condition on the first. A demonstration is compiled into an effect program: the 3D keypoint trajectories of the objects that moved, two points marking where each was held, and the configuration the scene ends in, with the demonstrator removed. PEWAM, a 71.5M-parameter world-action model, generates effect, robot execution, action and terminal state as four streams with independent flow-matching times, so clamping a program and sampling the execution turns inference into programming, re-solved closed loop from the live scene. On held-out LIBERO-Goal tasks, one demonstration's program completes 40 of 90 episodes, where the same backbone given a goal image or language, and published demonstration-conditioned methods, complete at most 19; on three of Meta-World's held-out classes it exceeds the best published results, though not on the five-class mean. Because a program is a set of coordinates, a person can edit it: the placement follows a shifted terminal state and the grasp turns with rotated contact points. The same program runs on four robot arms without retraining, and after a push, re-solving completes 23 of 60 episodes where replaying the demonstration completes 6. On a Franka arm, fine-tuned on real demonstrations of other tasks, programs compiled from single human videos complete 36 of 40 trials, against 22 for the same backbone conditioned on the video's last frame as a goal image.
