Robotics · 2026-10-12

CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization

Yinlong Dai, Homanga Bharadhwaj, Dylan P. Losey

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

CASTER formalizes zero-shot manipulation from synthetic videos as constrained optimization: synthetic videos seed an initial guess of object motion, and a robot refines trajectories in a digital twin to satisfy semantic consistency and physical feasibility. The authors analyze requirements on video generation via “semantic basins of attraction” and report improved average zero-shot task success (76.6%) over the best-performing baseline by 34.1%.

Why it matters

The paper targets a deployment gap: generated videos can be visually plausible but physically infeasible or ungrounded for robots. By casting transfer as constrained optimization and deriving video-generation requirements (probabilistic seeding within semantic basins), CASTER provides a unified framework to explain prior methods and motivate objective/constraint choices for more reliable zero-shot manipulation.

Method

Extract object trajectories from synthetic videos, map them to task features ϕ, then optimize robot-induced trajectories to minimize feature deviation while enforcing semantic consistency and physical feasibility constraints (probabilistically relaxed).
Use a digital twin to enforce physical feasibility constraints with safety margins accounting for sim-to-real mismatch, and sample/optimize trajectories within this constrained setup.
Derive a basin-of-attraction view showing success depends on whether video-generated features probabilistically fall within a semantic basin where optimization can locally modify motions without losing task alignment; remove simplifications in state-of-the-art approaches via feature-weighted objectives.

Abstract (from arXiv)

Prior research on learning from humans often requires real human demonstrations. But advances in video generation models offer a new paradigm: we can now prompt these generators to make synthetic videos of the task. On the plus side, these synthetic videos take place in the robot's current environment, and do not require actual human teaching. However, while the generated videos appear to show plausible motions, they lack both physical consistency and robot grounding, and thus the robot cannot directly roll out the trajectory shown by the video. In this paper we seek to formalize the process of going from synthetic video generation to real-world robot trajectories for zero-shot manipulation tasks. We recognize that this problem is an instance of constrained optimization. The object trajectories extracted from the synthetic video provide an initial guess at the correct motion, and the robot must refine these trajectories (for instance, within a digital twin) while enforcing physical feasibility and semantic consistency. Our optimization-based formalism provides insights into the video generation properties necessary for success. Specifically, we show that the video generation must probabilistically seed the search within a semantic basin of attraction where the trajectory can be locally modified without losing its task alignment. Our analysis further reveals that state-of-the-art approaches can be viewed as instances of our unified formalism with key simplifications. In our experiments we remove these simplifications to more closely match the underlying optimization, leading to more robust zero-shot trajectories from synthetic videos and improving average task success by 34.1% compared to the best-performing baseline. See videos at our project website: https://collab.me.vt.edu/CASTER

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-08 · Indexed by Pub-AI: 2026-10-12 UTC

preprint