Robotics · 2026-10-07
AutodidactWAM: Cross-Modal Self-Distillation from Generated Video to Robot Actions
Sergei Kurchev, Iaroslav Kolomiets, Miguel Altamirano Cabrera, Artem Lykov, Dzmitry Tsetserukou
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors adapt a Cosmos 3 world-action model (video+robot actions) to a new embodiment (Unitree G1 with BrainCo five-fingered hands) and find a video–action asymmetry: generated videos are often plausible, while native co-generated actions are systematically mis-targeted. AutodidactWAM extracts action targets from the model’s own generated video via a frozen hand-pose estimator + IK, then fine-tunes only action-related layers using self-distillation objectives.
Why it matters
The work targets a practical failure mode in embodied adaptation: a model can render plausible task executions yet still output actions that do not reliably execute them. AutodidactWAM aims to recover executable action targets from the model’s own generated video without additional task-specific teleoperation, and reports improved real-robot closed-loop success rates at multiple task stages.
Method
- Generate a self-distillation pair with Cosmos 3: from an anchor frame+instruction+seed, obtain native action a− and generated video V; use a frozen perception stack Φ (hand detection + hand-pose model + IK) on V to recover an action estimate â.
- Build a preferred target a+ (e.g., SFT uses â; Flow-DPO uses a mix with the round’s start prediction) and keep the video teacher-forced so gradients update only action tokens (action-branch layers/LoRA).
- Fine-tune with either supervised relabeling or a rectified-flow adaptation of Diffusion-DPO (Flow-DPO), optionally adding a Cartesian trajectory anchoring (DTW) term; the distillation pair is refreshed in rounds.
Limitation
A^ is an estimate, not ground truth: it inherits RoboHaMeR’s domain gap and any generation artifacts, so the extracted target carries residual bias forward. Φ also defines what Flow-DPO optimizes toward, so the policy could in principle learn a satisfaction mode that does not generalize (reward-hacking visible only in closed loop). Reach is bounded by how far Φ generalizes; transfer to substantially different objects or tasks without retraining is untested.
Abstract (from arXiv)
World-action models (WAMs) such as Cosmos 3 jointly generate future video and robot actions from an observation and instruction. Adapting one such model with a lightweight LoRA fine-tune to a previously unseen robot, a Unitree G1 humanoid with five-fingered BrainCo hands, exposes a video-action asymmetry: the video renders plausible task executions, while the co-generated action is systematically mis-targeted. We evaluate closed-loop real-robot trials at three cumulative stages: pre-grasp, grasp, and pick-and-place. The native action succeeds only approximately 17%, 10%, and 7% of the time, respectively, and performs worse on held-out objects. We propose AutodidactWAM, a hand-pose estimator trained without teleoperation, followed by inverse kinematics, that runs on the model's generated video to recover action estimates. Paired with the native prediction, these recovered actions provide preferred targets for fine-tuning only the action-related layers, while the generated video is teacher-forced. We compare supervised relabeling with a rectified-flow adaptation of Diffusion-DPO. After one-time embodiment adaptation, self-distillation requires no additional task-specific teleoperation. The recovered-action gate reaches approximately 75%, 47%, and 42% pre-grasp, grasp, and pick-and-place success, compared with 17%, 10%, and 7% for the native action. A hybrid objective combining preference supervision, supervised target fitting, and Cartesian trajectory anchoring (DPO+SFT+DTW) performs best: on Oreo, the training object, it reaches 90% pre-grasp and 20% full-task success; on a held-out object, it reaches 80% and 30%. Plain Flow-DPO reaches 0% success despite 1.000 validation preference accuracy, indicating that the combination of training objectives, rather than the contrastive objective alone, drives the observed gains.
Related papers
- OpenWAM: An Open Framework for Composable World-Action Models
- Self-Distilling Future Video Representations for World Action Models
- WorldLine: Action-Driven Visual Simulation for Robotic Manipulation
- EVO-WAM: Evolving World Action Models through Video-Action Verification
- De-Diffusing WAMs: Action-Sufficient Direct Prediction for Robot Control
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-06 · Indexed by Pub-AI: 2026-10-07 UTC
AutodidactWAM: Cross-Modal Self-Distillation from Generated Video to Robot Actions
Preprint posted: 2026-10-06 · Original record ↗
- arxiv
- 2610.08119
Explicit source versions (1)
- v1 · 2026-10-06
- Preprint available
Source date: 2026-10-06 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World-action models (WAMs) such as Cosmos 3 jointly generate future video and robot actions from an observation and instruction. Adapting one such model with a lightweight LoRA fine-tune to a previously unseen robot, a Unitree G1 humanoid with five-fingered BrainCo hands, exposes a video-action asymmetry: the video renders plausible task executions, while the co-generated action is systematically mis-targeted. We evaluate closed-loop real-robot trials at three cumulative stages: pre-grasp, grasp, and pick-and-place. The native action succeeds only approximately 17%, 10%, and 7% of the time, respectively, and performs worse on held-out objects. We propose AutodidactWAM, a hand-pose estimator trained without teleoperation, followed by inverse kinematics, that runs on the model's generated video to recover action estimates. Paired with the native prediction, these recovered actions provide preferred targets for fine-tuning only the action-related layers, while the generated video is teacher-forced. We compare supervised relabeling with a rectified-flow adaptation of Diffusion-DPO. After one-time embodiment adaptation, self-distillation requires no additional task-specific teleoperation. The recovered-action gate reaches approximately 75%, 47%, and 42% pre-grasp, grasp, and pick-and-place success, compared with 17%, 10%, and 7% for the native action. A hybrid objective combining preference supervision, supervised target fitting, and Cartesian trajectory anchoring (DPO+SFT+DTW) performs best: on Oreo, the training object, it reaches 90% pre-grasp and 20% full-task success; on a held-out object, it reaches 80% and 30%. Plain Flow-DPO reaches 0% success despite 1.000 validation preference accuracy, indicating that the combination of training objectives, rather than the contrastive objective alone, drives the observed gains.
