Robotics · 2026-10-19

AWAM: Active Trajectory-Conditioned View Imagination for Robust Robot Manipulation

Jiawei Geng, Zhenyang Liu, Yuzheng Wu, Yanwei Fu

arXiv:2610.23396PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AWAM uses a task-conditioned virtual camera trajectory to imagine geometry-consistent observations along the planned path before acting. The predicted views are generated with a geometry-conditioned world model, compressed into a reusable predictive memory once per episode, then fused with live observations via cross-attention for diffusion-based action prediction. The authors report high success on occlusion and distribution-shift benchmarks and in real-robot trials.

Why it matters

The authors target a perceptual gap: in open-world manipulation, occlusion and viewpoint shifts can remove task-relevant evidence needed for control. By separating virtual active perception (virtual viewpoint queries + predictive memory) from physical robot motion, AWAM aims to provide complementary spatial context when the initial view is incomplete.

Method

  • Predict a task-conditioned virtual camera trajectory from initial RGB-D + language, using demonstrated wrist-camera paths; the robot never physically executes the predicted trajectory before manipulation.
  • Use a geometry-conditioned world model (WorldStereo, adapted) to render/predict view-consistent representations along the virtual path, then compress them into a fixed predictive world memory (inter-view attention; cached for the episode).
  • During closed-loop control, retrieve predictive context from the cached memory via residual cross-attention with live static- and wrist-camera tokens, then use a conditional diffusion transformer to generate short-horizon action chunks.

Limitation

The current system relies on calibrated cameras and depth information to construct geometry-aware virtual observations. It also constructs M0 once from the initial scene and keeps it fixed; when objects are substantially displaced or articulated, parts of the cached memory may become stale (fresh live observations provide primary feedback, but the memory is not explicitly updated).

Abstract (from arXiv)

Rapid progress in embodied intelligence has enabled increasingly capable robots to follow language instructions and manipulate diverse objects. However, strong results in structured environments do not automatically transfer to open-world conditions, where clutter, occlusion, and changing viewpoints can obscure the spatial relationships required for action. To address this perceptual gap, a virtual active-perception paradigm is introduced. AWAM, an Active World-Action Model actively plans where to observe and imagines what will be seen along the planned trajectory before acting. Task-related virtual camera motion is predicted and used as a structured query into a geometry-conditioned world model, producing a reusable spatial prior for manipulation, which is later compressed into a fixed memory and fused with live observations through cross-attention for diffusion-based action prediction. AWAM achieves competitive success rates with 75.9% on LIBERO-Occ, 62.4% on LIBERO-Plus, 76.5% on RLBench-OG Occlusion1, and 94.4% in real-robot trials. These results support improved robustness to the evaluated occlusions and distribution shifts, addressing part of the gap between structured manipulation and less controlled environments.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-16 · Indexed by Pub-AI: 2026-10-19 UTC

preprint

AWAM: Active Trajectory-Conditioned View Imagination for Robust Robot Manipulation

Jiawei Geng, Zhenyang Liu, Yuzheng Wu, Yanwei Fu

Preprint posted: 2026-10-16 · Original record ↗

arxiv
2610.23396
Explicit source versions (1)
  • v1 · 2026-10-16
  1. Preprint available

    Source date: 2026-10-16 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Rapid progress in embodied intelligence has enabled increasingly capable robots to follow language instructions and manipulate diverse objects. However, strong results in structured environments do not automatically transfer to open-world conditions, where clutter, occlusion, and changing viewpoints can obscure the spatial relationships required for action. To address this perceptual gap, a virtual active-perception paradigm is introduced. AWAM, an Active World-Action Model actively plans where to observe and imagines what will be seen along the planned trajectory before acting. Task-related virtual camera motion is predicted and used as a structured query into a geometry-conditioned world model, producing a reusable spatial prior for manipulation, which is later compressed into a fixed memory and fused with live observations through cross-attention for diffusion-based action prediction. AWAM achieves competitive success rates with 75.9% on LIBERO-Occ, 62.4% on LIBERO-Plus, 76.5% on RLBench-OG Occlusion1, and 94.4% in real-robot trials. These results support improved robustness to the evaluated occlusions and distribution shifts, addressing part of the gap between structured manipulation and less controlled environments.