Robotics · 2026-10-01
Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models
Qi Lyu, Jiahua Dong, Hao Shen, Xudong Wang, Hongyuan Yu, Baichen Liu, Henghui Ding, Zhi Han, Nicu Sebe, Ivan Laptev, Fahad Shahbaz Khan, Salman Khan
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors report an Action Experience Dictionary (AED) for World Action Models that encodes historical action trajectories into shared action embeddings for cross-task skill reuse. They retrieve AED embeddings with a pretrained action tokenizer, visually condition them via cross-attention, and prepend them to action tokens for prediction. A motion-aware transition loss supervises visual feature change over random temporal intervals to reduce reliance on background cues. They evaluate on LIBERO, RoboTwin, and real-world cross-embodiment tasks.
Why it matters
The paper targets two stated issues with existing World Action Models: (1) lack of explicit reuse of action experience across manipulation tasks, and (2) difficulty learning cross-task semantic relationships because redundant background interferes with extracting action-relevant information. The proposed AED plus visual conditioning and motion-aware transition supervision is intended to make action experience transferable and more motion-focused.
Method
- Construct a shared learnable Action Experience Dictionary (AED) of action embeddings; use a pretrained action tokenizer to map aggregated historical actions to AED indices and retrieve task-relevant embeddings.
- Visually condition retrieved action embeddings by compressing visual latent patch tokens into fixed latent visual tokens and using cross-attention in a dictionary encoder; prepend the resulting embeddings to noisy action tokens for action-chunk prediction.
- Add a motion-aware transition (MT) loss that predicts visual feature change between randomly sampled future intervals from AED-conditioned action hidden states to encourage action-related motion and reduce background reliance.
Limitation
The authors state two limitations: the computation/training pipeline speed can be affected by not caching visual features in advance, and the finite history window limits access to longer-term dependencies; they plan to address both in future work.
Abstract (from arXiv)
World Action Models (WAMs) couple visual dynamics prediction with action generation, yet they do not explicitly support the reuse of action experience across manipulation tasks. Furthermore, existing WAMs struggle to capture underlying cross-task semantic relationships that could guide target action prediction, as redundant background elements interfere with the extraction of key visual information. To address these challenges, we develop a novel Action Experience Dictionary (AED) that encodes historical physical action trajectories into shared action embeddings to support skill reuse and model cross-task relationships. Specifically, we first aggregate historical actions to align with visual observations and retrieve action embeddings from the AED using a pretrained action tokenizer. Subsequently, we visually condition the pooled embeddings through cross-attention and prepend them to noisy action tokens, providing interaction context and action intent for prediction. To model action-related motion and reduce reliance on irrelevant background cues, we introduce a motion-aware transition loss that supervises visual feature change prediction over random temporal intervals. Experiments on simulation benchmarks and in real-world cross-embodiment settings verify the effectiveness of our AED. The anonymous project website is available at \href{https://github.com/JiahuaDong/AED}{AED}.
Related papers
- OnlyAct: Extracting Action Representations from Foundation Models for World Models
- What Visual Representations Do World Action Models Need?
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
- WorldBridge: Bridging Video and Action for Task Generalization in World Action Models
- UniLAP: A Unified Latent Action Policy for Cross-Embodiment Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-30 · Indexed by Pub-AI: 2026-10-01 UTC
Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models
Preprint posted: 2026-09-30 · Original record ↗
- arxiv
- 2609.40219
Explicit source versions (1)
- v2 · 2026-10-02
- Explicit source version detected · v2
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
- Preprint available
Source date: 2026-09-30 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
World Action Models (WAMs) couple visual dynamics prediction with action generation, yet they do not explicitly support the reuse of action experience across manipulation tasks. Furthermore, existing WAMs struggle to capture underlying cross-task semantic relationships that could guide target action prediction, as redundant background elements interfere with the extraction of key visual information. To address these challenges, we develop a novel Action Experience Dictionary (AED) that encodes historical physical action trajectories into shared action embeddings to support skill reuse and model cross-task relationships. Specifically, we first aggregate historical actions to align with visual observations and retrieve action embeddings from the AED using a pretrained action tokenizer. Subsequently, we visually condition the pooled embeddings through cross-attention and prepend them to noisy action tokens, providing interaction context and action intent for prediction. To model action-related motion and reduce reliance on irrelevant background cues, we introduce a motion-aware transition loss that supervises visual feature change prediction over random temporal intervals. Experiments on simulation benchmarks and in real-world cross-embodiment settings verify the effectiveness of our AED. The project code is available at https://github.com/JiahuaDong/AED .
