Interleaved next-step embedding modeling for multimodal sequential imitation
Train a single Transformer agent to imitate expert behavior by autoregressively predicting interleaved next-step embeddings of observations (and associated rewards) and actions, using a causal mask. Modality-specific encoders/decoders handle different data types (e.g., images via a CNN, continuous vectors via a linear layer, discrete values via projection), and the training loss averages per-element losses computed according to each modality (e.g., MSE for continuous/image targets and cross-entropy for discrete targets).
Discovered 2026-11-10 · Sources checked 2026-11-10
Why it matters
This turns multi-task sequential decision-making into a unified next-step prediction problem that can span heterogeneous observation/action formats within one architecture, enabling a generalist approach rather than training separate models per environment or modality.
What changed
Compared to single-environment RL-style modeling, the approach uses (1) an interleaved sequence representation of observation/reward embeddings with action embeddings, (2) causal autoregressive prediction of shifted next embeddings, and (3) modality-specific embeddings plus per-modality loss computation aggregated across the sequence.
Where it helps
- Generalist imitation learning across multiple environments
- Unified multimodal policy learning from heterogeneous inputs (images, text, continuous/discrete)
- Training sequential decision agents with a single Transformer backbone
- Auxiliary prediction objectives for improving imitation learning stability
Evidence & reproduction
Reproduction not evaluated. A linked paper or repository does not establish independent reproduction.
Limitations & open questions
The supplied source describes the overall setup as behavioral cloning from expert trajectories, implying it cannot outperform the provided expert and does not directly leverage sub-optimal trajectories in the way offline RL methods might.
Sources
- Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent ↗
Hugging Face · primary · Published 2024-04-22 · Checked 2026-11-10
