JEPA · 2026-10-30
JEDI: Joint-Embedding Demonstration Compression for In-Context Imitation Learning
Jiazhao Liang, Zhewen He, Shuaihang Yuan, Hao Huang, Yi Fang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
JEDI compresses in-context imitation demonstrations into k latent slots trained to predict the query episode’s next-stage keyframe in a frozen video-JEPA (V-JEPA) space. The predictor selects which cached demonstrations to place in context. Post-training enables conditioning on human-hand demonstrations via keypoint tokens. Experiments on LIBERO and real robots show improved many-shot context performance and better success/demonstration following than action-only compression.
Why it matters
ICIL is limited by prompt/context length when demonstrations are long multimodal trajectories. JEDI provides a demonstration compression objective that preserves execution-relevant stage details via query-future prediction in a frozen video-JEPA space, and uses predictive scoring to choose demonstrations. The paper also evaluates context usage with paired-context swaps to show action-only compression loses stage-dependent information that JEDI preserves.
Method
- Compress each demonstration independently into k latent slots via a perceiver compressor; cache the resulting slots in a demonstration bank.
- Train slots with a demonstration-conditioned predictive objective: predict the query episode’s next-stage keyframe latent in a frozen video-JEPA embedding space, with task/process counterfactual negatives.
- At test time, rank/select demonstrations from the bank by prediction error (pseudo-queries) and run a flow-matching policy that attends to the selected cached slots; optionally post-train for human-hand keypoint conditioning.
Limitation
The support set must be drawn from a demonstration bank, and the approach relies on the bank’s known futures at test time for predictive selection (the future of the query is unknown).
Abstract (from arXiv)
In-context imitation learning (ICIL) adapts robot policies through test-time demonstrations, but long multimodal trajectories constrain how many examples fit in context. We ask what a compressed demonstration should preserve. We propose JEDI, which compresses demonstrations into latent slots trained to predict the query episode's next-stage keyframe in a frozen video-JEPA space. Task and process counterfactuals help the slots preserve execution details, while the predictor selects demonstrations from a bank. Post-training with keypoint tokens enables human-hand conditioning on robots. Experiments on LIBERO and real robots show that JEDI supports many-shot context; LIBERO objective controls show improved success and demonstration following over action-only compression. These findings support query-future prediction as a training objective for preserving execution-relevant information in compressed demonstrations.
Related papers
- ELA: End-to-End Latent Action Policy Learning from Video Demonstrations
- Learning Transferable World-Action Models from Task-Paired Human Videos
- RynnWorld-Latent: A Cross-Embodiment World Model Conditioned on Universal Latent Actions
- World In-Context Learning for Agentic Robot Manipulation
- Beyond Successful Robot Demonstrations: Learning from Human Interactions and Modeling Failure Dynamics
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC
JEDI: Joint-Embedding Demonstration Compression for In-Context Imitation Learning
Preprint posted: 2026-10-29 · Original record ↗
- arxiv
- 2610.40356
Explicit source versions (1)
- v1 · 2026-10-29
- Preprint available
Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40356v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
In-context imitation learning (ICIL) adapts robot policies through test-time demonstrations, but long multimodal trajectories constrain how many examples fit in context. We ask what a compressed demonstration should preserve. We propose JEDI, which compresses demonstrations into latent slots trained to predict the query episode's next-stage keyframe in a frozen video-JEPA space. Task and process counterfactuals help the slots preserve execution details, while the predictor selects demonstrations from a bank. Post-training with keypoint tokens enables human-hand conditioning on robots. Experiments on LIBERO and real robots show that JEDI supports many-shot context; LIBERO objective controls show improved success and demonstration following over action-only compression. These findings support query-future prediction as a training objective for preserving execution-relevant information in compressed demonstrations.
