JEPA · 2026-10-30

JEDI: Joint-Embedding Demonstration Compression for In-Context Imitation Learning

Jiazhao Liang, Zhewen He, Shuaihang Yuan, Hao Huang, Yi Fang

arXiv:2610.40356PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

JEDI compresses in-context imitation demonstrations into k latent slots trained to predict the query episode’s next-stage keyframe in a frozen video-JEPA (V-JEPA) space. The predictor selects which cached demonstrations to place in context. Post-training enables conditioning on human-hand demonstrations via keypoint tokens. Experiments on LIBERO and real robots show improved many-shot context performance and better success/demonstration following than action-only compression.

Why it matters

ICIL is limited by prompt/context length when demonstrations are long multimodal trajectories. JEDI provides a demonstration compression objective that preserves execution-relevant stage details via query-future prediction in a frozen video-JEPA space, and uses predictive scoring to choose demonstrations. The paper also evaluates context usage with paired-context swaps to show action-only compression loses stage-dependent information that JEDI preserves.

Method

  • Compress each demonstration independently into k latent slots via a perceiver compressor; cache the resulting slots in a demonstration bank.
  • Train slots with a demonstration-conditioned predictive objective: predict the query episode’s next-stage keyframe latent in a frozen video-JEPA embedding space, with task/process counterfactual negatives.
  • At test time, rank/select demonstrations from the bank by prediction error (pseudo-queries) and run a flow-matching policy that attends to the selected cached slots; optionally post-train for human-hand keypoint conditioning.

Limitation

The support set must be drawn from a demonstration bank, and the approach relies on the bank’s known futures at test time for predictive selection (the future of the query is unknown).

Abstract (from arXiv)

In-context imitation learning (ICIL) adapts robot policies through test-time demonstrations, but long multimodal trajectories constrain how many examples fit in context. We ask what a compressed demonstration should preserve. We propose JEDI, which compresses demonstrations into latent slots trained to predict the query episode's next-stage keyframe in a frozen video-JEPA space. Task and process counterfactuals help the slots preserve execution details, while the predictor selects demonstrations from a bank. Post-training with keypoint tokens enables human-hand conditioning on robots. Experiments on LIBERO and real robots show that JEDI supports many-shot context; LIBERO objective controls show improved success and demonstration following over action-only compression. These findings support query-future prediction as a training objective for preserving execution-relevant information in compressed demonstrations.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC

preprint

JEDI: Joint-Embedding Demonstration Compression for In-Context Imitation Learning

Jiazhao Liang, Zhewen He, Shuaihang Yuan, Hao Huang, Yi Fang

Preprint posted: 2026-10-29 · Original record ↗

arxiv
2610.40356
Explicit source versions (1)
  • v1 · 2026-10-29
  1. Preprint available

    Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

In-context imitation learning (ICIL) adapts robot policies through test-time demonstrations, but long multimodal trajectories constrain how many examples fit in context. We ask what a compressed demonstration should preserve. We propose JEDI, which compresses demonstrations into latent slots trained to predict the query episode's next-stage keyframe in a frozen video-JEPA space. Task and process counterfactuals help the slots preserve execution details, while the predictor selects demonstrations from a bank. Post-training with keypoint tokens enables human-hand conditioning on robots. Experiments on LIBERO and real robots show that JEDI supports many-shot context; LIBERO objective controls show improved success and demonstration following over action-only compression. These findings support query-future prediction as a training objective for preserving execution-relevant information in compressed demonstrations.