Latent Dynamics · 2026-10-14

Vision-Predictor-Action: Towards Efficient Embodied AI via Neuro-Symbolic JEPA Predictors

Yeonseok Lee

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

VPA replaces autoregressive token-based action decoding with a lightweight latent control loop: a vision encoder plus frozen text encoder condition a neuro-symbolic primitive selector, a JEPA predictor forecasts next latent states, and a flow-matching solver generates an entire action chunk in a few integration steps. On LIBERO-10, the authors report 40.0% success in 200 episodes (2 cameras) with low-latency decision steps on an Apple M4.

Why it matters

The authors target the latency bottleneck of token-by-token VLA action generation by making the per-decision sequential network depth independent of action horizon (K+3 steps vs horizon-dependent sequential decoding). They also analyze anti-collapse regularization and latent thresholding conditions, and provide an open-source PyTorch implementation with released checkpoints.

Method

Shared vision encoder (non-generative) and frozen text encoder embed observations, goal images, and instructions; neuro-symbolic primitive selector chooses a discrete skill from a fixed dictionary.
JEPA predictor forecasts the next latent state and an uncertainty scalar; regularized against representation collapse using VICReg.
Flow-matching solver generates a continuous action chunk jointly over H steps using K integration steps; execute an uncertainty-adaptive prefix.

Limitation

The authors report that the repertoire of high-level behaviors is bounded by a fixed primitive dictionary, zero-shot synthesis of skills outside that dictionary is not addressed, and the empirical results come from a single training run per configuration with 20 trials per task (not a controlled comparison with baselines under the same protocol).

Abstract (from arXiv)

Vision-Language-Action (VLA) models commonly generate robot actions token by token through Large Language Model (LLM) decoders, causing inference latency to grow with action sequence length and limiting closed-loop control frequency. We present Goal-Conditioned Vision-Predictor-Action (VPA), which replaces autoregressive decoders with a lightweight latent control loop. A shared vision encoder and a frozen text encoder embed observations, goal images, and instructions; a neuro-symbolic selector chooses a discrete skill; a Joint-Embedding Predictive Architecture (JEPA) predictor forecasts the next latent state; and a flow-matching solver generates an entire action chunk in a few integration steps, so that sequential network evaluations per decision step do not depend on the action horizon. We analyze inference complexity, regularization against representation collapse, and conditions under which latent sub-goal thresholding limits error accumulation across task phases. As an initial proof of concept, we evaluated VPA on the LIBERO-10 suite. The trainable model has under 20M parameters (checkpoint under 75 MB, excluding a frozen 63.4M-parameter CLIP text encoder); a decision step of a single-camera reference configuration takes 10-14 ms on an Apple M4; and trained only on LIBERO-10 demonstrations in under three hours on a single GPU, a two-camera policy succeeded in 40.0% of 200 evaluation episodes. These preliminary results from a single training run suggest the approach is workable. Code and weights are available at https://github.com/ylee-sling/Vision-Predictor-Action and https://huggingface.co/ylee-sling/vpa-libero10, and archived at https://doi.org/10.5281/zenodo.22960920.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-11 · Indexed by Pub-AI: 2026-10-14 UTC

preprint