Latent Dynamics · 2026-10-22

DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation

Xiujian Liang, Liuyuan Na, Zehao Du, Jiacheng Xu, Weixi Song, Mingyang Sun, Xingjun Ma, Cewu Lu, Jianhua Sun

arXiv:2610.29246PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

DepState-JEPA, built on V-JEPA 2.1, learns a depth-grounded shared latent state (DepState) from RGB–depth video. Depth is used both as a complementary stream and to place both modalities in a common metric 3D frame. A shared ray-conditioned decoder predicts missing and future RGB/depth features under action-conditioned latent transitions, with additional robot-state and kinematic (FK) consistency losses for manipulation control.

Why it matters

The authors aim for predictive robot representations that connect scene geometry (where things are) with action-induced changes (what actions change). DepState-JEPA provides a shared, spatially queryable latent interface grounded in metric 3D depth, and it reuses the same visual decoding pathways after action-conditioned state transitions, while also enforcing joint–TCP consistency for control-relevant prediction.

Method

  • Dual-stream RGB–Depth video JEPA: independent encoders produce grounded token features; shared–private projections aggregate shared information into DepState while retaining modality-private decoding inputs.
  • Depth-grounded shared-state construction: visible metric depth and intrinsics place RGB and depth features in a shared metric 3D frame; ray-conditioned readouts predict local missing features without target depth and are reused after action-conditioned transitions.
  • Action-conditioned latent transition with control heads: a temporal Transformer advances DepState under action and proprioception; robot-state and object-motion readouts are supervised with FK regularization for joint–TCP consistency.

Limitation

The robot protocol uses a calibrated fixed external camera and embodiment-specific training; moving-camera prediction, zero-shot transfer across embodiments, and broader object and workspace changes are outside the demonstrated scope. Whole-image feature accuracy does not by itself establish action sensitivity in moving regions or accurate physical trajectories. A single initial frame may omit occluded state or motion history, and fixed initial private memory adequacy for large displacement and newly visible surfaces requires further study. FK is a soft regularizer; neither it nor a valid SE(3) parameterization guarantees correct contact dynamics or collision-free control.

Abstract (from arXiv)

Predictive representations for robot manipulation should connect scene geometry with the state changes induced by actions. Our intuition is that understanding what an action will change starts with understanding where things are: complementary RGB and depth observations ground a common scene representation from which to infer those changes. Guided by this intuition, we introduce \textbf{DepState-JEPA}, a dual-stream RGB--Depth joint-embedding predictive architecture that connects spatial understanding with action-conditioned prediction through a shared latent state, \textbf{DepState}. Built on V-JEPA~2.1, the framework uses depth both as a complementary observation stream and to ground both modalities in metric 3D space, aggregating shared information into DepState while retaining modality-private features for decoding. Ray-conditioned readouts predict local features without target depth; after an action-conditioned transition, the same decoders predict future RGB and depth features using the predicted state and fixed initial private features. Robot-state readouts provide recursive proprioceptive inputs; object-trajectory supervision and calibrated forward kinematics shape state learning through motion targets and joint--TCP consistency. Comparisons show stronger RGB-encoder transfer after RGB-D adaptation, feature completion, conditioned visual forecasting, and manipulation success. Within-model ablations identify contributions from depth content, metric coordinates, and future-state matching, with additional control gains from FK regularization.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-21 · Indexed by Pub-AI: 2026-10-22 UTC

preprint

DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation

Xiujian Liang, Liuyuan Na, Zehao Du, Jiacheng Xu, Weixi Song, Mingyang Sun, Xingjun Ma, Cewu Lu, Jianhua Sun

Preprint posted: 2026-10-21 · Original record ↗

arxiv
2610.29246
Explicit source versions (2)
  • v1 · 2026-10-21
  • v2 · 2026-10-23
  1. Preprint available

    Source date: 2026-10-21 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (2)
  2. Explicit source version detected · v2

    Source date: 2026-10-23 · First detected: 2026-11-01 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Predictive representations for robot manipulation should connect scene geometry with the state changes induced by actions. Our intuition is that understanding what an action will change starts with understanding where things are: complementary RGB and depth observations ground a common scene representation from which to infer those changes. Guided by this intuition, we introduce DepState-JEPA, a dual-stream RGB--Depth joint-embedding predictive architecture that connects spatial understanding with action-conditioned prediction through a shared latent state, DepState. Built on V-JEPA~2.1, the framework uses depth both as a complementary observation stream and to ground both modalities in metric 3D space, aggregating shared information into DepState while retaining modality-private features for decoding. Ray-conditioned readouts predict local features without target depth; after an action-conditioned transition, the same decoders predict future RGB and depth features using the predicted state and fixed initial private features. Robot-state readouts provide recursive proprioceptive inputs; object-trajectory supervision and calibrated forward kinematics shape state learning through motion targets and joint--TCP consistency. Comparisons show stronger RGB-encoder transfer after RGB-D adaptation, feature completion, conditioned visual forecasting, and manipulation success. Within-model ablations identify contributions from depth content, metric coordinates, and future-state matching, with additional control gains from FK regularization. The project page is available at https://shawnleung-sii.github.io/depstate-jepa/.