Latent Dynamics · 2026-11-09

LEF-JEPA: Denoising-Trained Latent Energy Fields for Spatial Representation Learning

Zhiyuan Liu, Zhihao Xu, Sicheng Liu, Huanhuan Huang

arXiv:2611.08989PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

LEF-JEPA strengthens spatial structure in JEPA-style latent representations by learning a target-guided latent energy field and transferring this supervision to the visual encoder via a one-sided relative-energy constraint. The energy module is used only during pretraining (no inference overhead). An Ising-Hamiltonian realization is interfaced with a physical coherent Ising machine for efficient low-energy state search and supports fewer local updates/forward latency on hardware.

Why it matters

JEPA masked-position supervision can produce coarse latent regions with weak object boundaries. The authors propose a pretraining-only energy mechanism that learns local recovery directions from multi-level targets and then constrains encoder representations to occupy low-energy regions relative to a target reference, aiming to improve spatial structure for prediction and action-oriented planning without over-encoding appearance details.

Method

  • Augments masked latent prediction with target-guided conditional latent energy learning: a neural conditional energy transformer shapes local recovery directions using multi-level targets; energy module is used only during pretraining.
  • Transfers structural supervision to the visual encoder using a one-sided relative-energy objective that encourages encoder outputs to lie below target-energy reference, without backpropagating through the K-step recovery during encoder updates.
  • For efficient energy optimization, formulates a conditional Ising Hamiltonian and integrates a physical photonic coherent Ising machine (CIM) into the closed training loop; hardware-retrieved low-energy states are differentiably rescored via the coefficient network.
Abstract (from arXiv)

A joint-embedding predictive architecture (JEPA) that applies prediction supervision only at masked positions struggles to clearly capture object shapes and boundaries in its latent representations. We further observe that some existing methods improve spatial structure while also retaining rich variations in texture and color. However, preserving more low-level visual information does not necessarily make representations better suited to prediction and action. This raises a key question: how can we strengthen the spatial structure in latent representations that supports prediction and action without over-encoding these details? We introduce LEF-JEPA, which learns spatial structure through target-guided latent recovery and transfers this supervision to the visual encoder through a relative-energy constraint; the energy module is used only during pretraining and adds no inference overhead. To reduce the cost of energy optimization during pretraining, we further construct a conditional Ising Hamiltonian that allows low-energy-state search to be performed by specialized Ising solvers. On physical hardware, we validate the feasibility of achieving lower forward latency with fewer local updates. Under a unified CIFAR-10 pretraining and frozen-encoder evaluation protocol, both realizations outperform V-JEPA2.1 in image classification, foreground segmentation, and visual planning.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-09 · Indexed by Pub-AI: 2026-11-09 UTC

preprint

LEF-JEPA: Denoising-Trained Latent Energy Fields for Spatial Representation Learning

Zhiyuan Liu, Zhihao Xu, Sicheng Liu, Huanhuan Huang

First announced by arXiv: 2026-11-09 · Original record ↗

arxiv
2611.08989
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-09 · First detected: 2026-11-09 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.08989v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

A joint-embedding predictive architecture (JEPA) that applies prediction supervision only at masked positions struggles to clearly capture object shapes and boundaries in its latent representations. We further observe that some existing methods improve spatial structure while also retaining rich variations in texture and color. However, preserving more low-level visual information does not necessarily make representations better suited to prediction and action. This raises a key question: how can we strengthen the spatial structure in latent representations that supports prediction and action without over-encoding these details? We introduce LEF-JEPA, which learns spatial structure through target-guided latent recovery and transfers this supervision to the visual encoder through a relative-energy constraint; the energy module is used only during pretraining and adds no inference overhead. To reduce the cost of energy optimization during pretraining, we further construct a conditional Ising Hamiltonian that allows low-energy-state search to be performed by specialized Ising solvers. On physical hardware, we validate the feasibility of achieving lower forward latency with fewer local updates. Under a unified CIFAR-10 pretraining and frozen-encoder evaluation protocol, both realizations outperform V-JEPA2.1 in image classification, foreground segmentation, and visual planning.