Latent Dynamics · 2026-11-03

Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel

Kunal Bhosikar, Nicole Meng, Charu Sharma, Yingjie Lao

arXiv:2611.00427PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors study test-time robustness of LeWorldModel (LeWM), a JEPA-style latent planner that minimizes squared Euclidean distance between predicted and goal embeddings. They find block pose dominates latent distance. A white-box ℓ∞ perturbation (ε=4/255) of a single input frame can steer the encoder to an attacker-chosen block pose in 73–95% of scenes while leaving encoded agent state largely unchanged, severely reducing planning success across benchmarks.

Why it matters

LeWM’s control objective routes entirely through encoder embeddings and a Euclidean latent distance to a goal image. The authors show that imperceptible, checkpoint-preserving pixel perturbations can spoof the latent goal state and derail planning, while model-side checks (SIGReg, prediction loss, latent “surprise”) may fail—especially for goal-image attacks. This highlights an input-to-latent mapping as a practical attack surface for JEPA planners.

Method

  • Analyze LeWM’s latent metric on Push-T using counterfactual renders varying single state factors (agent position, block position, block angle), measuring latent shifts and distance saturation.
  • Construct a white-box test-time ℓ∞ state-spoofing attack optimized via gradients through the public encoder so Eθ(o+δ) matches an attacker-chosen target embedding, then apply δ to goal images or live frames.
  • Evaluate closed-loop MPC/CEM planning success across LeWM benchmarks, and test runtime “surprise” and SIGReg statistics as detection mechanisms.

Limitation

The authors state that evaluating these attacks faithfully requires on-manifold counterfactual targets, since naive off-manifold re-renders can fabricate and mask failure modes.

Abstract (from arXiv)

Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding $3\times$ further than an equivalent agent shift, and swapping the block pose alone accounts for $99\%$ of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box $\ell_\infty$ perturbation of just $\epsilon{=}4/255$ applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in $73$--$95\%$ of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from $87\%$ to $32\%$, whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At $\epsilon{=}8/255$, the attack bridges $82$--$97\%$ of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC

preprint

Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel

Kunal Bhosikar, Nicole Meng, Charu Sharma, Yingjie Lao

First announced by arXiv: 2026-11-03 · Original record ↗

arxiv
2611.00427
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00427v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding $3\times$ further than an equivalent agent shift, and swapping the block pose alone accounts for $99\%$ of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box $\ell_\infty$ perturbation of just $ε{=}4/255$ applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in $73$--$95\%$ of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from $87\%$ to $32\%$, whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At $ε{=}8/255$, the attack bridges $82$--$97\%$ of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.