Latent Dynamics · 2026-11-03
Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel
Kunal Bhosikar, Nicole Meng, Charu Sharma, Yingjie Lao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors study test-time robustness of LeWorldModel (LeWM), a JEPA-style latent planner that minimizes squared Euclidean distance between predicted and goal embeddings. They find block pose dominates latent distance. A white-box ℓ∞ perturbation (ε=4/255) of a single input frame can steer the encoder to an attacker-chosen block pose in 73–95% of scenes while leaving encoded agent state largely unchanged, severely reducing planning success across benchmarks.
Why it matters
LeWM’s control objective routes entirely through encoder embeddings and a Euclidean latent distance to a goal image. The authors show that imperceptible, checkpoint-preserving pixel perturbations can spoof the latent goal state and derail planning, while model-side checks (SIGReg, prediction loss, latent “surprise”) may fail—especially for goal-image attacks. This highlights an input-to-latent mapping as a practical attack surface for JEPA planners.
Method
- Analyze LeWM’s latent metric on Push-T using counterfactual renders varying single state factors (agent position, block position, block angle), measuring latent shifts and distance saturation.
- Construct a white-box test-time ℓ∞ state-spoofing attack optimized via gradients through the public encoder so Eθ(o+δ) matches an attacker-chosen target embedding, then apply δ to goal images or live frames.
- Evaluate closed-loop MPC/CEM planning success across LeWM benchmarks, and test runtime “surprise” and SIGReg statistics as detection mechanisms.
Limitation
The authors state that evaluating these attacks faithfully requires on-manifold counterfactual targets, since naive off-manifold re-renders can fabricate and mask failure modes.
Abstract (from arXiv)
Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding $3\times$ further than an equivalent agent shift, and swapping the block pose alone accounts for $99\%$ of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box $\ell_\infty$ perturbation of just $\epsilon{=}4/255$ applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in $73$--$95\%$ of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from $87\%$ to $32\%$, whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At $\epsilon{=}8/255$, the attack bridges $82$--$97\%$ of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.
Related papers
- LeWAM: End-to-End World and Action Modeling with JEPAs
- Beyond One-Step Accuracy: State-Affine Latent Transition for Reliable Visual Planning
- Control-Geometry Straightening for Sampling-Based Latent Planning
- LRC-JEPA: Disentangling Dynamics and Residual Context for Efficient World Models
- Pretrained Features as Teachers, Not States: Compact Latent World Models for Visual Planning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-03 · Indexed by Pub-AI: 2026-11-03 UTC
Spoofing the Latent State: Imperceptible Goal-Image Attacks Derail Planning in LeWorldModel
First announced by arXiv: 2026-11-03 · Original record ↗
- arxiv
- 2611.00427
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-03 · First detected: 2026-11-03 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.00427v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
Joint-embedding predictive architectures (JEPAs) have recently emerged as general-purpose visual world models for control. A prominent example is LeWorldModel (LeWM), which trains end-to-end from pixels using an isotropic Gaussian anti-collapse regularizer (SIGReg) and plans by minimizing the squared Euclidean distance between predicted and goal embeddings. In this work, we investigate the test-time robustness of this latent planning interface. Analyzing the learned metric on Push-T, we first find that task-relevant object state heavily dominates latent distance: shifting the block by a single pixel displaces the embedding $3\times$ further than an equivalent agent shift, and swapping the block pose alone accounts for $99\%$ of the distance between scene pairs. However, because this pose information is distributed across a curved, high-dimensional manifold rather than a fixed linear subspace, the encoder cannot be trivially edited post-hoc to isolate or redirect it. We next demonstrate that this geometry is highly vulnerable at test time. A white-box $\ell_\infty$ perturbation of just $ε{=}4/255$ applied to a single input frame causes the encoder to register the block at an arbitrary, attacker-chosen pose in $73$--$95\%$ of scenes without disturbing the encoded agent state. Perturbing either the goal image or the live camera observation drops Push-T planning success from $87\%$ to $32\%$, whereas uniform random noise at twice the magnitude has negligible impact. This vulnerability persists across all four standard LeWM benchmarks (Push-T, Two-Room, Reacher, and OGBench-Cube). At $ε{=}8/255$, the attack bridges $82$--$97\%$ of the squared latent distance to the target embedding in every environment, inducing planning degradation proportional to each task's reliance on the spoofed entity.
