Robotics · 2026-10-06

RoboIRS: Inference-Time Internal Representation Steering for Generalist Robot Policies

Jiuzhou Lei, Chang Liu, Dayou Li, Zhiyuan Zhang, Xiao Liang, Yu She, Zhiwen Fan, Minghui Zheng

arXiv:2610.04681PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

RoboIRS is an inference-time internal representation steering method for frozen robot policies. It uses successful vs. failed rollouts to train linear classifiers, selects outcome-relevant intervention sites, and injects classifier-derived steering directions into transformer residual streams without updating policy parameters. On 15 LIBERO-PRO simulation tasks with a frozen π0.5 policy, RoboIRS raises average success rate from 44.4% to 66.2%. It also improves real-robot manipulation and transfers to Cosmos Policy.

Why it matters

The authors address performance degradation of VLA/WAM policies under out-of-distribution task variations by steering internal representations instead of retraining. Their results indicate that success-related internal computations can be identified from rollout outcomes and manipulated at inference time to improve success rates in both simulation and real-robot experiments, and to a different world-action model architecture.

Method

  • Collect residual-stream representations from successful and failed rollouts for each transformer layer, denoising step, and action-token position; train linear classifiers to distinguish success-like vs. failure-like residuals.
  • Select the action-token position per layer using held-out validation failure-detection AUC, then inject a scaled steering vector into the residual-stream intervention site during inference.
  • Apply the method without updating policy parameters; sweep steering strength (α) and choose steering sign empirically.

Limitation

The proposed method may be less effective on tasks where successful and failed rollouts do not exhibit clear differences in their actions, making it difficult to label and train a classifier. The experiments are also limited to a small number of VLA and WAM policy architectures, and the method requires empirical selection of hyperparameters such as steering strength.

Abstract (from arXiv)

Vision-language-action (VLA) and world-action models (WAMs) often degrade under out-of-distribution task variations despite retaining partial task capability. To recover such capability, we propose RoboIRS, an inference-time internal representation steering method that uses successful and failed rollouts to train linear classifiers, select outcome-relevant intervention locations, and derive task-specific steering directions without updating policy parameters. On 15 simulation tasks with a frozen $\pi 0.5$ policy, RoboIRS improves the average success rate from 44.4% to 66.2%, outperforming alternative inference-time intervention baselines while adding little inference time. We further validate RoboIRS on real-robot manipulation using the same $\pi 0.5$ policy and demonstrate its applicability to a world-action model Cosmos Policy, where the average success rate improves from 35.4% to 55.4%. These results show that directly steering internal robot-policy representations can improve the performance of robot policies at inference time. Project website is available at https://rollingoat.github.io/roboirs/.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC

preprint

RoboIRS: Inference-Time Internal Representation Steering for Generalist Robot Policies

Jiuzhou Lei, Chang Liu, Dayou Li, Zhiyuan Zhang, Xiao Liang, Yu She, Zhiwen Fan, Minghui Zheng

Preprint posted: 2026-10-03 · Original record ↗

arxiv
2610.04681
Explicit source versions (1)
  • v3 · 2026-10-08
  1. Preprint available

    Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
  2. Explicit source version detected · v3

    Source date: 2026-10-08 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Vision-language-action (VLA) and world-action models (WAMs) often degrade under out-of-distribution task variations despite retaining partial task capability. To recover such capability, we propose RoboIRS, an inference-time internal representation steering method that uses successful and failed rollouts to train linear classifiers, select outcome-relevant intervention locations, and derive task-specific steering directions without updating policy parameters. On 15 simulation tasks with a frozen $π0.5$ policy, RoboIRS improves the average success rate from 44.4% to 66.2%, outperforming alternative inference-time intervention baselines while adding little inference time. We further validate RoboIRS on real-robot manipulation using the same $π0.5$ policy and demonstrate its applicability to a world-action model Cosmos Policy, where the average success rate improves from 35.4% to 55.4%. These results show that directly steering internal robot-policy representations can improve the performance of robot policies at inference time. Project website is available at https://rollingoat.github.io/roboirs/.