Robotics · 2026-10-06
RoboIRS: Inference-Time Internal Representation Steering for Generalist Robot Policies
Jiuzhou Lei, Chang Liu, Dayou Li, Zhiyuan Zhang, Xiao Liang, Yu She, Zhiwen Fan, Minghui Zheng
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
RoboIRS is an inference-time internal representation steering method for frozen robot policies. It uses successful vs. failed rollouts to train linear classifiers, selects outcome-relevant intervention sites, and injects classifier-derived steering directions into transformer residual streams without updating policy parameters. On 15 LIBERO-PRO simulation tasks with a frozen π0.5 policy, RoboIRS raises average success rate from 44.4% to 66.2%. It also improves real-robot manipulation and transfers to Cosmos Policy.
Why it matters
The authors address performance degradation of VLA/WAM policies under out-of-distribution task variations by steering internal representations instead of retraining. Their results indicate that success-related internal computations can be identified from rollout outcomes and manipulated at inference time to improve success rates in both simulation and real-robot experiments, and to a different world-action model architecture.
Method
- Collect residual-stream representations from successful and failed rollouts for each transformer layer, denoising step, and action-token position; train linear classifiers to distinguish success-like vs. failure-like residuals.
- Select the action-token position per layer using held-out validation failure-detection AUC, then inject a scaled steering vector into the residual-stream intervention site during inference.
- Apply the method without updating policy parameters; sweep steering strength (α) and choose steering sign empirically.
Limitation
The proposed method may be less effective on tasks where successful and failed rollouts do not exhibit clear differences in their actions, making it difficult to label and train a classifier. The experiments are also limited to a small number of VLA and WAM policy architectures, and the method requires empirical selection of hyperparameters such as steering strength.
Abstract (from arXiv)
Vision-language-action (VLA) and world-action models (WAMs) often degrade under out-of-distribution task variations despite retaining partial task capability. To recover such capability, we propose RoboIRS, an inference-time internal representation steering method that uses successful and failed rollouts to train linear classifiers, select outcome-relevant intervention locations, and derive task-specific steering directions without updating policy parameters. On 15 simulation tasks with a frozen $\pi 0.5$ policy, RoboIRS improves the average success rate from 44.4% to 66.2%, outperforming alternative inference-time intervention baselines while adding little inference time. We further validate RoboIRS on real-robot manipulation using the same $\pi 0.5$ policy and demonstrate its applicability to a world-action model Cosmos Policy, where the average success rate improves from 35.4% to 55.4%. These results show that directly steering internal robot-policy representations can improve the performance of robot policies at inference time. Project website is available at https://rollingoat.github.io/roboirs/.
Related papers
- Memory Without Curation: Learning Multi-Stream Latent Memory for Robot Policies
- Dream to Remember: Continual Steering of Generalist Robot Policies with World Models
- What Should Robot Policies Learn to Predict? Rethinking Supervision Targets in Vision-Language-Action Learning
- La-JEPA: Compact Robot Policies with Latent World Modeling
- RoboFQE: Rollout-Free Evaluation of Visual Robot Policies from Offline Trajectories
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-03 · Indexed by Pub-AI: 2026-10-06 UTC
RoboIRS: Inference-Time Internal Representation Steering for Generalist Robot Policies
Preprint posted: 2026-10-03 · Original record ↗
- arxiv
- 2610.04681
Explicit source versions (1)
- v3 · 2026-10-08
- Preprint available
Source date: 2026-10-03 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
- Explicit source version detected · v3
Source date: 2026-10-08 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:id explicit version (official metadata)
Abstract · arxiv
Vision-language-action (VLA) and world-action models (WAMs) often degrade under out-of-distribution task variations despite retaining partial task capability. To recover such capability, we propose RoboIRS, an inference-time internal representation steering method that uses successful and failed rollouts to train linear classifiers, select outcome-relevant intervention locations, and derive task-specific steering directions without updating policy parameters. On 15 simulation tasks with a frozen $π0.5$ policy, RoboIRS improves the average success rate from 44.4% to 66.2%, outperforming alternative inference-time intervention baselines while adding little inference time. We further validate RoboIRS on real-robot manipulation using the same $π0.5$ policy and demonstrate its applicability to a world-action model Cosmos Policy, where the average success rate improves from 35.4% to 55.4%. These results show that directly steering internal robot-policy representations can improve the performance of robot policies at inference time. Project website is available at https://rollingoat.github.io/roboirs/.
