Robotics · 2026-10-14
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
Wenzhan Li, Yiran Qin, Heng Zhou, Huirui Wang, Yulan Guo, Ruimao Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose MERL, a trust-calibrated VLA policy post-training method that couples online simulator evolution (a visual world model plus reward proxy head) with two-level trust control. Stage-level trust schedules the imagined-data ratio and rollout horizon, while chunk-level trust selects and weights imagined replay. On perturbed LIBERO-PRO, MERL attains 79.7% average success and improves high-threshold learning speed by 10.9%.
Why it matters
MERL targets a reliability–efficiency tension in model-based VLA RL post-training: simulators improve over time, but imagined rollouts vary in reliability. The proposed trust interface audits imagined experience at both stage and chunk levels so it can be used safely and efficiently for policy optimization as the simulator evolves.
Method
- Online-evolving simulator for VLA post-training: a world model (visual prediction) plus a reward proxy head, both updated from grounded trajectories.
- Two-level trust calibration: stage-level trust maps simulator mismatch to imagined-data ratio and rollout horizon; chunk-level trust maps local residuals to replay probability and optimization weight.
- Residual predictor for recursive imagination without oracle future: estimates observation and proxy reliability using predicted latents and rollout depth; frozen during imagination and policy optimization.
Limitation
This study covers fixed-data offline adaptation and closed-loop physical deployment on two tabletop tasks. It does not evaluate online physical RL or additional policy-driven data collection. Remaining failures include fixed-camera depth ambiguity, gripper slippage, camera–robot calibration drift, and contact dynamics that short-horizon RGB prediction may not capture. The limited trial count and overlapping intervals constrain conclusions about the proxy-specific gain.
Abstract (from arXiv)
Vision-Language-Action (VLA) models are often pretrained via imitation learning, making reinforcement learning (RL) post-training important when deployment moves beyond demonstration coverage. Existing refinement paradigms expose a reliability--efficiency tension: model-free refinement avoids simulator bias but is interaction-intensive, whereas model-based refinement can exploit unreliable imagination even as its simulator adapts. We propose Model-Evolving Reinforcement Learning (MERL), a trust-calibrated framework that couples policy optimization with an online-updated simulator comprising a world model and a reward proxy head. Across stages, simulator reliability controls the imagined-data ratio and rollout horizon; within rollouts, chunk-level trust controls replay selection and optimization weight. For recursive imagination without exact grounded continuations, a residual predictor estimates trust without querying future ground truth. On perturbed LIBERO-PRO tasks, MERL reaches 79.7% average success, improves high-threshold learning speed by 10.9% over the matched model-free control, and outperforms same-protocol external baselines across three training seeds. Fixed-data adaptation from 50 demonstrations per task achieves 80.0% average success on two physical-robot tasks.
Related papers
- DriftOPD: Sequence-Level Reverse-KL Distillation for One-Step VLA Policies
- WoVR+: World Model as Reliable Simulator for Versatile VLA Policy Optimization with RL
- DRILL: Training World Models to Improve Policies, Not Predict Pixels
- CoDA: Co-Evolution of Data Engines and Action Policies
- Beyond Prediction: Steering VLM Agents with Retrospective World Modeling
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-12 · Indexed by Pub-AI: 2026-10-14 UTC
Trust-Calibrated VLA Policy Post-Training with Evolving Imagination
Preprint posted: 2026-10-12 · Original record ↗
- arxiv
- 2610.16517
Explicit source versions (1)
- v1 · 2026-10-12
- Preprint available
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Vision-Language-Action (VLA) models are often pretrained via imitation learning, making reinforcement learning (RL) post-training important when deployment moves beyond demonstration coverage. Existing refinement paradigms expose a reliability--efficiency tension: model-free refinement avoids simulator bias but is interaction-intensive, whereas model-based refinement can exploit unreliable imagination even as its simulator adapts. We propose Model-Evolving Reinforcement Learning (MERL), a trust-calibrated framework that couples policy optimization with an online-updated simulator comprising a world model and a reward proxy head. Across stages, simulator reliability controls the imagined-data ratio and rollout horizon; within rollouts, chunk-level trust controls replay selection and optimization weight. For recursive imagination without exact grounded continuations, a residual predictor estimates trust without querying future ground truth. On perturbed LIBERO-PRO tasks, MERL reaches 79.7% average success, improves high-threshold learning speed by 10.9% over the matched model-free control, and outperforms same-protocol external baselines across three training seeds. Fixed-data adaptation from 50 demonstrations per task achieves 80.0% average success on two physical-robot tasks.