Driving · 2026-10-28

PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

Hanqing Wang, Zhaotian Cai, Ruichuan An, Bohan Zeng, Tao Zhang, Yilin Chen, Xu Jiang, Kaiquan Chen, Chongyang Wu, Haoran Li, Fan Yang, Mingan Lin, Wentao Zhang, Zenan Zhou, Youzheng Wu, Xiaodong He

arXiv:2610.37381PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

PDRVLA is a vision-language-action model that adds a lightweight Physical Dynamics Module to extract physical tokens from intermediate vision-language features. The physical module is trained to predict future RGB, depth, and optical-flow features, with auxiliary supervision from a frozen pretrained video world model. The learned physical tokens condition action generation without online future-video generation. Trained on publicly available data expanded via human-to-robot synthesis and annotations.

Why it matters

The authors address high latency in video world models by avoiding online future video generation at control time. By learning compact physical tokens from intermediate vision-language features and supervising them with future-feature prediction plus frozen video world-model assistance, PDRVLA aims to provide action-relevant physical dynamics for VLA policies.

Method

  • Physical Dynamics Module extracts a fixed set of physical tokens from intermediate VLM features; action expert uses these tokens plus final-layer VLM features to generate actions.
  • Direct prediction learns future RGB, depth, and optical-flow latent features as targets; auxiliary supervision uses a frozen pretrained video world model conditioned on physical tokens.
  • Training uses publicly available data expanded through egocentric human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation; deployment uses physical tokens without running the video generator.

Limitation

The paper text provided does not state a limitation of the approach.

Abstract (from arXiv)

Vision--Language--Action (VLA) models generate robot actions from visual observations and language instructions, but their understanding of physical dynamics remains limited. Video world models can capture scene dynamics, yet generating future videos introduces high latency into the control loop. We introduce PDRVLA, whose lightweight Physical Dynamics Module extracts physical tokens from vision--language features to condition action generation without online video generation. The module learns through direct prediction of future RGB, depth, and optical-flow features, with auxiliary supervision through a frozen pretrained video world model. Training relies entirely on publicly available data, expanded through human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation. Across four simulation benchmarks, PDRVLA achieves success rates of 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin~2.0, and 81.5% on SimplerEnv. Ablations show improved future prediction with world-model supervision and evaluate how subtask conditioning, future object-pose supervision, and ECoT training affect action success.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-27 · Indexed by Pub-AI: 2026-10-28 UTC

preprint

PDRVLA: A Vision-Language-Action Model with Physical Dynamics Representations

Hanqing Wang, Zhaotian Cai, Ruichuan An, Bohan Zeng, Tao Zhang, Yilin Chen, Xu Jiang, Kaiquan Chen, Chongyang Wu, Haoran Li, Fan Yang, Mingan Lin, Wentao Zhang, Zenan Zhou, Youzheng Wu, Xiaodong He

Preprint posted: 2026-10-27 · Original record ↗

arxiv
2610.37381
Explicit source versions (1)
  • v1 · 2026-10-27
  1. Preprint available

    Source date: 2026-10-27 · First detected: 2026-10-28 UTC · Historical baseline

    Source evidence (3)
Abstract · arxiv

Vision--Language--Action (VLA) models generate robot actions from visual observations and language instructions, but their understanding of physical dynamics remains limited. Video world models can capture scene dynamics, yet generating future videos introduces high latency into the control loop. We introduce PDRVLA, whose lightweight Physical Dynamics Module extracts physical tokens from vision--language features to condition action generation without online video generation. The module learns through direct prediction of future RGB, depth, and optical-flow features, with auxiliary supervision through a frozen pretrained video world model. Training relies entirely on publicly available data, expanded through human-to-robot synthesis, subtask annotation, and embodied chain-of-thought (ECoT) annotation. Across four simulation benchmarks, PDRVLA achieves success rates of 99.3% on LIBERO, 89.9% on LIBERO-Plus, 93.6% on RoboTwin~2.0, and 81.5% on SimplerEnv. Ablations show improved future prediction with world-model supervision and evaluate how subtask conditioning, future object-pose supervision, and ECoT training affect action success.