Robotics · 2026-10-14
Disentangling Task-Relevant Latent Dynamics for Reward Modeling
Dayou Li, Jionghao Wang, Peihao Wang, Jiuzhou Lei, Hao Wang, Jichen Chai, Hezhen Hu, Minghui Zheng, Zhangyang Wang, Xin Li, Wenping Wang, Zhiwen Fan
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
DiTaR learns dense rewards from heterogeneous, action-free robot videos by modeling task-relevant latent dynamics rather than entangled visual variation. It uses an asymmetric two-stage latent representation to suppress task-irrelevant variation, then trains a history- and language-conditioned rectified-flow model and derives a directional temporal-difference reward from changes in compatibility. The authors report stronger visual RL on MetaWorld-v3 and ManiSkill and improved real-world offline RL via reward relabeling.
Why it matters
Video-based reward learning can be misled when learned rewards respond to nuisance visual changes instead of task-relevant execution. DiTaR targets this by separating task-relevant latent dynamics from irrelevant visual variation and producing directional dense feedback derived from temporal changes in dynamics compatibility.
Method
- Asymmetric two-stage task-relevant latent encoding on multi-frame video latents with KL-regularized bottlenecks: Stage 1 uses language semantics; Stage 2 freezes Stage 1 components and removes decoder language access so a second latent recovers task-relevant information from video.
- History- and language-conditioned rectified-flow model in the task-relevant latent space to model task dynamics from recent visual history and task instruction.
- Directional dense reward from temporal-difference changes in an endpoint residual-derived compatibility score (positive temporal difference indicates increasing compatibility).
Limitation
The paper excerpt does not state a specific limitation.
Abstract (from arXiv)
Reward design remains a major bottleneck for reinforcement learning in robotic manipulation. Handcrafted dense rewards can compensate for the limited guidance provided by sparse rewards, but require substantial task-specific reward engineering. Learning rewards from videos provides a scalable, data-driven alternative to manual reward design. However, existing video-based rewards are often computed in entangled visual spaces where task-relevant state changes are confounded with extraneous visual variation. As a result, the learned reward may respond to visual distractors instead of actual task execution. To address this limitation, we propose DiTaR (Disentangling Task-Relevant Latent Dynamics for Reward Modeling), a framework for learning dense rewards from heterogeneous videos. DiTaR learns a continuous task-relevant representation from multi-frame video latents through an asymmetric two-stage procedure that suppresses task-irrelevant visual variation. In the task-relevant latent space, we train a history- and language-conditioned rectified-flow model to capture task dynamics and derive a directional dense reward from temporal changes in dynamics compatibility. Together, these components enable reward modeling directly over task-relevant dynamics rather than generic visual variation. DiTaR substantially improves policy learning on MetaWorld-v3 and ManiSkill over video-based reward baselines, and transfers to real-world offline RL through reward relabeling. Reward analyses further show that DiTaR provides more reliable rewards.
Related papers
- CausalDreamer: Learning Predictive World Models with Latent Disentanglement
- Video2STL: Grounding VLM-Generated Temporal Specifications for Robot Learning
- OneLA: Learning Many Worlds from Latent Actions
- World Action Learning via Interaction-Centric Spectral Latent Guidance
- Sparse-TD-JEPA: Improving Zero-Shot Reinforcement Learning With Sparse Latent Task Representations
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-09 · Indexed by Pub-AI: 2026-10-14 UTC
Disentangling Task-Relevant Latent Dynamics for Reward Modeling
Preprint posted: 2026-10-09 · Original record ↗
- arxiv
- 2610.14159
Explicit source versions (1)
- v1 · 2026-10-09
- Preprint available
Source date: 2026-10-09 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Reward design remains a major bottleneck for reinforcement learning in robotic manipulation. Handcrafted dense rewards can compensate for the limited guidance provided by sparse rewards, but require substantial task-specific reward engineering. Learning rewards from videos provides a scalable, data-driven alternative to manual reward design. However, existing video-based rewards are often computed in entangled visual spaces where task-relevant state changes are confounded with extraneous visual variation. As a result, the learned reward may respond to visual distractors instead of actual task execution. To address this limitation, we propose DiTaR (Disentangling Task-Relevant Latent Dynamics for Reward Modeling), a framework for learning dense rewards from heterogeneous videos. DiTaR learns a continuous task-relevant representation from multi-frame video latents through an asymmetric two-stage procedure that suppresses task-irrelevant visual variation. In the task-relevant latent space, we train a history- and language-conditioned rectified-flow model to capture task dynamics and derive a directional dense reward from temporal changes in dynamics compatibility. Together, these components enable reward modeling directly over task-relevant dynamics rather than generic visual variation. DiTaR substantially improves policy learning on MetaWorld-v3 and ManiSkill over video-based reward baselines, and transfers to real-world offline RL through reward relabeling. Reward analyses further show that DiTaR provides more reliable rewards.