Latent Dynamics · 2026-10-26
ACWM-RM: A Reference-Free Reward Model from Complementary Mismatch Preferences
Shibo Hong, Siyu Ren, FengJiao Chen, Zhongyuan Peng, Wei Shi, Yixin Cao
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce AVM-Eval (67.2k pairs) to test whether inverse dynamics model (IDM) action-following rewards transfer from real to generated videos. They find real-video IDM accuracy does not reliably evaluate generated videos. They propose ACWM-RM, a reference-free reward model trained on AVM-Pref (506.3k pairs) to jointly assess visual quality (VQ) and action-following fidelity (AF) without ground-truth videos during reward computation.
Why it matters
Action-conditioned world model (ACWM) post-training with RL needs rewards that rank videos by both visual quality (VQ) and action-following fidelity (AF). The paper shows that IDM-based AF rewards can be unreliable on generated videos even when they are accurate on real videos, motivating reference-free joint VQ+AF reward modeling. ACWM-RM is designed to avoid ground-truth video references at reward time and to address the mismatch between real-video IDM training and generated-video evaluation.
Method
- Construct AVM-Eval (67.2k pairs) with action-mismatch pairs (AF on real videos) and video-mismatch pairs (AF on generated videos) to measure IDM reliability across domains.
- Build AVM-Pref (506.3k pairs): action-mismatch pairs for AF supervision and video-mismatch pairs for VQ supervision using global/local PSNR and LPIPS, exposing the model to generated-video distribution.
- Train ACWM-RM by first pretraining a bimanual action-conditioned predictor on real videos using a frozen V-JEPA 2.1 encoder, then jointly training with a patch-weighted score head on AVM-Pref to score VQ+AF without reference videos.
Limitation
The authors state that task-level balancing in AVM-Pref/AVM-Eval is limited by the availability of candidate pairs that satisfy all criteria, and cross-model pairing further restricts eligible candidates for some tasks.
Abstract (from arXiv)
Reinforcement learning has gained attention for improving action-conditioned world models (ACWMs), as it enables reward-driven optimization of visual quality (VQ) and action-following fidelity (AF) beyond flow-matching optimization. Realizing this potential requires reward signals that reliably rank videos by both VQ and AF. However, existing rewards capture only one dimension: metric-based rewards assess VQ but require ground-truth videos and do not directly assess AF, while inverse dynamics model (IDM) rewards assess AF without references but do not guarantee VQ. Moreover, IDMs, typically trained on real videos, may provide unreliable rewards for generated videos containing visual artifacts and physical inconsistencies. In this work, we introduce the Action and Video Mismatch Benchmark (AVM-Eval), comprising 67.2k pairs that evaluate how reliably IDMs assess AF on real and generated videos. We then propose ACWM-RM, a reward model that jointly assesses VQ and AF without ground-truth videos during reward computation. To train it, we first construct the Action and Video Mismatch Preference Dataset (AVM-Pref) with 506.3k pairs providing complementary supervision: action-mismatch pairs supervise AF, while video-mismatch pairs supervise VQ and expose the model to generated videos. We then pretrain an action-conditioned predictor on real videos to learn forward dynamics before jointly training it with a patch-weighted score head on AVM-Pref to transfer these dynamics to AF assessment on generated videos. Evaluation on AVM-Eval reveals the limited reliability of IDMs on generated videos, while downstream RL experiments show that ACWM-RM outperforms reward baselines in both criteria.
Related papers
- Disentangling Task-Relevant Latent Dynamics for Reward Modeling
- $R^2$-WAM: Repair-and-Reject Post-Training for World Action Models
- Cycle WAM: Coupling Action and Video through Cycle Consistency
- SWAM: Scaling World Action Models with Cross-Modal Alignment
- EVO-WAM: Evolving World Action Models through Video-Action Verification
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-23 · Indexed by Pub-AI: 2026-10-26 UTC
ACWM-RM: A Reference-Free Reward Model from Complementary Mismatch Preferences
Preprint posted: 2026-10-23 · Original record ↗
- arxiv
- 2610.32222
Explicit source versions (1)
- v1 · 2026-10-23
- Preprint available
Source date: 2026-10-23 · First detected: 2026-11-07 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Reinforcement learning has gained attention for improving action-conditioned world models (ACWMs), as it enables reward-driven optimization of visual quality (VQ) and action-following fidelity (AF) beyond flow-matching optimization. Realizing this potential requires reward signals that reliably rank videos by both VQ and AF. However, existing rewards capture only one dimension: metric-based rewards assess VQ but require ground-truth videos and do not directly assess AF, while inverse dynamics model (IDM) rewards assess AF without references but do not guarantee VQ. Moreover, IDMs, typically trained on real videos, may provide unreliable rewards for generated videos containing visual artifacts and physical inconsistencies. In this work, we introduce the Action and Video Mismatch Benchmark (AVM-Eval), comprising 67.2k pairs that evaluate how reliably IDMs assess AF on real and generated videos. We then propose ACWM-RM, a reward model that jointly assesses VQ and AF without ground-truth videos during reward computation. To train it, we first construct the Action and Video Mismatch Preference Dataset (AVM-Pref) with 506.3k pairs providing complementary supervision: action-mismatch pairs supervise AF, while video-mismatch pairs supervise VQ and expose the model to generated videos. We then pretrain an action-conditioned predictor on real videos to learn forward dynamics before jointly training it with a patch-weighted score head on AVM-Pref to transfer these dynamics to AF assessment on generated videos. Evaluation on AVM-Eval reveals the limited reliability of IDMs on generated videos, while downstream RL experiments show that ACWM-RM outperforms reward baselines in both criteria.
