Video World Models · 2026-10-21
RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation
Siyuan Ma, Yu Qi, Cheng Ye, Ruotian Peng, Xinyi Xu, Zitiantao Lin, Zedong Cai, Yijian Huang, Weiyang Liu, Siyuan Wang, Yandong Wen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
RoboJudge benchmarks and trains multimodal LLM judges for embodied video generation. RoboJudge-Bench uses 800 human-annotated videos scored on Physical Adherence (physical realism/consistency) and Instruction Alignment (agent/object/goal correctness). The authors fine-tune a 9B judge (RoboJudge-9B) on RoboJudge-Train (12K videos, 25K human-verified scores), improving agreement with humans and enabling downstream video generator ranking.
Why it matters
Reliable evaluation of embodied video rollouts needs agreement with human judgments on both physical validity and task instruction following. The paper systematically measures where existing MLLM judges and non-MLLM metrics fail, then trains a dedicated judge that improves correlation with human ratings and transfers to external out-of-distribution benchmarks.
Method
- Introduce RoboJudge-Bench: 800 generated embodied-manipulation videos from 8 source corpora and 10 video generators, rated by humans along Physical Adherence (PA) and Instruction Alignment (IA).
- Construct RoboJudge-Train: 12,351 training videos with 25K human-verified PA/IA scores and model-assisted rationales; fine-tune a 9B MLLM from Qwen3.5-9B with SFT then GRPO.
- Use RoboJudge-9B as a downstream evaluator to rank video generation models using the same PA and IA rubric.
Abstract (from arXiv)
Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.
Related papers
- PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements
- UniWALM: Unified World Action Language Model for Embodied Robot Control
- Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
- DreamTrue: Action-Faithful Robot World Model with Counterfactual Post-Training
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-20 · Indexed by Pub-AI: 2026-10-21 UTC
RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation
Preprint posted: 2026-10-20 · Original record ↗
- arxiv
- 2610.27782
Explicit source versions (1)
- v1 · 2026-10-20
- Preprint available
Source date: 2026-10-20 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · OAI arXiv created; not header datestamp (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video generation models are promising foundations for embodied world models, but only if their generated rollouts are physically valid and faithful to task instructions. Multimodal large language models (MLLMs) are increasingly used to judge these properties, yet their agreement with human judgments on embodied videos has not been systematically evaluated. To address this gap, we introduce RoboJudge-Bench, a benchmark for evaluating MLLMs as judges of embodied video generation. The benchmark measures two fundamental dimensions, Physical Adherence and Instruction Alignment, across 800 human-annotated videos from 8 source corpora and 10 video generators, spanning grippers, human hands, and dexterous hands. Experiments reveal uneven alignment between existing judges and human ratings: general-purpose MLLMs show lower agreement on physical adherence, while specialized video judges designed for general-domain physical reasoning do not consistently transfer to embodied scenarios. Motivated by these findings, we construct RoboJudge-Train, comprising 12K videos annotated with 25K human-verified scores and model-assisted rationales for Physical Adherence and Instruction Alignment. We fine-tune a 9B MLLM on this dataset with supervised learning followed by reinforcement learning, yielding RoboJudge-9B. Among the model judges evaluated on RoboJudge-Bench, RoboJudge-9B achieves the highest overall agreement, reaching a Pearson correlation of 0.719 with a statistically significant 0.089 gain over Gemini-3.7-Flash, the best-performing proprietary baseline in our evaluation. Training on RoboJudge-Train also yields gains on external benchmarks, including VideoPhy2 and Physion-Eval. We further use RoboJudge-9B as a downstream evaluator to benchmark video generation models according to Physical Adherence and Instruction Alignment.