Driving · 2026-11-13
PhysCapBench: A Cross-Domain Benchmark for Physically Grounded Video Captioning
Haofeng Wang, Pengcheng Wen, Ruiqi Li, Yuhao Zhang, Tiange Zhang, Boren Zheng, Xuanyi Liu, Siyu Tao, Danni Yu, Gan Lin, Haocheng Tang, Tianzizhe Guo, Mingyuan Su, Boyuan Chen, Zhongyuan Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
PhysCapBench is a cross-domain benchmark for physically grounded video captioning across third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates captions using seven physical dimensions. The authors report that physical captions reduce reconstruction LPIPS by 8.6–9.5% (third-person) and 26.9–30.9% (egocentric) vs metadata captions, and instruction enrichment yields higher PhysCapBench scores and policy success (e.g., +7.5 percentage points for π0.5 on a RoboTwin 2.0 subtask).
Why it matters
The paper addresses a data challenge in Physical AI pretraining: semantically correct captions can omit interactions and state transitions that define physical processes. PhysCapBench makes physical content of language supervision inspectable and comparable across domains using human-verified references and dimension-wise evaluation. The authors also test downstream utility via video reconstruction under physical-caption conditioning and manipulation-policy fine-tuning with enriched physical captions as instructions.
Method
- Introduce PhysCapBench: a unified physical-caption benchmark spanning third-person scenes, egocentric manipulation, and handheld-gripper demonstrations with human-verified annotations for 1,500 sequences.
- Use an evidence-grounded physical annotation pipeline producing structured physical representations and score captions along seven physical evaluation dimensions.
- Evaluate downstream utility via video reconstruction (first-frame-conditioned generation using physical vs metadata captions) and manipulation-policy fine-tuning using enriched physical captions as instructions.
Abstract (from arXiv)
Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution. A foundational data challenge is to measure and preserve this physical information as training supervision. In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. Rich physical experience in video does not automatically translate into sufficient language supervision. This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.
Related papers
- RoboJudge: Multimodal Language Models as Judges for Embodied Video Generation
- PVBench: Quantifying Physical Plausibility in Video Generation with Object-Level Physical Measurements
- Physis-Lang: Self-Evolving Language as a Physical Representation for Video World Model
- PhysTrace: Interpretable Temporal Evaluation of Physical Consistency in Video Generative Models
- SWAM: Scaling World Action Models with Cross-Modal Alignment
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-12 · Indexed by Pub-AI: 2026-11-13 UTC
PhysCapBench: A Cross-Domain Benchmark for Physically Grounded Video Captioning
Preprint posted: 2026-11-12 · Original record ↗
- arxiv
- 2611.16424
Explicit source versions (1)
- v1 · 2026-11-12
- Preprint available
Source date: 2026-11-12 · First detected: 2026-11-13 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.16424v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Physical AI aims to learn transferable physical knowledge from real-world experience to understand, predict, and act in the physical world. Internet videos and embodied demonstrations provide complementary observations of motion, interaction, and state evolution. A foundational data challenge is to measure and preserve this physical information as training supervision. In video-language pretraining, captions express observed processes in language, yet a semantically correct description can omit the interactions and transitions that define those processes. Rich physical experience in video does not automatically translate into sufficient language supervision. This gap calls for a yardstick to assess annotation quality, compare captioning strategies, and guide training-data refinement. Motivated by our data construction for world-model pretraining, we introduce PhysCapBench, to our knowledge the first unified physical-caption benchmark spanning all three domains: third-person scenes, egocentric manipulation, and handheld-gripper demonstrations. It provides human-verified annotations for 1,500 video sequences and evaluates accuracy and coverage through seven physical dimensions. We examine downstream utility through video reconstruction and manipulation-policy fine-tuning. Physical captions reduce reconstruction LPIPS by 8.6-9.5% on third-person videos and 26.9-30.9% on egocentric videos relative to metadata captions. Training-instruction enrichment improves both PhysCapBench scores and policy success, including a 7.5-percentage-point gain for pi0.5 on a RoboTwin 2.0 subtask. PhysCapBench makes the physical content of language supervision inspectable and comparable, providing a cross-domain basis for its evaluation and improvement.
