Video World Models · 2026-10-20
Time-JEPA: Grounding Video World Models with Physical Time Series
Shenyi Li, Ziyang Zhang, Wen Wu, Ming Liu, Chao Zhang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Time-JEPA fine-tunes a video world model with low-dimensional, temporally aligned physical time series as auxiliary supervision. Physical time series are aligned to the model’s temporal grid and pooled per tubelet, and an auxiliary regression head predicts aligned physical state evolution during fine-tuning. At evaluation, video is the only input.
Why it matters
The authors address a gap where existing physical-state supervision often focuses on individual frames/pairs and underuses full temporal evolution. Time-JEPA provides a shared interface that uses readily available physical time series to encourage video world model representations to preserve how physical state evolves over time, improving decoding, multi-horizon prediction, and kinematic reconstruction across autonomous driving, robot manipulation, and human motion, including under distribution shifts.
Method
- Fine-tune a V-JEPA-style masked latent-prediction model with temporally aligned physical time-series supervision using a shared auxiliary interface across domains.
- Align physical measurements to the encoder’s temporal tubelets, pool and normalize per tubelet, and regress aligned normalized state values from pooled temporal video features using a validity-masked Smooth-L1 loss.
- At evaluation, discard the auxiliary head and perform video-only probing for current-state decoding, multi-horizon future-state prediction, kinematic reconstruction, and external physical evaluations.
Limitation
The authors report that adding explicit physical-law constraints to Time-JEPA lowers both current-state metrics and future-state R2 and raises future-state nRMSE in all four settings (Tables K3 and K4).
Abstract (from arXiv)
Video world models learn how visual environments evolve by predicting missing or future content. However, physical-state information can provide useful context for predicting how environments evolve. Prior work incorporates this information through different mechanisms: domain-specific physical laws can complicate multi-domain training, while supervision on individual frames or frame pairs gives limited attention to how physical state evolves over time. We introduce Time-JEPA, which fine-tunes a video world model using low-dimensional, temporally aligned physical time series as auxiliary supervision. Physical time series connect video representations with physical state at corresponding temporal positions, encouraging the model to preserve how physical state evolves. In multiple domains including autonomous driving, robot manipulation, and human motion, Time-JEPA improves current-state decoding, multi-horizon future-state prediction, and kinematic reconstruction over matched video-fine-tuned controls across both condition and state-variable distribution shifts. Multi-domain training provides aggregate gains over matched single-domain training in all three tasks. External evaluations further suggest that time-series supervision improves sensitivity to physical anomalies and the representation of physical interactions, including in unseen scenarios. Aggregate gains on a second video backbone indicate that the method can be applied to more backbones. These results suggest that physical time series help video world models better understand the physical world.
Related papers
- World Models Need a Physical Registry: Learning Internal Predictive States for Physically Plausible World Models
- Exploring Temporal Context-Aware Physical Grounding in Latent World Models
- Reading, Writing, Sharing, and Interacting with the World through Video
- STRIKE: Learning Visual State Transitions for Physical World Modeling
- {\Delta}WAM: Distilling Action Tangent Fields into World Action Models
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-14 · Indexed by Pub-AI: 2026-10-20 UTC
Time-JEPA: Grounding Video World Models with Physical Time Series
Preprint posted: 2026-10-14 · Original record ↗
- arxiv
- 2610.23982
Explicit source versions (1)
- v1 · 2026-10-14
- Preprint available
Source date: 2026-10-14 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Video world models learn how visual environments evolve by predicting missing or future content. However, physical-state information can provide useful context for predicting how environments evolve. Prior work incorporates this information through different mechanisms: domain-specific physical laws can complicate multi-domain training, while supervision on individual frames or frame pairs gives limited attention to how physical state evolves over time. We introduce Time-JEPA, which fine-tunes a video world model using low-dimensional, temporally aligned physical time series as auxiliary supervision. Physical time series connect video representations with physical state at corresponding temporal positions, encouraging the model to preserve how physical state evolves. In multiple domains including autonomous driving, robot manipulation, and human motion, Time-JEPA improves current-state decoding, multi-horizon future-state prediction, and kinematic reconstruction over matched video-fine-tuned controls across both condition and state-variable distribution shifts. Multi-domain training provides aggregate gains over matched single-domain training in all three tasks. External evaluations further suggest that time-series supervision improves sensitivity to physical anomalies and the representation of physical interactions, including in unseen scenarios. Aggregate gains on a second video backbone indicate that the method can be applied to more backbones. These results suggest that physical time series help video world models better understand the physical world.