Digest

Reset

1167 papers

2026-10-23 · arXiv:2610.31668

Reading, Writing, Sharing, and Interacting with the World through Video

Shengqiong Wu, Meng Luo, Bobo Li et al.

This survey frames video foundation models and video world models using a unifying “world modeling” lens: treating an observed video as a partial, lossy, temporally ordered projection of a latent temporal world. It organizes prior work into four operations—reading, writing, sharing, and interacting—and distinguishes temporal coherence from consistency of the underlying world state, highlighting shifts toward persistent state, controllable evolution, and evaluation via interaction and decisions.

Video World ModelsCode availableAI summary
2026-10-23 · arXiv:2610.31670

Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

Yunfei Xie, Howard H. Qian, Shiyi Lan et al.

The authors propose Sensorimotor Alignment (S²), a training-free offline metric that measures overlap between nearest-neighbor neighborhoods in a model’s sensory representation (image + text features) and motor representation (end-effector trajectories). They report positive correlations between S² and downstream robot task success across embodied model families, and show S² is more predictive than VQA scores or video-quality metrics.

DrivingAI summary
2026-10-22 · arXiv:2610.28698

Scaling in Games: Continued Pre-Training For Embodied Agents In Diverse Virtual Worlds

Kuan Zhang, Yukun Chen, Zhihao Yang et al.

The authors study continued pre-training (CPT) for embodied keyboard–mouse agents across a controlled grid varying data volume, mixture diversity, and model size (0.8B–27B). More data improves in-corpus performance; diversity improves transfer to related worlds; model size improves data efficiency but can generalize worse to distant worlds. They fit an empirical three-factor scaling law for validation loss and find CPT competitive in 3D and best in real-time, with persistent distant-world gaps.

DrivingAI summary
2026-10-22 · arXiv:2610.28753

Understanding the Inner Workings of a State-Space Foundation Model for Limit Order Books

Valentin Mohl, Alexandre Bismuth, Aramis Marti-Shahandeh et al.

The authors study a message-only 8.2M-parameter Mamba-3 state-space foundation model trained autoregressively on tokenized limit-order-book (LOB) message streams. They report that the model’s recurrent state contains decodable market microstructure dynamics: order-flow imbalance (OFI) is decoded at R^2≈0.59 (far above controls), generalizes to unseen stocks, and is causally steerable via monotonic dose responses.

Latent DynamicsAI summary
2026-10-22 · arXiv:2610.28779

Second Thoughts: Exploring and Planning through Comparative Imagination in World Models

Hieu Tran, Hong Yu

Second Thoughts (SDI) links exploration and planning decisions in world-model RL by comparing imagined action “siblings” to repeated rollouts of the actor’s fixed sequence from the same latent state. The spread of sibling returns drives an exploration bonus, and a gate accepts a planned action only when its predicted advantage exceeds actor-repeat variability by a fixed factor. Evaluation uses the learned actor alone.

Latent DynamicsAI summary
2026-10-22 · arXiv:2610.28827

WorldStream: Streaming Interactive 3D Worlds with Hierarchical Planning and Persistent State

Xiaoxiao Ma, Weiyang Jin, Yanwen Ma et al.

WorldStream is a streaming interactive 3D world model with hierarchical planning, deterministic local compilation, and a single persistent world record. Macro planning sets multi-tile settlement/ecology/transport/hydrology; Micro specializes tile semantics under measured terrain and accepted neighbor commitments. Plans run ahead with budgeted admission; compiled geometry is evicted but recompiled on revisit. Persistent state also grounds language-driven actions and edits.

2026-10-22 · arXiv:2610.28829

HorizonComposer: Spatiotemporally Consistent Driving Video Editing with Enriched Traffic Semantics

Mauricio Soroco, Yuqiu Liu, Zaid Tasneem et al.

HorizonComposer is a driving video editor that distills and protects traffic semantics during instruction-guided global appearance edits. It builds hybrid paired data from temporally aligned real videos and synthetic edits, trains with offline RL using dense pixel-level rewards, and mitigates “temporal shock” using “thinking frames” for smoother transitions from static conditioning to video dynamics.

DrivingAI summary
2026-10-22 · arXiv:2610.28945

Self-Improving Agents with World Model Predictive Paths

Haori Lu, Tianyu Gao, Yaoyao Liu et al.

The authors propose a test-time, MPPI-inspired framework for LLM agents: at each step, the agent proposes multiple candidate actions, uses a world model to predict one-step outcomes, scores each outcome for task progress and risk, and applies MPPI-style (Gibbs) weights over predictive costs to guide a final controller to execute one action. A dual-branch experience memory improves both candidate generation/selection and next-observation prediction.

Latent DynamicsAI summary
2026-10-22 · arXiv:2610.29061

OccJEPA: Occlusion as Masking for Amodal Video Segmentation

Genpei Zhang, Runmin Jiang, Xingzu Zhan et al.

OccJEPA (Occlusion-as-masking JEPA) treats occlusion as masking in a self-supervised video world model. Tokens not covered by the object’s visible mask are dropped; the V-JEPA 2.1 predictor is fine-tuned to fill them in, and a lightweight decoder produces amodal masks. The authors report deterministic inference and 32× faster per frame than Diffusion-VAS, with improved accuracy on SAIL-VOS, TAO-Amodal, and Kubric.

Video World ModelsCode availableAI summary
2026-10-22 · arXiv:2610.29088

Causal world models under physical parameter shifts

Chu Lin, Yi Sun

The authors study how causal world-model design affects prediction under physical parameter shifts. In a synthetic shortcut benchmark, a frozen intervention-derived dependency mask reduces mean OOD RMSE from 1.769 to 0.697 (near an oracle 0.694). In CausalWorld link-mass extrapolation, a mass-affine “known-PD” mechanism turns out-of-range masses into consistent predictive gains (median joint R2 0.609→0.727), while other predictors degrade when given out-of-range masses.

Latent DynamicsAI summary
2026-10-22 · arXiv:2610.29139

Steering Black-Box Large Language Model Agents with Pitfall Anticipation World Model

Jiongxiao Wang, Han Ding, Youzhi Luo et al.

The authors propose Pitfall Anticipation World Model (PAWM) to steer frozen black-box LLM agents. Instead of predicting next observations every step, PAWM selectively intervenes when it foresees an impending pitfall and emits forward-looking guidance (or <pass>). PAWM is trained with SFT on a synthetic pitfall dataset and then co-trained with GRPO in the agent loop. The authors report improved Claude Sonnet 4.5 success on ScienceWorld and WebArena-Lite.

DrivingAI summary
2026-10-22 · arXiv:2610.29168

How Representational Priors Shape Generalization in World Action Models

Junha Chun, Youngjoon Jeong, Taesup Kim

The authors study world action models (WAMs) and find that representational geometry inherited from generative backbones predicts generalization under visual shifts. They define nuisance orthogonality (feature displacement orthogonal to the empirical in-distribution manifold) and show higher orthogonality is associated with stronger LIBERO-Plus policy success after WAM training. After training, they analyze nuisance suppression (reduced residual displacement) as an explanation, with ImageWAM showing stronger suppression than FastWAM.

2026-10-22 · arXiv:2610.29187

A Decade of Learning-Based Control: From Theoretic Foundations to Practical Deployment

Angela P. Schoellig, Matthias A. M\"uller, Sebastian Trimpe et al.

Tutorial reviewing learning-based control advances over the past decade, focusing on combining machine learning with control-theoretic structure to design autonomous systems with stability and safety guarantees. It covers foundations for stability/safety, dynamics learning for control under uncertainty, and direct data-driven policy synthesis/optimization (RL, imitation learning, Bayesian optimization). The authors also share implementations and illustrative examples and provide additional resources on a project website.

DrivingAI summary
2026-10-22 · arXiv:2610.29244

Failure Modes of Vision-Language Models for Human Activity Understanding in Thermal HRI

Han Hu, He Chen

The authors evaluate seven locally deployable VLMs on two person-centric thermal HRI datasets. They test fine-grained action recognition, action–object interaction and temporal grounding, and candidate-set backoff/abstention. Models recognize broad activity categories but struggle on fine-grained actions, produce inaccurate event timelines, and hallucinate object-related actions. Intervention gains from input preprocessing/reasoning are limited and model abstention remains unreliable.

RoboticsAI summary
2026-10-22 · arXiv:2610.29246

DepState-JEPA: Learning Depth-Grounded State Representation for Visual Perception and Manipulation

Xiujian Liang, Liuyuan Na, Zehao Du et al.

DepState-JEPA, built on V-JEPA 2.1, learns a depth-grounded shared latent state (DepState) from RGB–depth video. Depth is used both as a complementary stream and to place both modalities in a common metric 3D frame. A shared ray-conditioned decoder predicts missing and future RGB/depth features under action-conditioned latent transitions, with additional robot-state and kinematic (FK) consistency losses for manipulation control.

Latent DynamicsAI summary
2026-10-22 · arXiv:2610.29282

Relational Geometry Forcing for World Action Models

Yuan Xu, Yixiang Chen, Qisen Ma et al.

World Action Models (WAMs) learn visual dynamics and actions but their objectives don’t explicitly constrain 3D geometry for precise manipulation. The authors propose Relational Geometry Forcing (RGF), which transfers geometry priors from a frozen 3D foundation model by aligning pairwise feature relations (intra-frame, cross-view, and action-temporal) during training only, without adding a geometry branch for inference.

DrivingAI summary
2026-10-22 · arXiv:2610.29308

LLM-Based Exogenous World Modeling for Dynamic Scheduling

Ni Zhang, Haofeng Yuan, Jianan Zhou et al.

AWMP (Afterstate World-Model Planning) tackles dynamic scheduling by using a pretrained LLM as an exogenous world-model component that forecasts future job-arrival pressure via a compact early–late belief. Afterstate decomposition keeps deterministic shop transitions exact, while online residual self-calibration and surprise-triggered belief refresh adapt forecasts. Short scenario rollouts and meta-evolved planning yield lower total weighted tardiness than the best of eight classical priority-dispatching rules.

DrivingAI summary
2026-10-22 · arXiv:2610.29329

Plan First, Act Later: Streaming Reasoning for Executable Memory in Robot Control

Qinghang Liu, Conglin Wang, Zongsheng Li et al.

PAL is a hierarchical memory framework for long-horizon vision-language robot manipulation. It maintains a short-term visual buffer, a semantic-event memory, and an executable subtask_list. A high-level VLM streams observations into memory and selects the current subtask for a low-level VLA policy. Training uses two stages: dense update-timing supervision plus self-labeling with task-grounded verification.

RoboticsAI summary
2026-10-22 · arXiv:2610.29385

Parallelizable Probabilistic Latent Reasoning for Diverse Reasoning Trajectories

Seokhyun Lee, Seongjun Lee, Changhee Lee

PLRM (Probabilistic Latent Reasoning Models) moves reasoning from verbal chain-of-thought into a compact continuous latent space using a variational, JEPA-style objective. A parallelizable posterior infers latent distributions from observed reasoning traces, while a problem-conditioned prior predicts the next latent distribution via teacher forcing. A discrete strategy variable with a mixture-based prior models multimodal reasoning trajectories without collapsing them.

Latent DynamicsAI summary
2026-10-22 · arXiv:2610.29408

LiveAct: Towards Real-Time Multi-Agent World Models with Heterogeneous Action Interfaces

Xinyu Wang, Huafeng Shi, Yue Ma et al.

LiveAct is a real-time multi-agent video world model for interactive environments with heterogeneous action interfaces. It introduces a Heterogeneous Action Interface (HAI) that binds each indexed action stream to its intended visual agent (Agent-Aware Spatial Modulation) and adapts action semantics across tasks while preserving synchronized numerical values (Schema-Guided Action Adaptation). A causal generator enables efficient streaming.