Digest

Reset

1167 papers

2026-10-23 · arXiv:2610.30319

TerraTransfer++: Demonstration-Free End-to-End Planner

Zikang Xiong, Weixin Li, Zhouchonghao Wu et al.

TerraTransfer++ trains a self-play driving policy in a vectorized simulator, then distills its rollouts into a flow-matching trajectory teacher that also outputs plans. Finally, it aligns a vision student (frozen head, trainable adapters) to reproduce the teacher via action KL divergence, a batch-relational low-rank structural loss, and (for planning) trajectory-matching. The authors report demonstration-free training with no logged-trajectory supervision at any stage.

DrivingAI summary
2026-10-23 · arXiv:2610.30320

Self-Distilling Future Video Representations for World Action Models

Huiwon Jang, Suhyeok Jang, Hyeonmo Kang et al.

The authors propose FRED, a self-distillation method for World Action Models that improves the quality of future video representations produced from noise in a single denoising step. FRED uses a momentum teacher to align student intermediate representations (from noisy future observations) with targets extracted from noisy ground-truth future observations, and conditions on a short history of observations. Across real-world manipulation tasks, it improves zero-shot generalization and human-to-robot skill transfer.

RoboticsAI summary
2026-10-23 · arXiv:2610.30461

Short-to-Long: Detached Features for Long-Horizon Value Learning

Tong Che, Boris Ivanovic, Marco Pavone

Short-to-Long (S2L) trains multiple value networks with shorter discounts and transfers their hidden features to a long-horizon behavior learner using stop-gradient (detached additions). Each source follows its own smaller-discount target while the behavior learner keeps the original long-horizon objective/discount. The authors report aggregate gains across Atari, online continuous control, and offline continuous control, including +0.049 median paired HNS on 40/57 Atari games.

DrivingAI summary
2026-10-23 · arXiv:2610.30488

From Single-Actor Environments to Multi-Actor Event Worlds: Can LLMs Be Event World Models?

Juntong Ni, Zewen Liu, Bohan Wang et al.

The authors study “event worlds” where multiple actors evolve the system and only an event log is observed. They ask whether LLMs can predict future observations from the log plus the agent’s proposed actions. In Diplomacy and Polymarket, prediction improves when all actors’ recorded actions are given, but with only the agent’s action models struggle when other actors strongly affect outcomes, limiting rollouts and plan success; imagined experience can still help policy learning.

2026-10-23 · arXiv:2610.30657

Anticipating Social Consequences with Distributional World Models

Viet-Thanh Pham, Thuy-Trang Vu, Zhuang Li et al.

DSWM (Distributional Social World Model) predicts, for each candidate action, multiple plausible social response clusters with probabilities plus immediate and final outcome values. The model is trained as a 4B world model from repeated simulations using supervised fine-tuning and GRPO. At inference, the actor uses these predicted alternatives to choose actions. The authors report best average performance on 13 of 15 metrics across three seen and two unseen environments, with interaction backbones absent from training.

Latent DynamicsAI summary
2026-10-23 · arXiv:2610.30717

PhyEvolve: Multi-Agent State-Evolution Planning for Physics-Aware Video Generation

Yang-Hao Zhou, Lixing Tan, Zhiqi Li et al.

PhyEvolve is a multi-agent prompt-enhancement framework for physics-aware text-to-video generation. A Director grounds the scene and separates physical actions from observation changes, a Physical Evolver constructs causally linked state transitions (including out-of-sight evolution), and a Composer rewrites the prompt. It improves existing generators via prompt replacement without retraining or modifying the generator. The authors report state-of-the-art gains on four physics video benchmarks.

2026-10-23 · arXiv:2610.30724

Learning from Next-Frame Prediction: Towards Unified and Scalable Visual Representation Learning

Yifeng Yao, Jinghan Li, Zhihao Zhang et al.

VANE is a self-supervised visual pretraining framework that learns by next-frame prediction in a frozen semantic embedding space. A causal encoder summarizes past frames; a state predictor infers the representation of the entire next frame. An auxiliary generative decoder reconstructs future-frame patches to preserve fine-grained spatial detail. The same objective is extended to still images via local-to-global crop sequences, enabling unified image–video training.

2026-10-23 · arXiv:2610.30781

DeXplicit: Making Physical State Explicit in Dexterous Manipulation World Models

Adam Lee, Mutian Xu, Yukang Cao et al.

DeXplicit is a dexterous manipulation world model that predicts metric 3D object-centric motion and deformation from an initial scene and a future hand trajectory, then renders video via 3D-guided appearance transport. The authors report moved-point error reductions (27–40% vs PointWorld) and improved object fidelity IoU+ (47% on HOT3D), plus efficient causal generation using two denoising steps per block (57× speedup from a 14B bidirectional configuration).

DrivingAI summary
2026-10-23 · arXiv:2610.30789

An Introduction to World Models for Generalist Agents

Zhihui Zhu

The authors provide an introductory, unified perspective on “world models” for generalist agents using a POMDP framework. They connect world modeling to classical control, dynamical systems, signal processing, and reinforcement learning, and explain how world modeling and policy learning can be related through conditional prediction (potentially without requiring separate models).

Latent DynamicsAI summary
2026-10-23 · arXiv:2610.30810

Adaptive Aerial Loco-Manipulation Policy Learning via LLM-Assisted Proactive Task Generation

Jingyu Ru, Xiangyuan Xie, Lixuan Jin et al.

ALMA learns an adaptive aerial loco-manipulation flight policy under external arm commands. It uses a contextual (GRU) policy trained with predictive context learning (action-conditioned prediction of future measurement representations in a JEPA-like setup). It also uses an LLM-assisted evolutionary process to generate proactive arm-motion tasks targeting observed tracking difficulties. In simulation it reports up to 24.7% lower position RMSE vs the strongest baseline and real-world zero-shot with 79.6% lower hover position RMSE vs PID.

DrivingAI summary
2026-10-23 · arXiv:2610.30814

DRILL: Training World Models to Improve Policies, Not Predict Pixels

Zhaolu Kang, Tailong Luo, Lei Wei et al.

DRILL trains video world models for downstream control by updating the model to maximize the policy return after an inner GRPO update on imagined rollouts, instead of optimizing one-step pixel prediction. DRILL-VWI is a closed-form surrogate with no extra simulator rollouts; DRILL-IMG is a meta-gradient method differentiating through the inner update with truncated Hessian-vector products. Across manipulation simulators, DRILL-IMG improves mean success rate while increasing one-step pixel MSE.

2026-10-23 · arXiv:2610.30988

In-Context Learning World Models for Robot Dual-Arm Manipulation of Soft Garment with Partial Observations

Hengyi Sim, Quentin Rouxel, Dennis Hong et al.

The authors present a Soft-Object World Model for bimanual garment folding with partial observations. It uses a transformer-based state estimator for generative full-state reconstruction and an action-conditioned dynamics predictor with in-context learning to infer latent shape/material cues at test time without weight updates. The model is trained on large-scale simulated soft-object point-cloud interactions and combined with a goal-conditioned action planner.

RoboticsAI summary
2026-10-23 · arXiv:2610.30991

UniFlowWAM: A Unified World-Action Model with Visual Semantics and 4D Geometry

Jialin Tian, Zhuofan Li, Xinkai Wang et al.

UniFlowWAM is a unified diffusion transformer that treats action, scene-level 3D flow, and future visual features as co-equal modalities. It uses one expert per modality with joint attention and trains with a ten-objective role grid (target/condition/absent) so the model can act directly without flow at test time. The authors evaluate in simulation on RoboCasa-GR1, LIBERO, LIBERO-Plus and on real bimanual manipulation tasks.

DrivingAI summary
2026-10-23 · arXiv:2610.31066

Dual Exam: Controlled Evaluation of Multimodal Agents and World Models

Zhefan Rao, Meng Chu, Xuan Billy Zhang et al.

Dual Exam evaluates multimodal Agents and World Models with controlled separation: Agents are tested under a fixed environment response mechanism, and World Models are tested under fixed action interventions. It reports an Agent Index over 1,088 items and a World Model WM Overall over 400 cases across eight scenarios, highlighting failures in task execution and state preservation.

Latent DynamicsAI summary
2026-10-23 · arXiv:2610.31194

Inside-JEPA: From Prediction to Relational Grounding in Latent World Models

Hossein Resani, Javen Qinfeng Shi

Inside-JEPA builds relational channels for ordered entity pairs into a JEPA latent state. The authors show that adding an inverse-dynamics loss—recovering actions from changes in relational channels—grounds these channels without relation labels. On Something-Something V2, prediction alone matches random; inverse supervision improves relational AUROC and supports relational action-conditioned forecasting and one-step goal-based action selection.

JEPAAI summary
2026-10-23 · arXiv:2610.31275

Learning Robot Behavior Priors from Non-Robotic Synthetic Data

Mateusz Odrowaz-Sypniewski, Michael Amir, Jasmine Bayrooti et al.

The authors test whether an in-context Transformer for robot action prediction can be pretrained solely on non-robotic, procedurally generated synthetic data from mathematical control abstractions (MDPs, contextual bandits, linear dynamical systems). They report that such a model achieves non-trivial zero-shot action prediction on unseen simulated and real robot datasets via in-context learning, and that specialized control priors improve over a generic causal tabular prior.

RoboticsAI summary
2026-10-23 · arXiv:2610.31429

PhysTrace: Interpretable Temporal Evaluation of Physical Consistency in Video Generative Models

Taekoan Yoo, Wonkyung Jung, Kyunghun Kim et al.

PhysTrace is a training-free framework for tracing history-dependent physical consistency in video generative models. It uses an MLLM to produce history-conditioned “Expected State” captions for each temporal window, aligns them with observed video windows via temporal video–language alignment, and uses the temporal alignment trace for discrimination and violation localization. On IntPhys2 it achieves 67.2% possible–impossible accuracy, and on IntPhys2-Timing it achieves Hit Rate up to 77.4%.

Video World ModelsCode availableAI summary
2026-10-23 · arXiv:2610.31474

Skeleton-Grounded Latent Action Model for Robot World Models

Qi Lv, Jiayi Lin, Yuxin Jiang et al.

SkeLAM is a skeleton-guided latent action model for controllable robot world modeling. It decomposes latent actions into a skeleton-grounded kinematic component and a residual component, trained with cross-modal swap reconstruction and branch dropout to support action-conditioned inference. A lightweight action adapter maps robot actions to the kinematic space, followed by world-model fine-tuning. On DROID, it improves end-effector motion and visual fidelity versus DreamDojo.

RoboticsAI summary
2026-10-23 · arXiv:2610.31494

AgentVQA: A Unified Benchmark for Agentic Visual Understanding

Advait Gupta, Sachin Konan, Tom Yin et al.

AgentVQA is an offline multiple-choice benchmark for agentic visual understanding across five domains (Web Agents, Egocentric Videos, Robotics, Games, Spatial Planning). It is built from 1,000 manually verified questions adapted from 23 datasets. Across 22 VLMs, the best model reaches 58.5% accuracy. The authors report errors are mainly due to failures to ground and maintain accurate representations of evolving visual state.

AI summary
2026-10-23 · arXiv:2610.31617

DiffDreamer: Coupling World Models with Differentiable Simulation for Agile Quadrotor Control

Fanxing Li, Yuxiang Huang, Shengyang Wang et al.

DiffDreamer couples a learned world model with a differentiable quadrotor simulator for actor optimization. In one step, the world model propagates latent states for external, history-dependent dynamics while the simulator propagates the quadrotor’s explicit physical state. The actor objective is split into a critic-driven term for non-differentiable task rewards and a simulator-driven term for differentiable motion rewards, improving training convergence and final performance in aggressive tracking and dynamic-gate racing.

DrivingAI summary