Digest

Reset

1180 papers

2025-06-11 · arXiv:2506.09985

V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning

Mido Assran, Adrien Bardes, David Fan et al.

Pretrains an action-free JEPA video model on over 1 million hours of internet video, then post-trains a latent action-conditioned world model (V-JEPA 2-AC) on under 62 hours of unlabeled robot video and uses it to plan pick-and-place with image goals, zero-shot on Franka arms in two labs.

JEPACode availableEditor pick
2025-01-07 · arXiv:2501.03575

Cosmos World Foundation Model Platform for Physical AI

NVIDIA, :, Niket Agarwal et al.

Cosmos is NVIDIA's platform for 'world foundation models' for Physical AI: a video curation pipeline, pre-trained diffusion and autoregressive world models, examples of post-training for downstream tasks, and video tokenizers, released open-source with open-weight models.

Video World ModelsCode availableEditor pick
2024-05-20 · arXiv:2405.12399

Diffusion for World Modeling: Visual Details Matter in Atari

Eloi Alonso, Adam Jelley, Vincent Micheli et al.

DIAMOND trains an RL agent inside a diffusion world model, arguing that discrete-latent compression can lose visual details that matter for control; it reaches a mean human-normalized score of 1.46 on Atari 100k.

Latent DynamicsAI summary
2024-02-23 · arXiv:2402.15391

Genie: Generative Interactive Environments

Jake Bruce, Michael Dennis, Ashley Edwards et al.

Genie is an 11B-parameter generative interactive environment trained without action labels on unlabelled internet videos; a spatiotemporal video tokenizer, autoregressive dynamics model and latent action model let users act frame by frame in generated worlds.

2024-02-15 · arXiv:2404.08471

Revisiting Feature Prediction for Learning Visual Representations from Video

Adrien Bardes, Quentin Garrido, Jean Ponce et al.

V-JEPA trains video models solely with a feature-prediction objective (predicting masked spatio-temporal regions in representation space), with no pretrained image encoders, text, negatives or reconstruction, on 2 million public videos.

JEPACode availableAI summary
2023-10-25 · arXiv:2310.16828

TD-MPC2: Scalable, Robust World Models for Continuous Control

Nicklas Hansen, Hao Su, Xiaolong Wang

TD-MPC2 improves TD-MPC, which does local trajectory optimization in the latent space of a decoder-free (implicit) world model; one set of hyperparameters works across 104 online RL tasks, and a single 317M-parameter agent performs 80 tasks across domains and embodiments.

Latent DynamicsAI summary
2023-10-09 · arXiv:2310.06114

Learning Interactive Real-World Simulators

Sherry Yang, Yilun Du, Kamyar Ghasemipour et al.

UniSim learns a universal simulator of real-world interaction by orchestrating diverse datasets (image, robotics, navigation) so it can simulate the visual outcome of both high-level instructions and low-level controls; policies trained purely inside it deploy zero-shot on real robots.

2023-09-29 · arXiv:2309.17080

GAIA-1: A Generative World Model for Autonomous Driving

Anthony Hu, Lloyd Russell, Hudson Yeo et al.

GAIA-1 is a generative world model for driving that takes video, text and action inputs, casts world modelling as next-token prediction over discrete tokens, and decodes with a video diffusion model to produce realistic driving scenes with control over ego-vehicle behaviour and scene features.

DrivingEditor pick
2023-09-18 · arXiv:2309.09777

DriveDreamer: Towards Real-world-driven World Models for Autonomous Driving

Xiaofeng Wang, Zheng Zhu, Guan Huang et al.

DriveDreamer is a diffusion-based world model learned from real-world driving data (evaluated on nuScenes) that generates controllable driving videos and future driving policies, trained in two stages (structured traffic constraints first, then future-state prediction).

DrivingAI summary
2023-01-10 · arXiv:2301.04104

Mastering Diverse Domains through World Models

Danijar Hafner, Jurgis Pasukonis, Jimmy Ba et al.

DreamerV3 is a general model-based RL algorithm that learns a world model and improves behavior by imagining future scenarios; with a single configuration it outperforms specialized methods across over 150 tasks, and is the first to collect diamonds in Minecraft from scratch without human data.

Latent DynamicsCode availableAI summary
2022-09-01 · arXiv:2209.00588

Transformers are Sample-Efficient World Models

Vincent Micheli, Eloi Alonso, François Fleuret

IRIS is a data-efficient RL agent that learns inside a world model built from a discrete autoencoder plus an autoregressive Transformer, and reaches a mean human-normalized score of 1.046 on Atari 100k (about two hours of gameplay).

Latent DynamicsCode availableAI summary
2022-06-28 · arXiv:2206.14176

DayDreamer: World Models for Physical Robot Learning

Philipp Wu, Alejandro Escontrela, Danijar Hafner et al.

DayDreamer applies the Dreamer algorithm to four physical robots that learn online in the real world without simulators, using the same hyperparameters across tasks: a quadruped, two robot arms and a wheeled robot.

RoboticsAI summary
2020-10-05 · arXiv:2010.02193

Mastering Atari with Discrete World Models

Danijar Hafner, Timothy Lillicrap, Mohammad Norouzi et al.

DreamerV2 learns behaviors purely from predictions in the latent space of a world model that uses discrete latent representations and is trained separately from the policy, and is the first such agent to reach human-level performance on the 55-game Atari benchmark.

Latent DynamicsAI summary
2019-12-03 · arXiv:1912.01603

Dream to Control: Learning Behaviors by Latent Imagination

Danijar Hafner, Timothy Lillicrap, Jimmy Ba et al.

Dreamer learns behaviors purely by latent imagination: it trains an actor and a value function on trajectories imagined in a learned world model, backpropagating analytic value gradients through the learned dynamics.

Latent DynamicsAI summary
2019-11-19 · arXiv:1911.08265

Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model

Julian Schrittwieser, Ioannis Antonoglou, Thomas Hubert et al.

MuZero combines tree-based search with a learned model that predicts only the quantities needed for planning (reward, policy and value), reaching superhuman performance in Go, chess and shogi without being given the rules and a new state of the art on 57 Atari games.

Latent DynamicsAI summary
2019-03-01 · arXiv:1903.00374

Model-Based Reinforcement Learning for Atari

Lukasz Kaiser, Mohammad Babaeizadeh, Piotr Milos et al.

SimPLe (Simulated Policy Learning) trains a video prediction model of an Atari game and learns a policy inside it, using only 100k environment interactions, about two hours of real-time play.

Latent DynamicsCode availableAI summary
2018-11-12 · arXiv:1811.04551

Learning Latent Dynamics for Planning from Pixels

Danijar Hafner, Timothy Lillicrap, Ian Fischer et al.

PlaNet learns a latent dynamics model from images and chooses actions by fast online planning in latent space, using a model with both deterministic and stochastic transition parts and a multi-step 'latent overshooting' training objective.

Latent DynamicsAI summary
2018-03-27 · arXiv:1803.10122

World Models

David Ha, Jürgen Schmidhuber

Trains a VAE to compress each frame and a recurrent mixture-density network to predict the next latent, then fits a tiny linear controller on their features; the agent can also be trained entirely inside the model's own generated 'dream' and transferred back to the real environment.

Latent DynamicsAI summary