Video World Models · 2026-10-06

PWM: Personalized World Models with Online Reinforcement Learning

Zhexin Lou, Guancheng Lu, Zeyu Zhang, Yi Zhang, Yang Zhao, Hao Tang

arXiv:2610.04920PDFCode

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

PWM customizes an action-conditioned interactive world model to a specific scene using a short scene video. It learns a compact LoRA adapter per scene while keeping a frozen Yume-5B backbone, using online reinforcement learning updates from fresh rollouts. PWM is instantiated with GRPO and with DiffusionNFT. A benchmark, PWM-Bench (150 tasks), evaluates on held-out continuations and reports improved customization over native Yume.

Why it matters

The authors target personalized interactive world generation: learning a scene’s visual identity from a short support video while preserving the quality of action-conditioned generation. PWM-Bench provides a paired held-out evaluation protocol (support-based training, test-only scoring) across domains, enabling direct comparison to native Yume and matched SFT.

Method

  • Learn a per-scene rank-8 LoRA adapter on top of a frozen Yume-5B interactive world model; only the adapter parameters are updated during online adaptation from a short support trajectory.
  • Use online reward-guided optimization from native generation windows (support windows with fresh rollouts): GRPO uses group-relative optimization with a base-policy anchor; DiffusionNFT uses reward-weighted flow-matching without likelihood estimation.
  • Evaluate with PWM-Bench (150 scene tasks with held-out non-overlapping continuations), scoring scene fidelity and image quality using the same pipeline across methods, and reporting paired scene-level improvement rates and relative gains.

Limitation

The paper does not use any frame from the future evaluation portion for optimization or checkpoint construction (adaptation uses only the support windows; evaluation uses only the held-out test segment).

Abstract (from arXiv)

Pretrained world models can generate diverse environments, yet users often want to explore a particular scene specified by their own video. This requires learning the scene's visual identity while retaining the quality of action-conditioned generation. We introduce Personalized World Models (PWM), a framework for customizing interactive world models from short scene videos through online reinforcement learning. In PWM, the support trajectory and its associated controls provide reward feedback on continuations sampled from the current policy. In the GRPO instantiation, group-relative optimization updates a compact LoRA adapter using a unified reward for scene appearance, visual continuity, and motion, while base-policy anchoring regularizes changes to the pretrained generation prior of a frozen Yume-5B backbone. The same adaptation procedure is applied across real and rendered environments. We also instantiate PWM with DiffusionNFT as an alternative reward-guided optimization method for learning the scene-specific adapter. We also introduce PWM-Bench, comprising 150 customization tasks across Indoor, Outdoor, and Gaming, with paired evaluation on held-out continuations. The GRPO and DiffusionNFT instantiations of PWM improve customization over native Yume in 71.3% and 65.3% of the evaluated scenes, respectively, with positive mean gains across all three domains. For the GRPO instantiation, matched SFT comparisons further demonstrate higher mean customization gains and better mean image-quality scores in every domain, while retaining frame-level visual quality close to the pretrained model.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-04 · Indexed by Pub-AI: 2026-10-06 UTC

preprint

PWM: Personalized World Models with Online Reinforcement Learning

Zhexin Lou, Guancheng Lu, Zeyu Zhang, Yi Zhang, Yang Zhao, Hao Tang

Preprint posted: 2026-10-04 · Original record ↗

arxiv
2610.04920
Explicit source versions (1)
  • v2 · 2026-10-12
  1. Explicit source version detected · v2

    Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
  2. Preprint available

    Source date: 2026-10-04 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Pretrained world models can generate diverse environments, yet users often want to explore a particular scene specified by their own video. This requires learning the scene's visual identity while retaining the quality of action-conditioned generation. We introduce Personalized World Models (PWM), a framework for customizing interactive world models from short scene videos through online reinforcement learning. In PWM, the support trajectory and its associated controls provide reward feedback on continuations sampled from the current policy. In the GRPO instantiation, group-relative optimization updates a compact LoRA adapter using a unified reward for scene appearance, visual continuity, and motion, while base-policy anchoring regularizes changes to the pretrained generation prior of a frozen Yume-5B backbone. The same adaptation procedure is applied across real and rendered environments. We also instantiate PWM with DiffusionNFT as an alternative reward-guided optimization method for learning the scene-specific adapter. We also introduce PWM-Bench, comprising 150 customization tasks across Indoor, Outdoor, and Gaming, with paired evaluation on held-out continuations. The GRPO and DiffusionNFT instantiations of PWM improve customization over native Yume in 71.3% and 65.3% of the evaluated scenes, respectively, with positive mean gains across all three domains. For the GRPO instantiation, matched SFT comparisons further demonstrate higher mean customization gains and better mean image-quality scores in every domain, while retaining frame-level visual quality close to the pretrained model.