Driving · 2026-10-08

Controllable Crowd Generation through World-Model Planning

JunGyu Lee, Jisu Shin, Seunghyun Shin, Hae-Gon Jeon

arXiv:2610.09438PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

Ctrl-CWM is a multi-agent controllable crowd world model that uses imagined futures for both crowd generation and run-time control. It learns human motion dynamics via trajectory prediction on real pedestrian videos (then freezes the encoder), trains an actor to propose one-step displacements and a critic to rank imagined trajectories, and uses CEM planning with a user cost to steer avoidance/attraction without retraining.

Why it matters

The authors aim to let crowd simulations respond to new user-specified objectives at run time while preserving learned human motion dynamics. By combining an imitation-trained dynamics representation with critic-guided imagination-based planning, Ctrl-CWM supports generation and control with a single model and can introduce objectives during simulation without retraining.

Method

  • Learn human motion dynamics with a scene-aware, goal-conditioned trajectory prediction model (Y-Net style), then freeze the state encoder for simulation.
  • Train an actor (shared across pedestrians) to propose next-step displacements; train a critic to rank imagined multi-step crowd trajectories based on motion quality/collision costs.
  • At run time, use CEM planning over candidate displacements with the critic score plus a user cost (e.g., avoidance/attraction) and replan each step; user costs can be changed without retraining.

Limitation

The authors note that the planning optimization is an approximation to joint multi-agent planning: next-step displacement optimization is done independently per controlled agent while other agents follow actor proposals, and the resulting displacements are executed simultaneously.

Abstract (from arXiv)

Crowd simulation plays a central role in robot navigation, autonomous driving, and urban planning. For these applications, realistic simulation requires crowds to adapt their behavior to environmental changes and user objectives. However, existing methods that rely on predefined control settings have limited flexibility in accommodating new user-specified objectives. To address this limitation, we propose Ctrl-CWM, a multi-agent Controllable Crowd World Model that integrates crowd generation and run-time control. Our key idea is to adapt the world-model principle of planning using imagined futures to crowd simulation. To this end, Ctrl-CWM consists of an encoder that learns a representation of human motion dynamics, an actor that proposes pedestrian displacements, a critic that evaluates imagined crowd trajectories, and a planner that selects actions. We first learn human motion dynamics through trajectory prediction on real-world pedestrian videos and then freeze the encoder to preserve them. Using this representation, the actor generates imagined crowd trajectories through repeated state updates, and the planner combines the critic's scores with user costs to select actions. Repeated planning advances the simulated crowd, while additional user costs introduce new control objectives without retraining. We extensively evaluate crowd generation under varied agent arrival conditions and run-time control across avoidance and attraction scenarios. Ctrl-CWM outperforms the state-of-the-art method on most crowd realism and collision metrics, and adapts crowd behaviors to user-specified objectives introduced during simulation. The project page is available at https://jungyu0413.github.io/Ctrl-CWM

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-07 · Indexed by Pub-AI: 2026-10-08 UTC

preprint

Controllable Crowd Generation through World-Model Planning

JunGyu Lee, Jisu Shin, Seunghyun Shin, Hae-Gon Jeon

Preprint posted: 2026-10-07 · Original record ↗

arxiv
2610.09438
Explicit source versions (1)
  • v1 · 2026-10-07
  1. Preprint available

    Source date: 2026-10-07 · First detected: 2026-10-25 UTC · Historical baseline

    Source evidence (1)
Abstract · arxiv

Crowd simulation plays a central role in robot navigation, autonomous driving, and urban planning. For these applications, realistic simulation requires crowds to adapt their behavior to environmental changes and user objectives. However, existing methods that rely on predefined control settings have limited flexibility in accommodating new user-specified objectives. To address this limitation, we propose Ctrl-CWM, a multi-agent Controllable Crowd World Model that integrates crowd generation and run-time control. Our key idea is to adapt the world-model principle of planning using imagined futures to crowd simulation. To this end, Ctrl-CWM consists of an encoder that learns a representation of human motion dynamics, an actor that proposes pedestrian displacements, a critic that evaluates imagined crowd trajectories, and a planner that selects actions. We first learn human motion dynamics through trajectory prediction on real-world pedestrian videos and then freeze the encoder to preserve them. Using this representation, the actor generates imagined crowd trajectories through repeated state updates, and the planner combines the critic's scores with user costs to select actions. Repeated planning advances the simulated crowd, while additional user costs introduce new control objectives without retraining. We extensively evaluate crowd generation under varied agent arrival conditions and run-time control across avoidance and attraction scenarios. Ctrl-CWM outperforms the state-of-the-art method on most crowd realism and collision metrics, and adapts crowd behaviors to user-specified objectives introduced during simulation. The project page is available at https://jungyu0413.github.io/Ctrl-CWM