Video World Models · 2026-11-10
Tiresias: Towards Language-Conditioned World Models
Fengyi Wu, Yifei Dong, Yetong Sha, Lingdong Kong, Qiyu Hu, Bohan Xiong, Qi Dai, Alexander G. Hauptmann, Zhi-Qi Cheng
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce LCWM, a language-conditioned world modeling task that maps an initial agent-view observation and an instruction to an action sequence fulfilling the instruction, and its benchmark LCWM-Bench spanning 15 datasets across navigation, robotics, 3D gaming, and driving. They propose Tiresias, which plans via selective imagination using a diffusion policy for trajectory proposals, trajectory/visual rankers, and a video-pretrained world model for imagining consequences.
Why it matters
The paper targets fragmentation in language-conditioned world modeling across domains with different control interfaces and evaluation protocols. LCWM-Bench provides a unified action parameterization, instruction styles, and metric suite to measure both planning (goal completion/trajectory quality) and imagination (visual fidelity) across multiple embodied domains, enabling cross-domain comparisons.
Method
- Define LCWM: given initial agent-view RGB observation and natural-language instruction, predict an action sequence that fulfills the instruction across four domains with a shared base action parameterization plus domain-specific channels.
- Introduce LCWM-Bench: 15 datasets integrated under a shared action interface and standardized evaluation, with 3 instruction styles (Concise/Intricate/Landmark-based) and splits for train/val seen/val unseen/test; report SR, nDTW, ATE, and RPE; overall score is macro-average of SR and nDTW across domains.
- Propose Tiresias: language-conditioned diffusion policy proposes candidate trajectories; trajectory ranker shortlists using language and motion; a language-conditioned video world model imagines outcomes for shortlisted candidates; visual ranker evaluates imagined rollouts and fuses trajectory+visual scores to select the final plan.
Abstract (from arXiv)
World models enable planning through internal simulation, while natural language provides intuitive task specification. Yet, language-conditioned world modeling remains fragmented across domains with disparate action spaces, datasets, and evaluation protocols. We introduce LCWM, a shared task that maps an initial agent-view observation and a language instruction to an action sequence that fulfills the instruction. Its accompanying benchmark, LCWM-Bench, integrates 15 datasets across navigation, robotics, 3D gaming, and driving under a shared action interface and evaluation protocol, pairing 142,987 episodes with 428,961 instructions in three styles. We further propose Tiresias, an agent that plans via selective imagination: a language-conditioned diffusion policy generates candidate trajectories, a trajectory ranker shortlists them, and a video-pretrained world model imagines their visual consequences. A visual ranker evaluates these rollouts, complementing trajectory scores to select the final plan. Tiresias achieves competitive planning and imagination performance on LCWM-Bench. Ablations demonstrate the complementary benefits of trajectory ranking and visual verification, while experiments with alternative proposal methods show that the trained rankers remain effective beyond our diffusion policy. Together, LCWM-Bench and Tiresias provide a unified evaluation setting and an effective starting point for studying language-guided planning through world models.
Related papers
- The Planning Limits of Latent World Models
- ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning
- Second Thoughts: Exploring and Planning through Comparative Imagination in World Models
- D-World: An Explicit World-Language-Action Model with Recursive Self-Improvement under Dual Control
- LaLa-World: Scaling Latent World Models with Natural Language Actions
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First announced by arXiv: 2026-11-10 · Indexed by Pub-AI: 2026-11-10 UTC
Tiresias: Towards Language-Conditioned World Models
First announced by arXiv: 2026-11-10 · Original record ↗
- arxiv
- 2611.10392
Explicit source versions (1)
- v1 · Date not supplied
- Preprint available
Source date: 2026-11-10 · First detected: 2026-11-10 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.10392v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv
World models enable planning through internal simulation, while natural language provides intuitive task specification. Yet, language-conditioned world modeling remains fragmented across domains with disparate action spaces, datasets, and evaluation protocols. We introduce LCWM, a shared task that maps an initial agent-view observation and a language instruction to an action sequence that fulfills the instruction. Its accompanying benchmark, LCWM-Bench, integrates 15 datasets across navigation, robotics, 3D gaming, and driving under a shared action interface and evaluation protocol, pairing 142,987 episodes with 428,961 instructions in three styles. We further propose Tiresias, an agent that plans via selective imagination: a language-conditioned diffusion policy generates candidate trajectories, a trajectory ranker shortlists them, and a video-pretrained world model imagines their visual consequences. A visual ranker evaluates these rollouts, complementing trajectory scores to select the final plan. Tiresias achieves competitive planning and imagination performance on LCWM-Bench. Ablations demonstrate the complementary benefits of trajectory ranking and visual verification, while experiments with alternative proposal methods show that the trained rankers remain effective beyond our diffusion policy. Together, LCWM-Bench and Tiresias provide a unified evaluation setting and an effective starting point for studying language-guided planning through world models.
