Driving · 2026-10-14
Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation
Haoyu Zhang, Siming Li, Zeyu Zhang, Yi Zhang, Tat-Seng Chua, Hao Tang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
Code2Worlds-2 extends Code2Worlds to produce executable, simulator-compatible embodied 4D worlds. It appends (1) a SimReady Scene Compiler that adds collision geometry, estimates physical attributes, infers a support graph, applies contact/dynamics constraints, and gravity-settles before exporting MuJoCo MJCF, and (2) a Simulator-in-the-Loop refiner that converts visual dynamics into executable rigid/articulated dynamics and refines using simulator critics. It introduces the SimCode4D benchmark.
Why it matters
The authors argue that coding-LLM-generated 4D worlds are visually plausible but not physically executable: they lack collision geometry, mass/friction/inertia, and joint structure, so a physics engine can’t resolve contact meaningfully. By compiling into MuJoCo MJCF and refining with simulator-signal feedback, Code2Worlds-2 aims to enable physical plausibility, simulator compatibility, and robot-interactable scenes for embodied tasks.
Method
- SimReady Scene Compiler (Module 1): convex-decomposes colliders, estimates physical attributes with a vision-language model, infers a lightweight support graph, instantiates MuJoCo contact/friction/dynamics constraints, runs gravity settling, then exports as MuJoCo MJCF.
- Embodied World Graph: an engine-agnostic intermediate representation with per-object physical fields (mass, friction, inertia, COM) and per-dynamics event signals/constraints; compiled once then serialized to MuJoCo MJCF for execution.
- Simulator-in-the-Loop 4D Refiner (Module 2): compiles the Post-Process Agent’s visual dynamics into executable rigid/articulated motion, logs simulator rollout signals, and critiques via penetration/contact/stability/forces/joint limits/reachability plus a VLM visual critic fused into a single verdict for refining physical settings.
Abstract (from arXiv)
Recent coding large language models (LLMs) can synthesize dynamic 4D worlds from natural language, but the resulting content remains a visual artifact. It lacks physical semantics such as the collision geometry, mass, friction, inertia, and joint structure, required to enable it to simulate the physical world, and its dynamics are scripted to look plausible rather than designed to execute. What is needed is an executable world: one that a physics engine can load and step, resolving contact physically. We present Code2Worlds-2, an extension of Code2Worlds that turns 4D world generation into an embodied world simulation. We keep the original dual-stream generation pipeline intact and append two modules. First, a SimReady Scene Compiler that turns a generated visual scene into a simulator-compatible representation: it assigns collision meshes via convex decomposition, estimates physical attributes with a vision-language model, infers a lightweight support graph, instantiates the engine's contact, friction, and dynamics constraints, and applies a gravity settling pass, before exporting the scene as MuJoCo MJCF. Second, a Simulator-in-the-Loop 4D Refiner that implements an automated agent harness-a compile, simulate, critique and refine loop-that converts visual dynamics into executable rigid and articulated dynamics and evolves the physical settings rather than only the code, replacing the purely visual motion critic with a simulator critic that inspects penetration depth, collision pairs, stability, contact forces, joint limits, and robot reachability. We further propose SimCode4D, a state-based benchmark with five task templates for evaluating embodied simulation. Experiments on SimCode4D and across natural, indoor, industrial, urban, and service scenes show that Code2Worlds-2 produces physically plausible, simulator-compatible, and robot-interactable scenes.
Related papers
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
- ExeWorldBench: Benchmarking Code-Based Executable 3D World Modeling
- EmbodiedWorld-200K: Toward Embodied World Agents via Embodied-Planning Dataset and Interactive World Models
- From Pixels to Executable Physics: Behavior-Grounded World Models for Embodied Action Consequence Prediction
- Awomo-SimDataEngine: Agentic Simulation-ReadyWorld Generation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-13 · Indexed by Pub-AI: 2026-10-14 UTC
Code2Worlds-2: Physically Plausible Agent Harness for Embodied World Simulation
Preprint posted: 2026-10-13 · Original record ↗
- arxiv
- 2610.17651
Explicit source versions (1)
- v1 · 2026-10-13
- Preprint available
Source date: 2026-10-13 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent coding large language models (LLMs) can synthesize dynamic 4D worlds from natural language, but the resulting content remains a visual artifact. It lacks physical semantics such as the collision geometry, mass, friction, inertia, and joint structure, required to enable it to simulate the physical world, and its dynamics are scripted to look plausible rather than designed to execute. What is needed is an executable world: one that a physics engine can load and step, resolving contact physically. We present Code2Worlds-2, an extension of Code2Worlds that turns 4D world generation into an embodied world simulation. We keep the original dual-stream generation pipeline intact and append two modules. First, a SimReady Scene Compiler that turns a generated visual scene into a simulator-compatible representation: it assigns collision meshes via convex decomposition, estimates physical attributes with a vision-language model, infers a lightweight support graph, instantiates the engine's contact, friction, and dynamics constraints, and applies a gravity settling pass, before exporting the scene as MuJoCo MJCF. Second, a Simulator-in-the-Loop 4D Refiner that implements an automated agent harness-a compile, simulate, critique and refine loop-that converts visual dynamics into executable rigid and articulated dynamics and evolves the physical settings rather than only the code, replacing the purely visual motion critic with a simulator critic that inspects penetration depth, collision pairs, stability, contact forces, joint limits, and robot reachability. We further propose SimCode4D, a state-based benchmark with five task templates for evaluating embodied simulation. Experiments on SimCode4D and across natural, indoor, industrial, urban, and service scenes show that Code2Worlds-2 produces physically plausible, simulator-compatible, and robot-interactable scenes.