Video World Models · 2026-11-05
Touch\'{e}World tech report : Towards Online Unified Camera--Hand Exploration in Generative Worlds
Jiaao Yu, Yanjian Zhang, Haoyang Jiang, Guillaume Wisniewski, Nadi Tomeh, Shijie Jia
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
TouchéWorld integrates continuous hand interaction into a camera-controlled video world model for online exploration. Given an initial image and a scene-description prompt, it generates video conditioned on temporally aligned camera and hand controls, using joint camera–hand data construction, parameter-efficient hand conditioning, causal adaptation, and few-step distillation for chunk-level online generation. The authors introduce TouchéBench to evaluate hand following, camera–hand coordination, online interaction, and object response.
Why it matters
The paper targets online unified camera–hand exploration in generative worlds by modeling both navigation (camera) and manipulation (continuous hand motion) with causal, chunk-level updates. TouchéBench is designed to separate control fidelity (hand following, coordination, online intervention) from action-dependent object response, and includes reference-based evaluation on held-out HOI4D plus purpose-recorded scenarios.
Method
- Joint camera–hand data construction from heterogeneous egocentric videos: annotation completion, consistency-based filtering for camera/hand estimates, and temporal alignment with updated intrinsics and image-space coordinates.
- Conditioning and adaptation: build on a pretrained camera-controlled image-to-video backbone; add spatial hand-skeleton conditioning and numerical hand-motion features via parameter-efficient adaptation (hand LoRA) with hand-weighted denoising loss.
- Online causal generation: teacher-forced causal adaptation and asymmetric Distribution Matching Distillation to produce a few-step causal generator; online inference generates chunks sequentially with updated camera/hand controls.
Abstract (from arXiv)
Recent video world models allow users to explore generated environments through camera navigation. Continuous hand motion provides a fine-grained action channel for interacting with the people and objects encountered during exploration. We present Touch\'{e}World, a framework that integrates gesture-driven interaction into a camera-controlled video world model. Given an initial image and a scene-description prompt, the model generates video conditioned on temporally aligned camera and hand controls. We construct joint camera--hand training data from heterogeneous egocentric videos through annotation completion, consistency-based filtering, and temporal alignment. Building on a pretrained camera-controlled world model, we combine spatial hand-skeleton conditioning with numerical hand-motion features through parameter-efficient adaptation. Causal adaptation and few-step distillation support chunk-level online generation, allowing users to update their controls based on generated visual feedback. We further introduce Touch\'{e}Bench to evaluate hand-motion following, camera--hand coordination, online interaction, and object response. Together, these components establish a framework for studying continuous exploration and hand-driven interaction within generative worlds.
Related papers
- Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
- Dexterous Tactile World Model
- UNITAS: A 3D-Native World Action Model for Embodied Manipulation
- UMI-WM: Interactive World Modeling for Bimanual Manipulation from Wrist Cameras
- PointWAM: 3D World Action Modeling for Dexterous Robotic Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-11-04 · Indexed by Pub-AI: 2026-11-05 UTC
TouchéWorld tech report : Towards Online Unified Camera--Hand Exploration in Generative Worlds
Preprint posted: 2026-11-04 · Original record ↗
- arxiv
- 2611.06061
Explicit source versions (1)
- v1 · 2026-11-04
- Preprint available
Source date: 2026-11-04 · First detected: 2026-11-05 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2611.06061v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Recent video world models allow users to explore generated environments through camera navigation. Continuous hand motion provides a fine-grained action channel for interacting with the people and objects encountered during exploration. We present TouchéWorld, a framework that integrates gesture-driven interaction into a camera-controlled video world model. Given an initial image and a scene-description prompt, the model generates video conditioned on temporally aligned camera and hand controls. We construct joint camera--hand training data from heterogeneous egocentric videos through annotation completion, consistency-based filtering, and temporal alignment. Building on a pretrained camera-controlled world model, we combine spatial hand-skeleton conditioning with numerical hand-motion features through parameter-efficient adaptation. Causal adaptation and few-step distillation support chunk-level online generation, allowing users to update their controls based on generated visual feedback. We further introduce TouchéBench to evaluate hand-motion following, camera--hand coordination, online interaction, and object response. Together, these components establish a framework for studying continuous exploration and hand-driven interaction within generative worlds.
