Video World Models · 2026-10-14
LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures
Yuming Fu, Weijia Wu, Jing Chen, Jiahao Tang, Feifei Chen, Hongyu Zhu, Xin Jin, Alex Jinpeng Wang
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors introduce LabInstruct, a benchmark for situated instructional video generation in real laboratories. Given an initial workspace image and a natural-language instruction, a model must generate a continuation that preserves workspace identity, executes the instructed actions in correct order, and reaches expected intermediate/final states. Evaluating 8 image-to-video models, the authors report that visually plausible generations often remain procedurally incorrect, showing a gap between visual realism and procedural reliability.
Why it matters
Self-driving laboratories need AI guidance that humans (or robots) can reliably follow to perform real procedures. LabInstruct directly evaluates procedural correctness (actions, causal order, state transitions, and outcomes) beyond visual fidelity, using real reference executions and structured annotations, making it easier to quantify whether video generators can produce instructionally reliable laboratory demonstrations.
Method
- Introduce LabInstruct: for each task, start from an initial workspace observation f0 and a natural-language instruction T, and require generation of f1:N that preserves workspace continuity/identity, executes instructed actions in correct order, and reaches expected intermediate and final states.
- Construct tasks from real lab procedure videos (Bilibili, YouTube, FineBio, ExpVid), with human-verified structured specifications annotated with objects, ordered actions, contacts, and state transitions; generate natural-language instructions from the verified specifications.
- Evaluate using a decoupled protocol: Core Visual Quality (object consistency, action fidelity, state correctness, physical plausibility) plus Critical completion with scene consistency and visual safety as validity gates; use GPT-5.6 Sol and human evaluation with a VLM-checklist workflow.
Limitation
Publicly available laboratory videos that satisfy their curation criteria remain limited; the benchmark is a relatively clean and controlled subset and the source distribution and coverage across disciplines/operation types are imbalanced. They also evaluate one generated sample per model–task pair under fixed generation settings in the main benchmark.
Abstract (from arXiv)
Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution. Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups. Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance. Because laboratory procedures are inherently visual, spatial, and dynamic, video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes. Recent advances in video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-language instruction. However, whether such models can reliably communicate real laboratory procedures has not been systematically studied. We introduce LabInstruct, a benchmark for situated instructional video generation in real laboratories. LabInstruct contains 204 tasks across 5 scientific disciplines, with real reference executions and structured annotations of objects, actions, contacts, and state transitions. Evaluating 8 frontier image-to-video models, we find that visually plausible generations frequently remain procedurally incorrect, revealing a substantial gap between visual realism and the reliability required for experimental instruction.
Related papers
- CASTER: Formalizing Zero-Shot Manipulation from Synthetic Videos as Constrained Optimization
- World Models' Last Exam in Physics
- Ego2Act: Evaluating Goal-Directed Manipulation in Egocentric Video Generation
- Real2Gym: Building Gyms from Videos, Bringing Skills to Robots
- Video2World: Benchmarking Coding Agents for Interactive World Modeling from Embodied Videos
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-09-29 · Indexed by Pub-AI: 2026-10-14 UTC
LabInstruct: Benchmarking Situated Instructional Video Generation for Lab Procedures
Preprint posted: 2026-09-29 · Original record ↗
- arxiv
- 2610.14020
Explicit source versions (1)
- v1 · 2026-09-29
- Preprint available
Source date: 2026-09-29 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Self-driving laboratories (SDLs) aim to automate the full experimental loop, from scientific decision-making to physical execution. Ideally, AI-generated plans could be carried out directly by robotic systems, but reliable automation remains difficult in complex, open-world laboratory environments, where experiments often involve fine-grained manipulation, long-horizon procedures, and substantial variation across tasks and setups. Humans therefore remain an important execution interface between AI-generated plans and physical experiments, creating a need for clear and effective human-facing experimental guidance. Because laboratory procedures are inherently visual, spatial, and dynamic, video is particularly well suited to communicating apparatus configurations, manipulation actions, temporal dependencies, and state changes. Recent advances in video generation now make it possible to synthesize experimental demonstrations directly from an initial workspace image and a natural-language instruction. However, whether such models can reliably communicate real laboratory procedures has not been systematically studied. We introduce LabInstruct, a benchmark for situated instructional video generation in real laboratories. LabInstruct contains 204 tasks across 5 scientific disciplines, with real reference executions and structured annotations of objects, actions, contacts, and state transitions. Evaluating 8 frontier image-to-video models, we find that visually plausible generations frequently remain procedurally incorrect, revealing a substantial gap between visual realism and the reliability required for experimental instruction.
