Unassigned · 2026-11-04

YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

Shengyu Wang, Tengyue Nong, Junming Liu, Haosong Peng, Siyu Li, Ruiqi Lei, Fushuo Huo, Yuhao Pan, Yunfeng Fan, Ding Wang, Nan Cheng, Haozhao Wang, Wenchao Xu

arXiv:2611.05280PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

YANAMI introduces YANAMI-400K (400K multi-turn multimodal dialogues with 2.0M+ images and 2.6M dialogue turns) and YANAMI-Bench (400 multi-turn dialogues) for multimodal stateful interaction. It evaluates Dialogue State Understanding (DSU) and Visual Realization (VR). The authors report that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction, while evaluations on YANAMI-Bench show existing UMMs struggle with long-horizon state maintenance and updates.

Why it matters

The paper targets a gap where prior multimodal multi-turn datasets often allow turns to be solved independently, providing limited supervision/evaluation for maintaining and updating evolving dialogue states across turns. By defining explicit state-management capabilities and evaluating DSU vs VR, YANAMI aims to support training and diagnosing models for truly stateful multimodal interaction spanning understanding, generation, and editing.

Method

  • Create YANAMI-400K with 400K multi-turn multimodal dialogues requiring continuous tracking, preservation, and revision of evolving multimodal dialogue states across four scenarios.
  • Incorporate six explicit state-management capabilities in each dialogue: coreference resolution, context retrieval, cross-turn consistency, spatial/physical consistency, constraint adherence, and correction-based state updating.
  • Evaluate models using a VLM-as-a-judge framework with two dimensions: Dialogue State Understanding (DSU) and Visual Realization (VR) (Visual Quality + State Consistency).

Limitation

The current version of YANAMI focuses on four representative interaction scenarios and six core state-management capabilities to provide a controlled foundation; future extensions may naturally incorporate additional application domains, languages, modalities, and longer interaction horizons, and complement the controlled evaluation with broader interactive studies.

Abstract (from arXiv)

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal stateful interaction, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce YANAMI, the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising YANAMI-400K and YANAMI-Bench. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates Dialogue State Understanding and Visual Realization, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-11-03 · Indexed by Pub-AI: 2026-11-04 UTC

preprint

YANAMI: A Large-Scale Multimodal Dataset and Benchmark for Multiturn Stateful Interaction

Shengyu Wang, Tengyue Nong, Junming Liu, Haosong Peng, Siyu Li, Ruiqi Lei, Fushuo Huo, Yuhao Pan, Yunfeng Fan, Ding Wang, Nan Cheng, Haozhao Wang, Wenchao Xu

Preprint posted: 2026-11-03 · Original record ↗

arxiv
2611.05280
Explicit source versions (1)
  • v1 · 2026-11-03
  1. Preprint available

    Source date: 2026-11-03 · First detected: 2026-11-04 UTC · Historical baseline

    Source evidence (2)
Abstract · arxiv

Unified Multimodal Models (UMMs) integrate visual understanding, generation, and editing within a single unified framework. However, existing unified multimodal datasets and benchmarks remain dominated by single-turn tasks. Even in recent multi-turn datasets, many turns can be solved independently without tracking information across turns. Consequently, they provide limited supervision and evaluation for multimodal stateful interaction, which requires models to understand, maintain, update, and faithfully realize evolving dialogue states across turns. To address this gap, we introduce YANAMI, the first large-scale multimodal dialogue dataset and benchmark for stateful interaction, comprising YANAMI-400K and YANAMI-Bench. YANAMI-400K contains 400K multi-turn multimodal dialogues, with more than 2M images and 2.6M dialogue turns. The dialogues span four representative scenarios: Multi-Image Fusion, Image Editing, World Modeling, and Visual Reasoning, and require models to continuously track, preserve, and revise evolving multimodal dialogue states across turns. YANAMI-Bench comprises 400 multi-turn dialogues for systematically evaluating multimodal stateful interaction. It jointly evaluates Dialogue State Understanding and Visual Realization, characterizing the gap between models' ability to understand evolving dialogue states and their ability to faithfully realize them in visual outputs. Experiments show that fine-tuning BAGEL on YANAMI-400K improves image editing, generation, and multimodal stateful interaction by 10.3%, 2.0%, and 13.23%, respectively, while preserving its general visual understanding capabilities. Evaluations on YANAMI-Bench further reveal that existing UMMs still struggle to maintain, update, and visually realize evolving multimodal dialogue states.