Robotics · 2026-10-23

In-Context Learning World Models for Robot Dual-Arm Manipulation of Soft Garment with Partial Observations

Hengyi Sim, Quentin Rouxel, Dennis Hong, Darwin Caldwell, Fei Chen

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors present a Soft-Object World Model for dual-arm garment folding under partial observations. A transformer-based generative state estimator reconstructs full point-cloud states via conditional flow matching, and an action-conditioned dynamics predictor uses in-context learning to infer latent shape/material cues from a variable-length history without weight updates. A goal-conditioned planner uses the world model to score candidate action trajectories in simulation and real experiments.

Why it matters

The paper targets data efficiency and adaptation for soft-object manipulation under occlusion and material-dependent deformations. By combining generative full-state estimation from partial point clouds with in-context latent identification for action-conditioned rollouts, the approach aims to generalize across robot embodiments, materials, and garment types, then integrate into goal-conditioned planning for real bimanual garment folding.

Method

Generative full-state estimation: a transformer (ViT backbone) reconstructs dense 3D soft-object point-cloud states from partially observed point clouds using conditional flow matching.
Action-conditioned dynamics prediction with in-context learning: a transformer predicts future deformations from actions while conditioning on a variable-length context of estimated states to infer latent shape/material cues without gradient updates.
Planning integration: the world model generates forward rollouts for goal-conditioned candidate action trajectories and selects actions using an objective based on future-state scoring; real trials use CD/EMD on segmented point clouds.

Limitation

The current study focuses primarily on quasi-static folding, where material-dependent effects may be less pronounced than geometric and configuration-dependent cues.

Abstract (from arXiv)

Robust manipulation of soft objects, such as garments, is a critical skill for service and household robots to adopt, but remains challenging due to its high-dimensional state space, and material-dependent deformations. Humans address this challenge by using mental world models that "imagine" the shape and material properties from partial observations, which enables them to predict how actions affect future states. As such, embedding such world models in robots allows for better generalization in manipulating a wide variety of garments in the real world. However, learned world models typically require large amounts of real world data that is strenuous to collect, and a mechanism to infer latent properties that govern dynamics. We present a Soft-Object World Model that performs generative full-state estimation and action-conditioned dynamics prediction using a transformer backbone trained on large-scale simulated soft object interactions, with measures to mitigate the sim-to-real gap. By learning to infer latent shape and material cues through in-context learning over a variable-length partial point-cloud history, the model generalizes across robot embodiments, material parameters, garment types and shapes, and manipulation coordination modes without weight updates. The world model is then integrated with a Goal-Conditioned action planner for motion synthesis. We validate the approach in simulation and real world for bimanual garment folding, reporting improvements in reconstruction fidelity, prediction accuracy, and task success.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-22 · Indexed by Pub-AI: 2026-10-23 UTC

preprint