ClothCode: Generative Modeling for Garment State Estimation and Dynamics Prediction

TL;DR

ClothCode is a unified discrete generative framework for garment state estimation and action-conditioned dynamics prediction. It tokenizes cloth geometry into material-aligned residual VQ patch tokens and uses task-specific Masked Generative Transformers for confidence-guided parallel masked refinement. Across three templates, the authors report 26.6% lower mean displacement error for state estimation and 37.0% for dynamics prediction vs strongest baselines, plus faster dynamics inference (16.7 ms vs 556.2 ms) and improved real-world MPC success (56.7%→78.9%).

Why it matters

The authors aim to make garment perception and dynamics models accurate enough for repeated planning queries under occlusion and complex contact. By using a shared discrete configuration space and masked iterative refinement, ClothCode targets both lower prediction error and lower inference cost during planning, and it is tested in a model-predictive control loop on a real dual-arm robot.

Method

• Encode garment configurations in a shared discrete space using material-aligned patches and two-level residual vector quantization (RVQ) tokens; decode with a shared topology-aware mesh decoder.
• For state estimation, a masked transformer predicts complete garment tokens from partial point clouds and predicts centroid translation separately via task-specific translation heads.
• For dynamics prediction, an action-conditioned masked transformer predicts future token sequences; confidence-guided mask-and-refine is used for iterative parallel refinement, enabling MPC rollout with lower per-window cost.

Limitation

The paper states future work will investigate transferable discrete garment representations across unseen garment topologies and their use in longer-horizon closed-loop manipulation.

Related papers

• In-Context Learning World Models for Robot Dual-Arm Manipulation of Soft Garment with Partial Observations
• Being-M0.7: A Latent World-Action Model for Humanoid Robots
• AffordCraft: Scalable Construction of Task-Ready Simulation Assets from Single Images
• Language-Forcing: Advancing Humanoid Loco-Manipulation via Semantic Alignment of Whole-Body World Action Models
• Ropedia W0: Scaling Loco-Manipulation World Models across Embodiments with Human Experience