Robotics · 2026-10-06

AffordCraft: Scalable Construction of Task-Ready Simulation Assets from Single Images

Haoyun Yang, Xueyang Zhou, Ziyi Xie, Yongchao Chen

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

AffordCraft builds task-ready articulated simulation assets from a single RGB image and a task instruction by retrieving and adapting a matching entry from a library, then validating it with a physical test. Without boxes/masks, it produces physically valid assets for 1,703/2,000 photographs (85.15%) across 31 categories, and 162/237 annotated objects on cluttered images pass after detection. Policies trained on scripted demonstrations complete constructed manipulation tasks.

Why it matters

Robot learning in simulation depends on simulator assets that include movable parts and valid physics. The authors argue prior image-to-asset methods often fail to settle or move correctly in simulation and are costly per image, while general agents require many calls. AffordCraft turns construction into retrieval and constrained adaptation of known mechanisms, then gates acceptance by a physical test, enabling scalable asset creation for policy training.

Method

Ground the task into (category c, part p, action a, motion family m); retrieve up to 20 library entries compatible with the requested mechanism and part, then select among candidates with a multimodal model.
Adapt the selected library entry by keeping its bodies and joints (including an articulation adapter) while estimating/setting pose, size, and physical parameters; collision shapes are built via a decomposition cascade.
Validate with a physical gate in simulation (export, inertia checks, settle under gravity, support contact, and joint family/part motion); perform repair once on failures and reselect up to 20 candidates per input.

Limitation

Most failures are attributed to the target not being localized: the paper states that 230 of 297 failures are inputs whose target is not localized (Figure 4).

Abstract (from arXiv)

Robot learning in simulation depends on the objects the simulator offers. Many tasks need objects with separate parts, joints that allow the required motion, and physical properties that remain valid under contact. Existing methods recover this structure anew for every image: generative models predict parts and joints that mostly fail to settle or move in simulation, and general-purpose agents need a long session of model calls for each photograph. AffordCraft builds such an asset from a single RGB image and a task instruction by retrieval instead of generation: it locates the object and the part to operate, selects a matching entry from a library of articulated assets, and fits it to the image while keeping its parts and joints intact. Without any box or mask marking the object, AffordCraft produces a physically valid asset for 1,703 of 2,000 photographs from 31 categories. Five generative methods pass on at most 45% of the same photographs and, at the median, need 10 to 78 times our GPU time per valid asset. On 50 cluttered images, 162 of 237 annotated objects pass the same physical test after automatic detection. Growing the library from 141 to 11,372 entries needs no change to the method and raises category coverage from 46% to 100% and the share of selections with the requested label from 18% to 51%. We also build manipulation tasks from the constructed assets, both with single objects and in composed scenes; policies trained on scripted demonstrations complete both kinds of tasks from initial states unseen in training.

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First public release: 2026-10-05 · Indexed by Pub-AI: 2026-10-06 UTC

preprint