Latent Dynamics · 2026-11-04

Coordinating Humanoid Loco-Manipulation through Action-Conditioned Latent Foresight

Ju Dong, Xinyu Xiao, Yiming Zhou, Tianshuo Yang, Xuerong Liu, Lei Zhang, Jian Chen, Haocheng Zhao, Yu Fu, Yujie Wei, Diwen Zheng, Alois Christian Knoll, Angela P. Schoellig, Ping Luo, Jianwei Zhang

arXiv:2611.04839PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

The authors present action-conditioned latent foresight, a latent world–action model that scores sampled motion–hand command sequences before execution using predicted visual progress toward demonstration-derived subgoals and a learned joint temporal prior. The robot executes the first commands of the best-scoring sequence and replans each step. An action consistency regularization reconstructs the first demonstrated command from observed and predicted feature changes.

Why it matters

Humanoid loco-manipulation can fail even when a command sequence looks plausible. The proposed approach explicitly verifies multiple proposed action sequences in latent space using (1) progress toward task subgoals and (2) temporal consistency across commands, then uses receding-horizon replanning. The authors report improved closed-loop success in simulation (SIMPLE) and on real robot tasks, including unseen object configurations.

Method

  • Samples motion–hand command sequences from vision, language, and robot state; scores each sequence prefix using predicted visual progress toward a demonstration-derived subgoal plus a joint temporal prior. It executes the first K commands of the best sequence and replans from the next observation.
  • Action-conditioned endpoint predictor estimates visual features each prefix reaches; a subgoal tracker advances to the next subgoal when the cosine distance to the goal is below εg.
  • Action consistency regularization trains an inverse action decoder to reconstruct the first demonstrated command from observed and predicted feature changes, encouraging action-relevant information retention.

Limitation

The visual score relies on task-specific demonstration subgoal images.

Abstract (from arXiv)

Humanoid loco-manipulation requires coordinated body motion and hand interaction over time, and a plausible-looking command sequence can still drop a held object or leave an interaction incomplete. We present action-conditioned latent foresight, a latent world-action model (WAM) that evaluates proposed command sequences before execution. An action generator proposes joint motion-hand command sequences from vision, language, and the robot state. An action-conditioned endpoint predictor estimates the visual features each sequence prefix reaches and measures progress toward a demonstration-derived subgoal, while a joint temporal prior scores temporal consistency given the robot state and preceding commands. The robot executes the first commands of the best-scoring sequence and replans from the next observation. During training, action consistency regularization reconstructs the first demonstrated command from observed and predicted feature changes, encouraging the predictor to retain action-relevant information. On ten tasks in the humanoid loco-manipulation simulation benchmark SIMPLE, our method achieves 89.7% average success, compared with 73.7% for Psi-Zero and 82.0% for DiT4DiT. On seven real-robot tasks, it achieves 85.7% under standard conditions and 65.7% under unseen object configurations, while GR00T N1.7 achieves 70.0% and 48.6%, respectively. Project page: https://ju6276.github.io/Action-Grounded-Foresight/

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-04 · Indexed by Pub-AI: 2026-11-04 UTC

preprint

Coordinating Humanoid Loco-Manipulation through Action-Conditioned Latent Foresight

Ju Dong, Xinyu Xiao, Yiming Zhou, Tianshuo Yang, Xuerong Liu, Lei Zhang, Jian Chen, Haocheng Zhao, Yu Fu, Yujie Wei, Diwen Zheng, Alois Christian Knoll, Angela P. Schoellig, Ping Luo, Jianwei Zhang

First announced by arXiv: 2026-11-04 · Original record ↗

arxiv
2611.04839
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-04 · First detected: 2026-11-04 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.04839v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Humanoid loco-manipulation requires coordinated body motion and hand interaction over time, and a plausible-looking command sequence can still drop a held object or leave an interaction incomplete. We present action-conditioned latent foresight, a latent world-action model (WAM) that evaluates proposed command sequences before execution. An action generator proposes joint motion-hand command sequences from vision, language, and the robot state. An action-conditioned endpoint predictor estimates the visual features each sequence prefix reaches and measures progress toward a demonstration-derived subgoal, while a joint temporal prior scores temporal consistency given the robot state and preceding commands. The robot executes the first commands of the best-scoring sequence and replans from the next observation. During training, action consistency regularization reconstructs the first demonstrated command from observed and predicted feature changes, encouraging the predictor to retain action-relevant information. On ten tasks in the humanoid loco-manipulation simulation benchmark SIMPLE, our method achieves 89.7% average success, compared with 73.7% for Psi-Zero and 82.0% for DiT4DiT. On seven real-robot tasks, it achieves 85.7% under standard conditions and 65.7% under unseen object configurations, while GR00T N1.7 achieves 70.0% and 48.6%, respectively. Project page: https://ju6276.github.io/Action-Grounded-Foresight/