Driving · 2026-10-30
LLM-guided Planar Loco-Manipulation for Quadruped Robots
Georges Jetti, Michael Khayyat, Jean-Pierre Sleiman, Stefano Arrigoni, Francesco Braghin
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
The authors propose an LLM-guided event-driven planner for interactive navigation with movable obstacles: it jointly selects a skill (navigation or planar manipulation), the relevant object, and a geometric target, producing a variable-length sequence of parameterized actions executed by low-level PPO policies. They fine-tune Qwen2.5-1.5B-Instruct with LoRA on synthetic data and report improved interactive navigation over baselines, plus an end-to-end whole-body planar pushing policy for manipulation.
Why it matters
Interactive navigation among movable obstacles can require physical interaction when no collision-free path exists. The paper targets this by expanding decision space beyond low-dimensional commands or fixed action sets: selecting skill, object, and geometric target lets the system reason about scene geometry. The end-to-end planar pushing policy aims to improve manipulation success and reduce completion time versus a hierarchical approach.
Method
- A language-conditioned, event-driven high-level planner maps a natural-language task and structured scene representation to a variable-length sequence of parameterized actions, predicting remaining skill horizon and emitting only the first action each time.
- The action representation is (skill, relevant object identifier when applicable, geometric target), enabling explicit control over which object is manipulated and where it should be moved.
- Low-level skills are trained with PPO in Isaac Lab with domain randomization; the manipulation skill is an end-to-end whole-body planar pushing policy using an asymmetric actor-critic architecture.
Limitation
The authors state that to extend the learned navigation and manipulation behaviors to a longer task structure, failures become more frequent and are particularly concentrated around violating action and environment constraints implicitly represented in the training data.
Abstract (from arXiv)
Robotic navigation typically assumes a collision-free route to the goal exists; in cluttered environments with movable obstacles, however, reaching the goal may require physical interaction. Classical solutions encode these behaviors through geometric rules and task-specific logic, while learning-based approaches often constrain the decision space to low-dimensional motion commands or predefined action sets. We instead consider a broader decision space, jointly selecting the skill, relevant object, and geometric target. Our language-conditioned task planner, fine-tuned with LoRA on synthetic data, maps natural-language tasks and a structured scene representation to variable-length sequences of parameterized actions, executed by low-level reinforcement-learning policies. The planner learns the geometric relations underlying interactive navigation and outperforms a classical planner and a prompted frontier model on interactive navigation tasks. Comparisons with models trained from scratch indicate that composition does not require pretraining when the representation is structured, whereas pretraining yields more reliable geometric targets. At the low level, we introduce an end-to-end whole-body planar pushing policy that achieves a 93.3% success rate in simulation, versus 83.5% for a hierarchical baseline, while reducing completion time by more than 60%. On hardware, the hierarchy yields the correct skill composition in 44/45 trials and end-to-end success in 34/45. Project website: https://anonymousprojects-dev.github.io/LLM-guided-LocoManipulation-Quadrupeds/
Related papers
- Action-Aligned Latent Planning for Versatile Humanoid Loco-Manipulation
- LocoWM: High-Precision Locomotion through World-Model-Guided Residual Adaptation
- PILOT: Guiding Long-Horizon World-Model Planning for Robot Manipulation with Visual Subgoals and Policy Priors
- Coordinating Humanoid Loco-Manipulation through Action-Conditioned Latent Foresight
- Vector-Quantized World Model: Latent Feedback Motion Planning for Robot Manipulation
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-29 · Indexed by Pub-AI: 2026-10-30 UTC
LLM-guided Planar Loco-Manipulation for Quadruped Robots
Preprint posted: 2026-10-29 · Original record ↗
- arxiv
- 2610.40248
Explicit source versions (1)
- v1 · 2026-10-29
- Preprint available
Source date: 2026-10-29 · First detected: 2026-10-30 UTC · Historical baseline
Source evidence (2)
- Official evidence ↗ · RSS guid=oai:arXiv.org:2610.40248v1; arxiv:announce_type=new; item pubDate (official metadata)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Robotic navigation typically assumes a collision-free route to the goal exists; in cluttered environments with movable obstacles, however, reaching the goal may require physical interaction. Classical solutions encode these behaviors through geometric rules and task-specific logic, while learning-based approaches often constrain the decision space to low-dimensional motion commands or predefined action sets. We instead consider a broader decision space, jointly selecting the skill, relevant object, and geometric target. Our language-conditioned task planner, fine-tuned with LoRA on synthetic data, maps natural-language tasks and a structured scene representation to variable-length sequences of parameterized actions, executed by low-level reinforcement-learning policies. The planner learns the geometric relations underlying interactive navigation and outperforms a classical planner and a prompted frontier model on interactive navigation tasks. Comparisons with models trained from scratch indicate that composition does not require pretraining when the representation is structured, whereas pretraining yields more reliable geometric targets. At the low level, we introduce an end-to-end whole-body planar pushing policy that achieves a 93.3% success rate in simulation, versus 83.5% for a hierarchical baseline, while reducing completion time by more than 60%. On hardware, the hierarchy yields the correct skill composition in 44/45 trials and end-to-end success in 34/45. Project website: https://anonymousprojects-dev.github.io/LLM-guided-LocoManipulation-Quadrupeds/
