Latent Dynamics · 2026-11-13

Latent Imagination for Parameterized Action Spaces

Atahan Cilan, \.Ihsan Mert Muhac{\i}ro\u{g}lu, \"Umit Can Bekar

arXiv:2611.15744PDF

AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.

TL;DR

HyLI is a model-based PAMDP agent that learns a hybrid mode-and-parameter policy entirely in the latent imagination of a DreamerV3-style world model and acts with a single forward pass. It uses counterfactual mode evaluation (CME) and success-window replay (SWR) to address PAMDP-specific learning issues. On seven PAMDP benchmarks, it matches or nears the strongest model-free baseline and is much faster than the planner DLPA.

Why it matters

Parameterized-action MDPs require choosing a discrete mode and continuous parameters that depend on the mode. HyLI targets the common trade-off in model-based PAMDP methods: sample efficiency versus slow action planning at decision time. By training in latent imagination and using CME/SWR, the authors aim to keep sample efficiency while making decisions fast enough for real-time control.

Method

  • Trains a hybrid actor (discrete mode head + per-mode parameter heads) on imagined trajectories from a DreamerV3 RSSM, acting with one forward pass at deployment.
  • Uses counterfactual mode evaluation (CME) to evaluate every (or top-probability) candidate mode with one imagined step per candidate from each imagined state, training the mode head via expected policy gradient and each parameter head with its own value.
  • Uses success-window replay (SWR): draws a fraction of each batch from windows containing a success event to prevent rare early successes from being diluted.

Limitation

The candidate set for CME is all modes when K ≤ 16, otherwise it uses the 8 most probable plus 8 uniformly sampled modes with renormalized probabilities, described as a biased approximation whose cost does not grow with K.

Abstract (from arXiv)

Parameterized-action MDPs (PAMDPs) ask an agent to choose a discrete action together with continuous parameters whose meaning depends on that choice. Model-free PAMDP methods act fast but need many interactions; model-based methods are sample-efficient but plan at every decision, which makes them slow to act. We introduce HyLI (Hybrid Latent Imagination), a model-based agent that learns a hybrid policy entirely in the latent imagination of a world model and acts with a single forward pass. To address learning problems that the PAMDP structure creates, HyLI adds counterfactual mode evaluation, which gives every discrete action a learning signal at every imagined state, and success-window replay, which keeps rare successes in training. On seven standard PAMDP benchmarks, HyLI is within one published standard deviation of, or above, the published score of the strongest model-free baseline in our comparison on all 7 tasks, reaches 90% of that score with 2.4x fewer environment steps than that baseline (5.0x when its reward-free pre-training is counted), and decides 67x faster than the model-based planner DLPA at its published planner settings. It is the only method in our comparison that is both more sample-efficient than that baseline and fast enough for real-time control.

Related papers

Publication record

Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.

First announced by arXiv: 2026-11-13 · Indexed by Pub-AI: 2026-11-13 UTC

preprint

Latent Imagination for Parameterized Action Spaces

Atahan Cilan, İhsan Mert Muhacıroğlu, Ümit Can Bekar

First announced by arXiv: 2026-11-13 · Original record ↗

arxiv
2611.15744
Explicit source versions (1)
  • v1 · Date not supplied
  1. Preprint available

    Source date: 2026-11-13 · First detected: 2026-11-13 UTC · Historical baseline

    Source evidence (1)
    • Official evidence ↗ · RSS guid=oai:arXiv.org:2611.15744v1; arxiv:announce_type=new; item pubDate (official metadata)
Abstract · arxiv

Parameterized-action MDPs (PAMDPs) ask an agent to choose a discrete action together with continuous parameters whose meaning depends on that choice. Model-free PAMDP methods act fast but need many interactions; model-based methods are sample-efficient but plan at every decision, which makes them slow to act. We introduce HyLI (Hybrid Latent Imagination), a model-based agent that learns a hybrid policy entirely in the latent imagination of a world model and acts with a single forward pass. To address learning problems that the PAMDP structure creates, HyLI adds counterfactual mode evaluation, which gives every discrete action a learning signal at every imagined state, and success-window replay, which keeps rare successes in training. On seven standard PAMDP benchmarks, HyLI is within one published standard deviation of, or above, the published score of the strongest model-free baseline in our comparison on all 7 tasks, reaches 90% of that score with 2.4x fewer environment steps than that baseline (5.0x when its reward-free pre-training is counted), and decides 67x faster than the model-based planner DLPA at its published planner settings. It is the only method in our comparison that is both more sample-efficient than that baseline and fast enough for real-time control.