Latent Dynamics · 2026-10-14
From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
Shenghua Lu, Zixuan Yang, Rongxu Zhao, Bohao Qu, Jifeng Hu, Hechang Chen
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
MeanFlow2Better (MF2B) reformulates offline RL as generative decision-chunk optimization in a high-level latent decision space. It uses a chunk-level latent decision encoder plus a shared conditional mean velocity field with two branches: Gaussian sources to positive behavioral regions and suboptimal decisions to high-value regions. Refinement is done in one-step Euler integration over Δt=1 (no multi-step iterative sampling/backprop through time).
Why it matters
The authors aim to avoid two common issues of generative offline policies: high inference cost from multi-step diffusion/flow sampling and poor utilization of suboptimal trajectories common in offline datasets. MF2B’s one-step mean-velocity design and dual-branch, value-guided latent decision flow target both issues while compressing sequences from T to K≈T/H.
Method
- Reformulates offline RL as generative optimization over high-level latent decision representations from trajectory chunks (encoder maps chunks to latent decisions; decoder reconstructs action sequences).
- Learns a shared conditional latent decision mean-velocity field with two branches: Source-to-Positive Alignment (Gaussian-Induced Decision Sources to positive behavioral regions) and Quality-Uplift (suboptimal decisions to high-value decisions) guided by hierarchical chunk-level advantage.
- Uses one-step refinement: predicts conditional mean velocity over the unit interval Δt=1 and completes refinement via one-step Euler integration, avoiding multi-step iterative sampling and backpropagation through time.
Limitation
The high-level latent decision representations of MF2B are constrained by the diversity of the offline dataset, and excessively narrow behavioral support may limit the quality of the learned decision space.
Abstract (from arXiv)
Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets. We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. MF2B maps trajectory chunks to high-level latent decision representations through an encoder, placing generation starting points within a structured decision space sculpted by behavioral data. The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions. Both flow mappings complete decision refinement through one-step ODE integration over the unit time interval $\Delta t=1$, requiring neither multi-step iterative sampling nor backpropagation through time. Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from $T$ to $K \approx T/H$ and significantly lower computational overhead. On the D4RL Gym-MuJoCo benchmark, MF2B achieves an average score of 90.91, matching or surpassing strong baselines.
Related papers
- VPFlow: Value-Predictive Planning with Imperfect World Models for Offline Goal Reaching
- In-Distribution Imagination for Model-Based Offline Reinforcement Learning
- Trajectory-Aware Policy Optimization: Closing the State Distribution Gap in Long-Horizon LLM RL
- Higher-Order Action Supervision Makes A Strong Policy Class
- Tree-Aware Posterior Distillation for Long-Horizon Agentic Reinforcement Learning
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-12 · Indexed by Pub-AI: 2026-10-14 UTC
From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
Preprint posted: 2026-10-12 · Original record ↗
- arxiv
- 2610.16984
Explicit source versions (1)
- v1 · 2026-10-12
- Preprint available
Source date: 2026-10-12 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Offline reinforcement learning aims to learn effective policies from fixed datasets without environment interaction. Although generative models such as diffusion and flow matching enhance policy expressiveness, they typically start from uninformative Gaussian noise and generate actions through multi-step iteration, leading to high computational costs and failing to effectively leverage suboptimal trajectories in datasets. We propose MeanFlow2Better (MF2B), which reformulates offline RL as generative decision chunk optimization from current performance status to better performance status in a high-level latent decision space. MF2B maps trajectory chunks to high-level latent decision representations through an encoder, placing generation starting points within a structured decision space sculpted by behavioral data. The model learns a shared conditional mean velocity field in this space, constructing two flow mappings: from Gaussian-Induced Decision Sources toward positive behavioral decision regions, and from suboptimal decisions toward high-value decision regions. Both flow mappings complete decision refinement through one-step ODE integration over the unit time interval $Δt=1$, requiring neither multi-step iterative sampling nor backpropagation through time. Furthermore, we propose hierarchical chunk-level advantage to guide the flow direction toward high-return regions, where the high-level decisions cover multiple low-level actions, leading to sequence length compression from $T$ to $K \approx T/H$ and significantly lower computational overhead. On the D4RL Gym-MuJoCo benchmark, MF2B achieves an average score of 90.91, matching or surpassing strong baselines.