Latent Dynamics · 2026-10-06
LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations
Ziyu Tian, Kaichen Shen, Wenbo Hao, Phillip Si, Peng Chen, Wei Zhu
AI-assisted summary based on the paper. Consult the linked source for methods and limitations. Benchmark numbers appear after editor verification.
TL;DR
LD-EnFF is a sequential Bayesian data-assimilation framework that performs forecast propagation and filtering updates in a compact augmented latent space. It uses LDNet for latent, parameter-conditioned dynamics and reconstruction, and combines a VAE-based latent observation encoder with a learned forward y⋆ operator to build a state-dependent latent likelihood. An ensemble flow-matching update (EnFF) generates posterior samples, jointly updating latent states and uncertain parameters, avoiding full-state simulation and repeated full-field likelihood evaluation.
Why it matters
The authors aim to address two coupled bottlenecks in data assimilation with sparse/noisy observations: (1) computational cost of repeated high-dimensional forward simulation, and (2) limited information from sparse observations about the full state. LD-EnFF targets both by moving online forecasting and likelihood evaluation into a learned latent space and using an explicit learned forward observation model for sparse measurements, enabling joint estimation of latent states and an uncertain parameter.
Method
- Train LDNet to learn uncertain-parameter-conditioned latent dynamics (with an explicit Euler advance) and a reconstruction map from latent states to the physical state.
- Train a variational observation encoder (VAE) and a learned forward y⋆ operator Hα that maps the augmented latent state (latent state + uncertain parameter) directly to the corresponding noiseless sparse physical observation.
- Online: advance a latent ensemble with LDNet; build a state-dependent local Gaussian latent-observation likelihood via the VAE; apply EnFF (flow matching with likelihood-based guidance) to obtain posterior samples for joint latent state and uncertain-parameter updates.
Abstract (from arXiv)
Data assimilation combines model forecasts with noisy, incomplete observations to estimate the evolving state of a dynamical system. Existing methods face two compounding challenges: high-dimensional nonlinear dynamics make repeated forward simulation computationally expensive, while sparse observations provide limited direct information about the full state. To address these challenges, we propose the Latent-Dynamics Ensemble Flow Filter (LD-EnFF), a sequential Bayesian filtering framework that performs both forecast propagation and filtering updates in a compact latent space. LD-EnFF combines a latent dynamics surrogate for ensemble propagation with a variational autoencoder (VAE)-based observation model that evaluates a state-dependent observation likelihood in latent space. At each assimilation step, an ensemble filtering update based on flow matching uses the forecast ensemble and this likelihood to generate posterior samples, jointly updating latent states and uncertain parameters. This design avoids repeated full-state simulation during forecasting and full-field reconstruction during likelihood evaluation. LD-EnFF substantially outperforms a broad range of data assimilation algorithms on benchmarks spanning Kolmogorov flow, tsunami propagation, and atmospheric modeling, all featuring complex dynamics and sparse, noisy observations.
Related papers
- SMORE: Stability-Promoting Mesh-Agnostic Model Reduction for Time-Dependent PDEs
- Variational Streaming Flow: Probabilistic Forecasting in Physical Time
- Learning PDE solution operators with variable initial conditions via Latent Dynamics Networks
- Mesh-Free Convolution: Learned Spectral Attenuation and Transport
- Correct then Forecast: Observer State-Space Models for Time Series Forecasting
Publication record
Bibliographic evidence describes where and when a record appeared. It does not validate scientific claims.
First public release: 2026-10-02 · Indexed by Pub-AI: 2026-10-06 UTC
LD-EnFF: Latent-Dynamics Ensemble Flow Filtering for Data Assimilation with Sparse Observations
Preprint posted: 2026-10-02 · Original record ↗
- arxiv
- 2610.04034
Explicit source versions (1)
- v1 · 2026-10-02
- Preprint available
Source date: 2026-10-02 · First detected: 2026-10-25 UTC · Historical baseline
Source evidence (1)
- Official evidence ↗ · atom:published (official metadata)
Abstract · arxiv
Data assimilation combines model forecasts with noisy, incomplete observations to estimate the evolving state of a dynamical system. Existing methods face two compounding challenges: high-dimensional nonlinear dynamics make repeated forward simulation computationally expensive, while sparse observations provide limited direct information about the full state. To address these challenges, we propose the Latent-Dynamics Ensemble Flow Filter (LD-EnFF), a sequential Bayesian filtering framework that performs both forecast propagation and filtering updates in a compact latent space. LD-EnFF combines a latent dynamics surrogate for ensemble propagation with a variational autoencoder (VAE)-based observation model that evaluates a state-dependent observation likelihood in latent space. At each assimilation step, an ensemble filtering update based on flow matching uses the forecast ensemble and this likelihood to generate posterior samples, jointly updating latent states and uncertain parameters. This design avoids repeated full-state simulation during forecasting and full-field reconstruction during likelihood evaluation. LD-EnFF substantially outperforms a broad range of data assimilation algorithms on benchmarks spanning Kolmogorov flow, tsunami propagation, and atmospheric modeling, all featuring complex dynamics and sparse, noisy observations.