AIセキュリティポータル K Program
An Ensemble Score Filter for Tracking High-Dimensional Nonlinear Dynamical Systems
Share
Abstract
We propose an ensemble score filter (EnSF) for solving high-dimensional nonlinear filtering problems with superior accuracy. A major drawback of existing filtering methods, e.g., particle filters or ensemble Kalman filters, is the low accuracy in handling high-dimensional and highly nonlinear problems. EnSF attacks this challenge by exploiting the score-based diffusion model, defined in a pseudo-temporal domain, to characterizing the evolution of the filtering density. EnSF stores the information of the recursively updated filtering density function in the score function, instead of storing the information in a set of finite Monte Carlo samples (used in particle filters and ensemble Kalman filters). Unlike existing diffusion models that train neural networks to approximate the score function, we develop a training-free score estimation that uses a mini-batch-based Monte Carlo estimator to directly approximate the score function at any pseudo-spatial-temporal location, which provides sufficient accuracy in solving high-dimensional nonlinear problems as well as saves a tremendous amount of time spent on training neural networks. High-dimensional Lorenz-96 systems are used to demonstrate the performance of our method. EnSF provides surprising performance, compared with the state-of-the-art Local Ensemble Transform Kalman Filter method, in reliably and efficiently tracking extremely high-dimensional Lorenz systems (up to 1,000,000 dimensions) with highly nonlinear observation processes.
A multicase comparative assessment of the ensemble Kalman filter for assimilation of radar observations. Part I: Storm-scale analyses
A. Aksoy, D. Dowell, C. Snyder
Published: 2009
A multicase comparative assessment of the ensemble Kalman filter for assimilation of radar observations. Part II: Short-range ensemble forecasts
Published: 2010
Particle markov chain monte carlo methods
C. Andrieu, A. Doucet, R. Holenstein
Published: 2010
Backward sde filter for jump diffusion processes and its applications in material sciences
F. Bao, Y. Cao, P. Maksymovych
Published: 2020
A first order scheme for backward doubly stochastic differential equations
F. Bao, Y. Cao, A. Meir, W. Zhao
Published: 2016
A hybrid sparse-grid approach for nonlinear filtering problems based on adaptive-domain of the Zakai equation approximations
F. Bao, Y. Cao, C. Webster, G. Zhang
Published: 2014
Data assimilation of synthetic data as a novel strategy for predicting disease progression in alopecia areata
F. Bao, N. Cogan, A. Dobreva, R. Paus
Published: 2021
A score-based nonlinear filter for data assimilation
F. Bao, Z. Zhang, G. Zhang
Published: 2024
Target tracking by multiple particle filtering
M. F. Bugallo, T. Lu, P. M. Djuric
Published: 2007
Implicit particle filters for data assimilation
A. Chorin, M. Morzfeld, X. Tu
Published: 2010
Implicit sampling for particle filters
A. J. Chorin, X. Tu
Published: 2009
Implicit sampling for particle filters
A. J. Chorin, X. Tu
Published: 2009
Diffusion models beat gans on image synthesis
Prafulla Dhariwal, Alexander Quinn Nichol
Published: 2021
IFS Documentation CY48R1 - Part I: Observations
ECMWF
Published: 2023
Sequential data assimilation with a nonlinear quasi-geostrophic model using monte carlo methods to forecast error statistics
G. Evensen
Published: 1994
Data Assimilation: The Ensemble Kalman Filter
Published: 2006
The ensemble Kalman filter for combined state and parameter estimation: Monte Carlo techniques for data assimilation in large systems
G. Evensen
Published: 2009
Novel approach to nonlinear/non-gaussian bayesian state estimation
N. Gordon, D. Salmond, A. Smith
Published: 1993
Surface quasi-geostrophic dynamics
I. M. Held, R. T. Pierrehumbert, S. T. Garner, K. L. Swanson
Published: 1995
Denoising diffusion probabilistic models
J. Ho, A. Jain, P. Abbeel
Published: 2020
Higher resolution in an operational ensemble Kalman filter
P. L. Houtekamer, X. Deng, H. L. Michell, S.-J. Baek, N. Gagnon
Published: 2014
Data assimilation using an ensemble kalman filter technique
P. L. Houtekamer, H. L. Mitchell
Published: 1998
Efficient data assimilation for spatiotemporal chaos: A local ensemble transform kalman filter
B. R. Hunt, E. J. Kostelich, I. Szunyogh
Published: 2007
Improved distributed particle filters for tracking in a wireless sensor network
K. Kang, V. Maroulas, I. Schizas, F. Bao
Published: 2018
Big data assimilation: Real-time 30-second-refresh heavy rain forecast using fugaku during tokyo olympics and paralympics
T. Miyoshi, A. Amemiya, S. Otsuka, Y. Maejima, J. Taylor, T. Honda, H. Tomita, S. Nishizawa, K. Sueki, T. Yamaura, Y. Ishikawa, S. Satoh, T. Ushio, K. Koike, A. Uno
Published: 2023
Filtering via simulation: auxiliary particle filters
M. K. Pitt, N. Shephard
Published: 1999
Convective-scale data assimilation for the Weather Research and Forecasting model using the local particle filter
J. Poterjoy, R. A. Sobash, J. L. Anderson
Published: 2017
Stochastic filtering with applications in finance
B. Ramaprasad
Published: 2010
Particle filtering and Gaussian mixtures - on a localized mixture coefficients particle filter (LMCPF) for global NWP
A. Rojahn, N. Schenk, P. J. van Leeuwen, R. Potthast
Published: 2023
Kilometre-scale ensemble data assimilation for the COSMO model (KENDA)
C. Schraff, H. Reich, A. Rhodin, A. Schomburg, K. Stephan, A. Peria´nez, R. Potthast
Published: 2016
Obstacles to high-dimensional particle filtering
C. Snyder, T. Bengtsson, P. Bickel, J. Anderson
Published: 2008
On dimension reduction in gaussian filters
A. Solonen, T. Cui, J. Hakkarainen, Y. Marzouk
Published: 2016
Generative modeling by estimating gradients of the data distribution
Yang Song, Stefano Ermon
Published: 2019
Coupling techniques for nonlinear ensemble filtering
A. Spantini, R. Baptista, Y. Marzouk
Published: 2022
Assessment of a nonlinear ensemble transform filter for high-dimensional data assimilation
J. Todter, P. Kirchgessner, L. Nerger, B. Ahrens
Published: 2016
A connection between score matching and denoising autoencoders
P. Vincent
Published: 2011
On the optimal filtering of diffusion processes
M. Zakai
Published: 1969
Share