We propose an ensemble score filter (EnSF) for solving high-dimensional
nonlinear filtering problems with superior accuracy. A major drawback of
existing filtering methods, e.g., particle filters or ensemble Kalman filters,
is the low accuracy in handling high-dimensional and highly nonlinear problems.
EnSF attacks this challenge by exploiting the score-based diffusion model,
defined in a pseudo-temporal domain, to characterizing the evolution of the
filtering density. EnSF stores the information of the recursively updated
filtering density function in the score function, instead of storing the
information in a set of finite Monte Carlo samples (used in particle filters
and ensemble Kalman filters). Unlike existing diffusion models that train
neural networks to approximate the score function, we develop a training-free
score estimation that uses a mini-batch-based Monte Carlo estimator to directly
approximate the score function at any pseudo-spatial-temporal location, which
provides sufficient accuracy in solving high-dimensional nonlinear problems as
well as saves a tremendous amount of time spent on training neural networks.
High-dimensional Lorenz-96 systems are used to demonstrate the performance of
our method. EnSF provides surprising performance, compared with the
state-of-the-art Local Ensemble Transform Kalman Filter method, in reliably and
efficiently tracking extremely high-dimensional Lorenz systems (up to 1,000,000
dimensions) with highly nonlinear observation processes.
参考文献
Monthly Weather Review
A multicase comparative assessment of the ensemble Kalman filter for assimilation of radar observations. Part I: Storm-scale analyses
A. Aksoy, D. Dowell, C. Snyder
Published: 2009
Monthly Weather Review
A multicase comparative assessment of the ensemble Kalman filter for assimilation of radar observations. Part II: Short-range ensemble forecasts
Published: 2010
J. R. Statist. Soc. B
Particle markov chain monte carlo methods
C. Andrieu, A. Doucet, R. Holenstein
Published: 2010
Communications in Computational Physics
Backward sde filter for jump diffusion processes and its applications in material sciences
F. Bao, Y. Cao, P. Maksymovych
Published: 2020
SIAM/ASA J. Uncertain. Quantif.
A first order scheme for backward doubly stochastic differential equations
F. Bao, Y. Cao, A. Meir, W. Zhao
Published: 2016
SIAM/ASA J. Uncertain. Quantif.
A hybrid sparse-grid approach for nonlinear filtering problems based on adaptive-domain of the Zakai equation approximations
F. Bao, Y. Cao, C. Webster, G. Zhang
Published: 2014
Mathematical Medicine and Biology: A Journal of the IMA
Data assimilation of synthetic data as a novel strategy for predicting disease progression in alopecia areata
F. Bao, N. Cogan, A. Dobreva, R. Paus
Published: 2021
Journal of Computational Physics
A score-based nonlinear filter for data assimilation
F. Bao, Z. Zhang, G. Zhang
Published: 2024
2007 IEEE Aerospace Conference
Target tracking by multiple particle filtering
M. F. Bugallo, T. Lu, P. M. Djuric
Published: 2007
Communications in Applied Mathematics and Computational Science
Implicit particle filters for data assimilation
A. Chorin, M. Morzfeld, X. Tu
Published: 2010
Proceedings of the National Academy of Sciences
Implicit sampling for particle filters
A. J. Chorin, X. Tu
Published: 2009
Proc. Nat. Acad. Sc. USA
Implicit sampling for particle filters
A. J. Chorin, X. Tu
Published: 2009
Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems (NeurIPS)
Diffusion models beat gans on image synthesis
Prafulla Dhariwal, Alexander Quinn Nichol
Published: 2021
ECMWF
IFS Documentation CY48R1 - Part I: Observations
ECMWF
Published: 2023
Journal of Geophysical Research: Oceans
Sequential data assimilation with a nonlinear quasi-geostrophic model using monte carlo methods to forecast error statistics
G. Evensen
Published: 1994
Springer-Verlag
Data Assimilation: The Ensemble Kalman Filter
Published: 2006
IEEE Control Syst. Mag.
The ensemble Kalman filter for combined state and parameter estimation: Monte Carlo techniques for data assimilation in large systems
G. Evensen
Published: 2009
IEE PROCEEDING-F
Novel approach to nonlinear/non-gaussian bayesian state estimation
N. Gordon, D. Salmond, A. Smith
Published: 1993
Journal of Fluid Mechanics
Surface quasi-geostrophic dynamics
I. M. Held, R. T. Pierrehumbert, S. T. Garner, K. L. Swanson
Published: 1995
Advances in Neural Information Processing Systems
Denoising diffusion probabilistic models
J. Ho, A. Jain, P. Abbeel
Published: 2020
Monthly Weather Review
Higher resolution in an operational ensemble Kalman filter
P. L. Houtekamer, X. Deng, H. L. Michell, S.-J. Baek, N. Gagnon
Published: 2014
Monthly Weather Review
Data assimilation using an ensemble kalman filter technique
P. L. Houtekamer, H. L. Mitchell
Published: 1998
Physica D: Nonlinear Phenomena
Efficient data assimilation for spatiotemporal chaos: A local ensemble transform kalman filter
B. R. Hunt, E. J. Kostelich, I. Szunyogh
Published: 2007
Comput. Statist. Data Anal.
Improved distributed particle filters for tracking in a wireless sensor network
K. Kang, V. Maroulas, I. Schizas, F. Bao
Published: 2018
Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC ’23
Big data assimilation: Real-time 30-second-refresh heavy rain forecast using fugaku during tokyo olympics and paralympics
T. Miyoshi, A. Amemiya, S. Otsuka, Y. Maejima, J. Taylor, T. Honda, H. Tomita, S. Nishizawa, K. Sueki, T. Yamaura, Y. Ishikawa, S. Satoh, T. Ushio, K. Koike, A. Uno
Published: 2023
J. Amer. Statist. Assoc.
Filtering via simulation: auxiliary particle filters
M. K. Pitt, N. Shephard
Published: 1999
Monthly Weather Review
Convective-scale data assimilation for the Weather Research and Forecasting model using the local particle filter
J. Poterjoy, R. A. Sobash, J. L. Anderson
Published: 2017
Stochastic filtering with applications in finance
B. Ramaprasad
Published: 2010
Journal of the Meteorological Society of Japan
Particle filtering and Gaussian mixtures - on a localized mixture coefficients particle filter (LMCPF) for global NWP
A. Rojahn, N. Schenk, P. J. van Leeuwen, R. Potthast
Published: 2023
Quarterly Journal of the Royal Meteorological Society
Kilometre-scale ensemble data assimilation for the COSMO model (KENDA)
C. Schraff, H. Reich, A. Rhodin, A. Schomburg, K. Stephan, A. Peria´nez, R. Potthast
Published: 2016
Mon. Wea. Rev.
Obstacles to high-dimensional particle filtering
C. Snyder, T. Bengtsson, P. Bickel, J. Anderson
Published: 2008
Inverse Problems
On dimension reduction in gaussian filters
A. Solonen, T. Cui, J. Hakkarainen, Y. Marzouk
Published: 2016
Curran Associates Inc.
Generative modeling by estimating gradients of the data distribution