IDEAS home Printed from https://ideas.repec.org/a/wly/envmet/v34y2023i1ne2757.html
   My bibliography  Save this article

Scalable spatio‐temporal smoothing via hierarchical sparse Cholesky decomposition

Author

Listed:
  • Marcin Jurek
  • Matthias Katzfuss

Abstract

We propose an approximation to the forward filter backward sampler (FFBS) algorithm for large‐scale spatio‐temporal smoothing. FFBS is commonly used in Bayesian statistics when working with linear Gaussian state‐space models, but it requires inverting covariance matrices which have the size of the latent state vector. The computational burden associated with this operation effectively prohibits its applications in high‐dimensional settings. We propose a scalable spatio‐temporal FFBS approach based on the hierarchical Vecchia approximation of Gaussian processes, which has been previously successfully used in spatial statistics. On simulated and real data, our approach outperformed a low‐rank FFBS approximation.

Suggested Citation

  • Marcin Jurek & Matthias Katzfuss, 2023. "Scalable spatio‐temporal smoothing via hierarchical sparse Cholesky decomposition," Environmetrics, John Wiley & Sons, Ltd., vol. 34(1), February.
    • Handle: RePEc:wly:envmet:v:34:y:2023:i:1:n:e2757
    DOI: 10.1002/env.2757
    as

    Download full text from publisher

    File URL: https://doi.org/10.1002/env.2757
    Download Restriction: no
    File URL: https://libkey.io/10.1002/env.2757?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jarociński, Marek, 2015. "A note on implementing the Durbin and Koopman simulation smoother," Computational Statistics & Data Analysis, Elsevier, vol. 91(C), pages 1-3.
    2. Fabio Sigrist & Hans R. Künsch & Werner A. Stahel, 2015. "Stochastic partial differential equation based modelling of large space–time data sets," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 77(1), pages 3-33, January.
    3. Sudipto Banerjee & Alan E. Gelfand & Andrew O. Finley & Huiyan Sang, 2008. "Gaussian predictive process models for large spatial data sets," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 70(4), pages 825-848, September.
    4. Tilmann Gneiting & Larissa Stanberry & Eric Grimit & Leonhard Held & Nicholas Johnson, 2008. "Rejoinder on: Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(2), pages 256-264, August.
    5. Stroud, Jonathan R. & Stein, Michael L. & Lesht, Barry M. & Schwab, David J. & Beletsky, Dmitry, 2010. "An Ensemble Kalman Filter and Smoother for Satellite Data Assimilation," Journal of the American Statistical Association, American Statistical Association, vol. 105(491), pages 978-990.
    6. Tilmann Gneiting & Larissa Stanberry & Eric Grimit & Leonhard Held & Nicholas Johnson, 2008. "Assessing probabilistic forecasts of multivariate quantities, with an application to ensemble predictions of surface winds," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(2), pages 211-235, August.
    7. J. Durbin, 2002. "A simple and efficient simulation smoother for state space time series analysis," Biometrika, Biometrika Trust, vol. 89(3), pages 603-616, August.
    8. Finley, Andrew O. & Sang, Huiyan & Banerjee, Sudipto & Gelfand, Alan E., 2009. "Improving the performance of predictive process modeling for large datasets," Computational Statistics & Data Analysis, Elsevier, vol. 53(8), pages 2873-2884, June.
    9. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Håvard Rue & Sara Martino & Nicolas Chopin, 2009. "Approximate Bayesian inference for latent Gaussian models by using integrated nested Laplace approximations," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 71(2), pages 319-392, April.
    2. Florian Ziel & Kevin Berk, 2019. "Multivariate Forecasting Evaluation: On Sensitive and Strictly Proper Scoring Rules," Papers 1910.07325, arXiv.org.
    3. Matthew J. Heaton & Abhirup Datta & Andrew O. Finley & Reinhard Furrer & Joseph Guinness & Rajarshi Guhaniyogi & Florian Gerber & Robert B. Gramacy & Dorit Hammerling & Matthias Katzfuss & Finn Lindgr, 2019. "A Case Study Competition Among Methods for Analyzing Large Spatial Data," Journal of Agricultural, Biological and Environmental Statistics, Springer;The International Biometric Society;American Statistical Association, vol. 24(3), pages 398-425, September.
    4. Fabian Krüger & Sebastian Lerch & Thordis Thorarinsdottir & Tilmann Gneiting, 2021. "Predictive Inference Based on Markov Chain Monte Carlo Output," International Statistical Review, International Statistical Institute, vol. 89(2), pages 274-301, August.
    5. Braun, Julia & Sabanés Bové, Daniel & Held, Leonhard, 2014. "Choice of generalized linear mixed models using predictive crossvalidation," Computational Statistics & Data Analysis, Elsevier, vol. 75(C), pages 190-202.
    6. Grothe, Oliver & Kächele, Fabian & Krüger, Fabian, 2023. "From point forecasts to multivariate probabilistic forecasts: The Schaake shuffle for day-ahead electricity price forecasting," Energy Economics, Elsevier, vol. 120(C).
    7. Thordis L. Thorarinsdottir & Tilmann Gneiting, 2010. "Probabilistic forecasts of wind speed: ensemble model output statistics by using heteroscedastic censored regression," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(2), pages 371-388, April.
    8. Gensler, André & Sick, Bernhard & Vogt, Stephan, 2018. "A review of uncertainty representations and metaverification of uncertainty assessment techniques for renewable energies," Renewable and Sustainable Energy Reviews, Elsevier, vol. 96(C), pages 352-379.
    9. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    10. Stover, Oliver & Nath, Paromita & Karve, Pranav & Mahadevan, Sankaran & Baroud, Hiba, 2024. "Dependence structure learning and joint probabilistic forecasting of stochastic power grid variables," Applied Energy, Elsevier, vol. 357(C).
    11. van der Meer, Dennis & Wang, Guang Chao & Munkhammar, Joakim, 2021. "An alternative optimal strategy for stochastic model predictive control of a residential battery energy management system with solar photovoltaic," Applied Energy, Elsevier, vol. 283(C).
    12. Mikkel L. Sørensen & Peter Nystrup & Mathias B. Bjerregård & Jan K. Møller & Peder Bacher & Henrik Madsen, 2023. "Recent developments in multivariate wind and solar power forecasting," Wiley Interdisciplinary Reviews: Energy and Environment, Wiley Blackwell, vol. 12(2), March.
    13. Roland Weigand, 2014. "Matrix Box-Cox Models for Multivariate Realized Volatility," Working Papers 144, Bavarian Graduate Program in Economics (BGPE).
    14. Bracher, Johannes & Held, Leonhard, 2022. "Endemic-epidemic models with discrete-time serial interval distributions for infectious disease prediction," International Journal of Forecasting, Elsevier, vol. 38(3), pages 1221-1233.
    15. Shinichiro Shirota & Andrew O. Finley & Bruce D. Cook & Sudipto Banerjee, 2023. "Conjugate sparse plus low rank models for efficient Bayesian interpolation of large spatial data," Environmetrics, John Wiley & Sons, Ltd., vol. 34(1), February.
    16. Zhang, Yao & Wang, Jianxue & Wang, Xifan, 2014. "Review on probabilistic forecasting of wind power generation," Renewable and Sustainable Energy Reviews, Elsevier, vol. 32(C), pages 255-270.
    17. Waley W. J. Liang & Herbert K. H. Lee, 2019. "Bayesian nonstationary Gaussian process models via treed process convolutions," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 13(3), pages 797-818, September.
    18. Puwasala Gamakumara & Anastasios Panagiotelis & George Athanasopoulos & Rob J Hyndman, 2018. "Probabilisitic forecasts in hierarchical time series," Monash Econometrics and Business Statistics Working Papers 11/18, Monash University, Department of Econometrics and Business Statistics.
    19. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    20. Warne, Anders, 2023. "DSGE model forecasting: rational expectations vs. adaptive learning," Working Paper Series 2768, European Central Bank.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:envmet:v:34:y:2023:i:1:n:e2757. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.interscience.wiley.com/jpages/1180-4009/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.