IDEAS home Printed from https://ideas.repec.org/a/eee/jmvana/v198y2023ics0047259x23000714.html
   My bibliography  Save this article

Covariance structure estimation with Laplace approximation

Author

Listed:
  • Sung, Bongjung
  • Lee, Jaeyong

Abstract

The Gaussian covariance graph model is popular for revealing the underlying dependency structures among random variables. In this paper, we consider a spike and slab prior, which is a mixture of point-mass and normal distribution, on the off-diagonal entries. The spike and slab prior naturally introduces sparsity to the covariance structure so that the resulting posterior renders covariance structure learning. Under the spike and slab prior, we calculate the posterior model probabilities of covariance structures and natural Bayesian quantities for model selection using the Laplace approximation. We show that the error due to the Laplace approximation becomes asymptotically marginal at a rate that depends on the posterior convergence rate of the covariance matrix under the Frobenius norm. We propose a covariance structure estimation method based on the approximated posterior model probabilities. We also propose a block coordinate descent algorithm to determine the mode of the posterior density conditional on the structure of the covariance. The posterior mode is an estimate of the covariance matrix once the structure is chosen and the Laplace approximation is computed around it. Through a simulation study based on five numerical models, we demonstrate that the proposed method outperforms its competitors. The proposed method is applied to the breast cancer and Parkinson’s disease datasets, as well as the prediction of telephone call counts using telephone call center data, and compared with its competitors in terms of the linear discriminant analysis classification accuracy.

Suggested Citation

  • Sung, Bongjung & Lee, Jaeyong, 2023. "Covariance structure estimation with Laplace approximation," Journal of Multivariate Analysis, Elsevier, vol. 198(C).
  • Handle: RePEc:eee:jmvana:v:198:y:2023:i:c:s0047259x23000714
    DOI: 10.1016/j.jmva.2023.105225
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0047259X23000714
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.jmva.2023.105225?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. repec:bfi:wpaper:2014-014 is not listed on IDEAS
    2. Conti, Gabriella & Frühwirth-Schnatter, Sylvia & Heckman, James J. & Piatek, Rémi, 2014. "Bayesian exploratory factor analysis," Journal of Econometrics, Elsevier, vol. 183(1), pages 31-57.
    3. Xu, Ping & Brock, Guy N. & Parrish, Rudolph S., 2009. "Modified linear discriminant analysis approaches for classification of high-dimensional microarray data," Computational Statistics & Data Analysis, Elsevier, vol. 53(5), pages 1674-1687, March.
    4. Kastner, Gregor, 2019. "Sparse Bayesian time-varying covariance estimation in many dimensions," Journal of Econometrics, Elsevier, vol. 210(1), pages 98-115.
    5. P. Tseng, 2001. "Convergence of a Block Coordinate Descent Method for Nondifferentiable Minimization," Journal of Optimization Theory and Applications, Springer, vol. 109(3), pages 475-494, June.
    6. C. M. Carvalho & J. G. Scott, 2009. "Objective Bayesian model selection in Gaussian graphical models," Biometrika, Biometrika Trust, vol. 96(3), pages 497-512.
    7. Cai, Tony & Liu, Weidong, 2011. "Adaptive Thresholding for Sparse Covariance Matrix Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 672-684.
    8. Wei Biao Wu, 2003. "Nonparametric estimation of large covariance matrices of longitudinal data," Biometrika, Biometrika Trust, vol. 90(4), pages 831-844, December.
    9. Frederick Wong, 2003. "Efficient estimation of covariance selection models," Biometrika, Biometrika Trust, vol. 90(4), pages 809-830, December.
    10. Jacob Bien & Robert J. Tibshirani, 2011. "Sparse estimation of a covariance matrix," Biometrika, Biometrika Trust, vol. 98(4), pages 807-820.
    11. Banerjee, Sayantan & Ghosal, Subhashis, 2015. "Bayesian structure learning in graphical models," Journal of Multivariate Analysis, Elsevier, vol. 136(C), pages 147-162.
    12. Olvi L. Mangasarian & W. Nick Street & William H. Wolberg, 1995. "Breast Cancer Diagnosis and Prognosis Via Linear Programming," Operations Research, INFORMS, vol. 43(4), pages 570-577, August.
    13. Lingrui Gan & Naveen N. Narisetty & Feng Liang, 2019. "Bayesian Regularization for Graphical Models With Unequal Shrinkage," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(527), pages 1218-1231, July.
    14. Kshitij Khare & Sang-Yun Oh & Bala Rajaratnam, 2015. "A convex pseudolikelihood framework for high dimensional partial correlation estimation with convergence guarantees," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 77(4), pages 803-825, September.
    15. A. Bhattacharya & D. B. Dunson, 2011. "Sparse Bayesian infinite factor models," Biometrika, Biometrika Trust, vol. 98(2), pages 291-306.
    16. Rothman, Adam J. & Levina, Elizaveta & Zhu, Ji, 2009. "Generalized Thresholding of Large Covariance Matrices," Journal of the American Statistical Association, American Statistical Association, vol. 104(485), pages 177-186.
    17. Ming Yuan & Yi Lin, 2007. "Model selection and estimation in the Gaussian graphical model," Biometrika, Biometrika Trust, vol. 94(1), pages 19-35.
    18. Yuan, Ming & Lin, Yi, 2005. "Efficient Empirical Bayes Variable Selection and Estimation in Linear Models," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 1215-1225, December.
    19. Jianhua Z. Huang & Naiping Liu & Mohsen Pourahmadi & Linxu Liu, 2006. "Covariance matrix selection and estimation via penalised normal likelihood," Biometrika, Biometrika Trust, vol. 93(1), pages 85-98, March.
    20. Carvalho, Carlos M. & Chang, Jeffrey & Lucas, Joseph E. & Nevins, Joseph R. & Wang, Quanli & West, Mike, 2008. "High-Dimensional Sparse Factor Modeling: Applications in Gene Expression Genomics," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1438-1456.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gautam Sabnis & Debdeep Pati & Anirban Bhattacharya, 2019. "Compressed Covariance Estimation with Automated Dimension Learning," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 81(2), pages 466-481, December.
    2. Seunghwan Lee & Sang Cheol Kim & Donghyeon Yu, 2023. "An efficient GPU-parallel coordinate descent algorithm for sparse precision matrix estimation via scaled lasso," Computational Statistics, Springer, vol. 38(1), pages 217-242, March.
    3. Lam, Clifford, 2020. "High-dimensional covariance matrix estimation," LSE Research Online Documents on Economics 101667, London School of Economics and Political Science, LSE Library.
    4. Bailey, Natalia & Pesaran, M. Hashem & Smith, L. Vanessa, 2019. "A multiple testing approach to the regularisation of large sample correlation matrices," Journal of Econometrics, Elsevier, vol. 208(2), pages 507-534.
    5. Benjamin Poignard & Manabu Asai, 2023. "Estimation of high-dimensional vector autoregression via sparse precision matrix," The Econometrics Journal, Royal Economic Society, vol. 26(2), pages 307-326.
    6. Lee, Kyoungjae & Jo, Seongil & Lee, Jaeyong, 2022. "The beta-mixture shrinkage prior for sparse covariances with near-minimax posterior convergence rate," Journal of Multivariate Analysis, Elsevier, vol. 192(C).
    7. Wang, Luheng & Chen, Zhao & Wang, Christina Dan & Li, Runze, 2020. "Ultrahigh dimensional precision matrix estimation via refitted cross validation," Journal of Econometrics, Elsevier, vol. 215(1), pages 118-130.
    8. Yang, Yihe & Zhou, Jie & Pan, Jianxin, 2021. "Estimation and optimal structure selection of high-dimensional Toeplitz covariance matrix," Journal of Multivariate Analysis, Elsevier, vol. 184(C).
    9. Yumou Qiu & Song Xi Chen, 2015. "Bandwidth Selection for High-Dimensional Covariance Matrix Estimation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(511), pages 1160-1174, September.
    10. Xingqi Du & Subhashis Ghosal, 2018. "Bayesian Discriminant Analysis Using a High Dimensional Predictor," Sankhya A: The Indian Journal of Statistics, Springer;Indian Statistical Institute, vol. 80(1), pages 112-145, December.
    11. Banerjee, Sayantan & Ghosal, Subhashis, 2015. "Bayesian structure learning in graphical models," Journal of Multivariate Analysis, Elsevier, vol. 136(C), pages 147-162.
    12. Li, Peili & Xiao, Yunhai, 2018. "An efficient algorithm for sparse inverse covariance matrix estimation based on dual formulation," Computational Statistics & Data Analysis, Elsevier, vol. 128(C), pages 292-307.
    13. Sylvia Fruhwirth-Schnatter, 2023. "Generalized Cumulative Shrinkage Process Priors with Applications to Sparse Bayesian Factor Analysis," Papers 2303.00473, arXiv.org.
    14. Xi Luo, 2011. "Recovering Model Structures from Large Low Rank and Sparse Covariance Matrix Estimation," Papers 1111.1133, arXiv.org, revised Mar 2013.
    15. Lee, Kwangmin & Lee, Jaeyong, 2023. "Post-processed posteriors for sparse covariances," Journal of Econometrics, Elsevier, vol. 236(1).
    16. Sylvia Fruhwirth-Schnatter & Darjus Hosszejni & Hedibert Freitas Lopes, 2023. "When it counts -- Econometric identification of the basic factor model based on GLT structures," Papers 2301.06354, arXiv.org.
    17. Choi, Young-Geun & Lim, Johan & Roy, Anindya & Park, Junyong, 2019. "Fixed support positive-definite modification of covariance matrix estimators via linear shrinkage," Journal of Multivariate Analysis, Elsevier, vol. 171(C), pages 234-249.
    18. Lam, Clifford & Fan, Jianqing, 2009. "Sparsistency and rates of convergence in large covariance matrix estimation," LSE Research Online Documents on Economics 31540, London School of Economics and Political Science, LSE Library.
    19. Jianqing Fan & Yuan Liao & Han Liu, 2016. "An overview of the estimation of large covariance and precision matrices," Econometrics Journal, Royal Economic Society, vol. 19(1), pages 1-32, February.
    20. Lam, Clifford, 2008. "Estimation of large precision matrices through block penalization," LSE Research Online Documents on Economics 31543, London School of Economics and Political Science, LSE Library.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jmvana:v:198:y:2023:i:c:s0047259x23000714. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/622892/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.