IDEAS home Printed from https://ideas.repec.org/p/zur/econwp/323.html
   My bibliography  Save this paper

The power of (non-)linear shrinking: a review and guide to covariance matrix estimation

Author

Listed:
  • Olivier Ledoit
  • Michael Wolf

Abstract

Many econometric and data-science applications require a reliable estimate of the covariance matrix, such as Markowitz portfolio selection. When the number of variables is of the same magnitude as the number of observations, this constitutes a difficult estimation problem; the sample covariance matrix certainly will not do. In this paper, we review our work in this area, going back 15+ years. We have promoted various shrinkage estimators, which can be classified into linear and nonlinear. Linear shrinkage is simpler to understand, to derive, and to implement. But nonlinear shrinkage can deliver another level of performance improvement, especially if overlaid with stylized facts such as time-varying co-volatility or factor models.

Suggested Citation

  • Olivier Ledoit & Michael Wolf, 2019. "The power of (non-)linear shrinking: a review and guide to covariance matrix estimation," ECON - Working Papers 323, Department of Economics - University of Zurich, revised Feb 2020.
    • Handle: RePEc:zur:econwp:323
    as

    Download full text from publisher

    File URL: https://www.zora.uzh.ch/id/eprint/170642/7/econwp323.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Ledoit, Olivier & Wolf, Michael, 2004. "A well-conditioned estimator for large-dimensional covariance matrices," Journal of Multivariate Analysis, Elsevier, vol. 88(2), pages 365-411, February.
    2. Jianqing Fan & Yuan Liao & Martina Mincheva, 2013. "Large covariance estimation by thresholding principal orthogonal complements," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 75(4), pages 603-680, September.
    3. Michael Wolf & Dan Wunderli, 2009. "Fund-of-funds construction by statistical multiple testing methods," IEW - Working Papers 445, Institute for Empirical Research in Economics - University of Zurich.
    4. Victor DeMiguel & Lorenzo Garlappi & Francisco J. Nogales & Raman Uppal, 2009. "A Generalized Approach to Portfolio Optimization: Improving Performance by Constraining Portfolio Norms," Management Science, INFORMS, vol. 55(5), pages 798-812, May.
    5. Olivier Ledoit & Michael Wolf, 2017. "Analytical nonlinear shrinkage of large-dimensional covariance matrices," ECON - Working Papers 264, Department of Economics - University of Zurich, revised Nov 2018.
    6. Ren, Yu & Shimotsu, Katsumi, 2009. "Improvement in finite sample properties of the Hansen-Jagannathan distance test," Journal of Empirical Finance, Elsevier, vol. 16(3), pages 483-506, June.
    7. Stephen A. Ross, 2013. "The Arbitrage Theory of Capital Asset Pricing," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 1, pages 11-30, World Scientific Publishing Co. Pte. Ltd..
    8. Cavit Pakel & Neil Shephard & Kevin Sheppard & Robert F. Engle, 2021. "Fitting Vast Dimensional Time-Varying Covariance Models," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 39(3), pages 652-668, July.
    9. Fama, Eugene F. & French, Kenneth R., 2015. "A five-factor asset pricing model," Journal of Financial Economics, Elsevier, vol. 116(1), pages 1-22.
    10. Hansen, Lars Peter & Jagannathan, Ravi, 1997. "Assessing Specification Errors in Stochastic Discount Factor Models," Journal of Finance, American Finance Association, vol. 52(2), pages 557-590, June.
    11. Schäfer Juliane & Strimmer Korbinian, 2005. "A Shrinkage Approach to Large-Scale Covariance Matrix Estimation and Implications for Functional Genomics," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 4(1), pages 1-32, November.
    12. George M. Korniotis, 2008. "Habit Formation, Incomplete Markets, and the Significance of Regional Risk for Expected Returns," The Review of Financial Studies, Society for Financial Studies, vol. 21(5), pages 2139-2172, September.
    13. Frost, Peter A. & Savarino, James E., 1986. "An Empirical Bayes Approach to Efficient Portfolio Selection," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 21(3), pages 293-305, September.
    14. Connor, Gregory & Korajczyk, Robert A., 1988. "Risk and return in an equilibrium APT : Application of a new test methodology," Journal of Financial Economics, Elsevier, vol. 21(2), pages 255-289, September.
    15. Robert F. Engle & Olivier Ledoit & Michael Wolf, 2019. "Large Dynamic Covariance Matrices," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(2), pages 363-375, April.
    16. Ledoit, Olivier & Wolf, Michael, 2003. "Improved estimation of the covariance matrix of stock returns with an application to portfolio selection," Journal of Empirical Finance, Elsevier, vol. 10(5), pages 603-621, December.
    17. Connor, Gregory & Korajczyk, Robert A, 1993. "A Test for the Number of Factors in an Approximate Factor Model," Journal of Finance, American Finance Association, vol. 48(4), pages 1263-1291, September.
    18. Luc Bauwens & Sébastien Laurent & Jeroen V. K. Rombouts, 2006. "Multivariate GARCH models: a survey," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 21(1), pages 79-109, January.
    19. Ledoit, Olivier & Wolf, Michael, 2015. "Spectrum estimation: A unified framework for covariance matrix estimation and PCA in large dimensions," Journal of Multivariate Analysis, Elsevier, vol. 139(C), pages 360-384.
    20. Engle, Robert & Colacito, Riccardo, 2006. "Testing and Valuing Dynamic Correlations for Asset Allocation," Journal of Business & Economic Statistics, American Statistical Association, vol. 24, pages 238-253, April.
    21. Silverstein, J. W., 1995. "Strong Convergence of the Empirical Distribution of Eigenvalues of Large Dimensional Random Matrices," Journal of Multivariate Analysis, Elsevier, vol. 55(2), pages 331-339, November.
    22. Simon A. Broda & Marc S. Paolella, 2009. "CHICAGO: A Fast and Accurate Method for Portfolio Risk Calculation," Journal of Financial Econometrics, Oxford University Press, vol. 7(4), pages 412-436, Fall.
    23. William F. Sharpe, 1964. "Capital Asset Prices: A Theory Of Market Equilibrium Under Conditions Of Risk," Journal of Finance, American Finance Association, vol. 19(3), pages 425-442, September.
    24. Abadir, Karim M. & Distaso, Walter & Žikeš, Filip, 2014. "Design-free estimation of variance matrices," Journal of Econometrics, Elsevier, vol. 181(2), pages 165-180.
    25. Silverstein, J. W. & Bai, Z. D., 1995. "On the Empirical Distribution of Eigenvalues of a Class of Large Dimensional Random Matrices," Journal of Multivariate Analysis, Elsevier, vol. 54(2), pages 175-192, August.
    26. Kwon, Yongchan & Choi, Young-Geun & Park, Taesung & Ziegler, Andreas & Paik, Myunghee Cho, 2017. "Generalized estimating equations with stabilized working correlation structure," Computational Statistics & Data Analysis, Elsevier, vol. 106(C), pages 1-11.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Taras Bodnar & Nestor Parolya & Erik Thors'en, 2022. "Two is better than one: Regularized shrinkage of large minimum variance portfolio," Papers 2202.06666, arXiv.org.
    2. Pier Francesco Procacci & Tomaso Aste, 2021. "Portfolio Optimization with Sparse Multivariate Modelling," Papers 2103.15232, arXiv.org.
    3. Ortiz, Roberto & Contreras, Mauricio & Mellado, Cristhian, 2023. "Regression, multicollinearity and Markowitz," Finance Research Letters, Elsevier, vol. 58(PC).
    4. Simon Hediger & Jeffrey Näf & Marc S. Paolella & Paweł Polak, 2023. "Heterogeneous tail generalized common factor modeling," Digital Finance, Springer, vol. 5(2), pages 389-420, June.
    5. Pier Francesco Procacci & Tomaso Aste, 2022. "Portfolio optimization with sparse multivariate modeling," Journal of Asset Management, Palgrave Macmillan, vol. 23(6), pages 445-465, October.
    6. Sylvia Frühwirth-Schnatter & Darjus Hosszejni & Hedibert Freitas Lopes, 2023. "When It Counts—Econometric Identification of the Basic Factor Model Based on GLT Structures," Econometrics, MDPI, vol. 11(4), pages 1-30, November.
    7. Alexandru V. Asimit & Ioannis Kyriakou & Simone Santoni & Salvatore Scognamiglio & Rui Zhu, 2022. "Robust Classification via Support Vector Machines," Risks, MDPI, vol. 10(8), pages 1-25, August.
    8. Ruslan Tepelyan & Achintya Gopal, 2023. "Generative Machine Learning for Multivariate Equity Returns," Papers 2311.14735, arXiv.org.
    9. Khashanah, Khaldoun & Simaan, Majeed & Simaan, Yusif, 2022. "Do we need higher-order comoments to enhance mean-variance portfolios? Evidence from a simplified jump process," International Review of Financial Analysis, Elsevier, vol. 81(C).
    10. Damir Filipovic & Paul Schneider, 2024. "Fundamental properties of linear factor models," Papers 2409.02521, arXiv.org, revised Oct 2024.
    11. Andrew Grant & Oh Kang Kwon & Steve Satchell, 2024. "Properties of risk aversion estimated from portfolio weights," Journal of Asset Management, Palgrave Macmillan, vol. 25(5), pages 427-444, September.
    12. Benoit Oriol & Alexandre Miot, 2023. "Ledoit-Wolf linear shrinkage with unknown mean," Papers 2304.07045, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gianluca De Nard & Olivier Ledoit & Michael Wolf, 2018. "Factor models for portfolio selection in large dimensions: the good, the better and the ugly," ECON - Working Papers 290, Department of Economics - University of Zurich, revised Dec 2018.
    2. De Nard, Gianluca & Zhao, Zhao, 2023. "Using, taming or avoiding the factor zoo? A double-shrinkage estimator for covariance matrices," Journal of Empirical Finance, Elsevier, vol. 72(C), pages 23-35.
    3. Lam, Clifford, 2020. "High-dimensional covariance matrix estimation," LSE Research Online Documents on Economics 101667, London School of Economics and Political Science, LSE Library.
    4. Thomas Conlon & John Cotter & Iason Kynigakis, 2021. "Machine Learning and Factor-Based Portfolio Optimization," Papers 2107.13866, arXiv.org.
    5. Olivier Ledoit & Michael Wolf, 2019. "Quadratic shrinkage for large covariance matrices," ECON - Working Papers 335, Department of Economics - University of Zurich, revised Dec 2020.
    6. Ledoit, Olivier & Wolf, Michael, 2017. "Numerical implementation of the QuEST function," Computational Statistics & Data Analysis, Elsevier, vol. 115(C), pages 199-223.
    7. Hediger, Simon & Näf, Jeffrey, 2024. "Combining the MGHyp distribution with nonlinear shrinkage in modeling financial asset returns," Journal of Empirical Finance, Elsevier, vol. 77(C).
    8. Firoozye, Nikan & Tan, Vincent & Zohren, Stefan, 2023. "Canonical portfolios: Optimal asset and signal combination," Journal of Banking & Finance, Elsevier, vol. 154(C).
    9. Olivier Ledoit & Michael Wolf, 2017. "Analytical nonlinear shrinkage of large-dimensional covariance matrices," ECON - Working Papers 264, Department of Economics - University of Zurich, revised Nov 2018.
    10. Aït-Sahalia, Yacine & Xiu, Dacheng, 2017. "Using principal component analysis to estimate a high dimensional factor model with high-frequency data," Journal of Econometrics, Elsevier, vol. 201(2), pages 384-399.
    11. Sven Husmann & Antoniya Shivarova & Rick Steinert, 2019. "Cross-validated covariance estimators for high-dimensional minimum-variance portfolios," Papers 1910.13960, arXiv.org, revised Oct 2020.
    12. Robert F. Engle & Olivier Ledoit & Michael Wolf, 2019. "Large Dynamic Covariance Matrices," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(2), pages 363-375, April.
    13. José Rangel & Robert Engle, 2012. "The Factor–Spline–GARCH Model for High and Low Frequency Correlations," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 109-124.
    14. Sven Husmann & Antoniya Shivarova & Rick Steinert, 2022. "Sparsity and stability for minimum-variance portfolios," Risk Management, Palgrave Macmillan, vol. 24(3), pages 214-235, September.
    15. Santos, André Alves Portela & Ferreira, Alexandre R., 2017. "On the choice of covariance specifications for portfolio selection problems," Brazilian Review of Econometrics, Sociedade Brasileira de Econometria - SBE, vol. 37(1), May.
    16. Zhao Zhao & Olivier Ledoit & Hui Jiang, 2019. "Risk reduction and efficiency increase in large portfolios: leverage and shrinkage," ECON - Working Papers 328, Department of Economics - University of Zurich, revised Jan 2020.
    17. Kwangmin Jung & Donggyu Kim & Seunghyeon Yu, 2022. "Next generation models for portfolio risk management: An approach using financial big data," Journal of Risk & Insurance, The American Risk and Insurance Association, vol. 89(3), pages 765-787, September.
    18. Kim, Soohun & Skoulakis, Georgios, 2018. "Ex-post risk premia estimation and asset pricing tests using large cross sections: The regression-calibration approach," Journal of Econometrics, Elsevier, vol. 204(2), pages 159-188.
    19. Mörstedt, Torsten & Lutz, Bernhard & Neumann, Dirk, 2024. "Cross validation based transfer learning for cross-sectional non-linear shrinkage: A data-driven approach in portfolio optimization," European Journal of Operational Research, Elsevier, vol. 318(2), pages 670-685.
    20. Ikeda, Yuki & Kubokawa, Tatsuya, 2016. "Linear shrinkage estimation of large covariance matrices using factor models," Journal of Multivariate Analysis, Elsevier, vol. 152(C), pages 61-81.

    More about this item

    Keywords

    Dynamic conditional correlations; factor models; large-dimensional asymptotics; Markowitz portfolio selection; rotation equivariance;
    All these keywords.

    JEL classification:

    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General
    • C58 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Financial Econometrics
    • G11 - Financial Economics - - General Financial Markets - - - Portfolio Choice; Investment Decisions

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zur:econwp:323. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Severin Oswald (email available below). General contact details of provider: https://edirc.repec.org/data/seizhch.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.