IDEAS home Printed from https://ideas.repec.org/a/eee/jmvana/v196y2023ics0047259x23000349.html
   My bibliography  Save this article

Randomized extrapolation for accelerating EM-type fixed-point algorithms

Author

Listed:
  • Saâdaoui, Foued

Abstract

Several extrapolation strategies have been proposed in the literature to accelerate the EM algorithm, with varying degrees of success. One advantage of extrapolation methods is their ease of implementation, as they only require working with the EM iterations and do not need auxiliary quantities, such as gradient and Hessian. In this paper, we introduce a new family of iterative schemes based on vector extrapolation methods. We also construct and numerically test a randomly relaxed version of the scheme. Our results demonstrate that these new strategies can significantly and stably accelerate the convergence of the EM algorithm compared to existing methods. Moreover, these strategies are highly versatile as they can accelerate any linearly convergent fixed point iteration, including EM-type algorithms. Finally, we provide statistical modeling experiments at the end of the paper to demonstrate the applicability and interest of these convergence acceleration schemes, whether applied to the EM algorithm or one of its variants.

Suggested Citation

  • Saâdaoui, Foued, 2023. "Randomized extrapolation for accelerating EM-type fixed-point algorithms," Journal of Multivariate Analysis, Elsevier, vol. 196(C).
    • Handle: RePEc:eee:jmvana:v:196:y:2023:i:c:s0047259x23000349
    DOI: 10.1016/j.jmva.2023.105188
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0047259X23000349
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jmva.2023.105188?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Allassonnière, Stéphanie & Chevallier, Juliette, 2021. "A new class of stochastic EM algorithms. Escaping local maxima and handling intractable sampling," Computational Statistics & Data Analysis, Elsevier, vol. 159(C).
    2. Foued SaÂdaoui, 2012. "A probabilistic clustering method for US interest rate analysis," Quantitative Finance, Taylor & Francis Journals, vol. 12(1), pages 135-148, November.
    3. Mingfeng Wang & Masahiro Kuroda & Michio Sakakihara & Zhi Geng, 2008. "Acceleration of the EM algorithm using the vector epsilon algorithm," Computational Statistics, Springer, vol. 23(3), pages 469-486, July.
    4. Mortaza Jamshidian & Robert I. Jennrich, 1997. "Acceleration of the EM Algorithm by using Quasi‐Newton Methods," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 59(3), pages 569-587.
    5. Saâdaoui, Foued, 2010. "Acceleration of the EM algorithm via extrapolation methods: Review, comparison and new methods," Computational Statistics & Data Analysis, Elsevier, vol. 54(3), pages 750-766, March.
    6. Michael Cohen & Siddhartha R. Dalal & John W. Tukey, 1993. "Robust, Smoothly Heterogeneous Variance Regression," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 42(2), pages 339-353, June.
    7. Roland, Christophe, 2010. "A note on the parameterized EM method," Statistics & Probability Letters, Elsevier, vol. 80(17-18), pages 1354-1357, September.
    8. Kuroda, Masahiro & Sakakihara, Michio, 2006. "Accelerating the convergence of the EM algorithm using the vector [epsilon] algorithm," Computational Statistics & Data Analysis, Elsevier, vol. 51(3), pages 1549-1561, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Masahiro Kuroda & Zhi Geng & Michio Sakakihara, 2015. "Improving the vector $$\varepsilon $$ ε acceleration for the EM algorithm using a re-starting procedure," Computational Statistics, Springer, vol. 30(4), pages 1051-1077, December.
    2. Foued Saâdaoui & Hana Rabbouch, 2024. "Structured multifractal scaling of the principal cryptocurrencies: Examination using a self‐explainable machine learning," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 43(7), pages 2917-2934, November.
    3. Tian, Guo-Liang & Tang, Man-Lai & Liu, Chunling, 2012. "Accelerating the quadratic lower-bound algorithm via optimizing the shrinkage parameter," Computational Statistics & Data Analysis, Elsevier, vol. 56(2), pages 255-265.
    4. Kuroda, Masahiro & Mori, Yuichi & Iizuka, Masaya & Sakakihara, Michio, 2011. "Acceleration of the alternating least squares algorithm for principal components analysis," Computational Statistics & Data Analysis, Elsevier, vol. 55(1), pages 143-153, January.
    5. Saâdaoui, Foued, 2023. "Skewed multifractal scaling of stock markets during the COVID-19 pandemic," Chaos, Solitons & Fractals, Elsevier, vol. 170(C).
    6. Berlinet, A.F. & Roland, Ch., 2012. "Acceleration of the EM algorithm: P-EM versus epsilon algorithm," Computational Statistics & Data Analysis, Elsevier, vol. 56(12), pages 4122-4137.
    7. Jurgen A. Doornik, 2018. "Accelerated Estimation of Switching Algorithms: The Cointegrated VAR Model and Other Applications," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 45(2), pages 283-300, June.
    8. Zhou, Lin & Tang, Yayong, 2021. "Linearly preconditioned nonlinear conjugate gradient acceleration of the PX-EM algorithm," Computational Statistics & Data Analysis, Elsevier, vol. 155(C).
    9. Kaarina Matilainen & Esa A Mäntysaari & Martin H Lidauer & Ismo Strandén & Robin Thompson, 2013. "Employing a Monte Carlo Algorithm in Newton-Type Methods for Restricted Maximum Likelihood Estimation of Genetic Parameters," PLOS ONE, Public Library of Science, vol. 8(12), pages 1-7, December.
    10. Gámiz, María Luz & Mammen, Enno & Martínez-Miranda, María Dolores & Nielsen, Jens Perch, 2022. "Missing link survival analysis with applications to available pandemic data," Computational Statistics & Data Analysis, Elsevier, vol. 169(C).
    11. Kenneth Lange & Eric C. Chi & Hua Zhou, 2014. "A Brief Survey of Modern Optimization for Statisticians," International Statistical Review, International Statistical Institute, vol. 82(1), pages 46-70, April.
    12. Iain L. MacDonald, 2021. "Is EM really necessary here? Examples where it seems simpler not to use EM," AStA Advances in Statistical Analysis, Springer;German Statistical Society, vol. 105(4), pages 629-647, December.
    13. Kohei Adachi, 2013. "Factor Analysis with EM Algorithm Never Gives Improper Solutions when Sample Covariance and Initial Parameter Matrices Are Proper," Psychometrika, Springer;The Psychometric Society, vol. 78(2), pages 380-394, April.
    14. Jochen Ranger & Christoph König & Benjamin W. Domingue & Jörg-Tobias Kuhn & Andreas Frey, 2024. "A Multidimensional Partially Compensatory Response Time Model on Basis of the Log-Normal Distribution," Journal of Educational and Behavioral Statistics, , vol. 49(3), pages 431-464, June.
    15. Saâdaoui, Foued, 2018. "Testing for multifractality of Islamic stock markets," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 496(C), pages 263-273.
    16. Saâdaoui, Foued & Ben Jabeur, Sami & Goodell, John W., 2022. "Causality of geopolitical risk on food prices: Considering the Russo–Ukrainian conflict," Finance Research Letters, Elsevier, vol. 49(C).
    17. Jonathan James, 2012. "A tractable estimator for general mixed multinomial logit models," Working Papers (Old Series) 1219, Federal Reserve Bank of Cleveland.
    18. Fung, Tsz Chai, 2022. "Maximum weighted likelihood estimator for robust heavy-tail modelling of finite mixture models," Insurance: Mathematics and Economics, Elsevier, vol. 107(C), pages 180-198.
    19. Saâdaoui, Foued, 2010. "Acceleration of the EM algorithm via extrapolation methods: Review, comparison and new methods," Computational Statistics & Data Analysis, Elsevier, vol. 54(3), pages 750-766, March.
    20. Takeshi Fukasawa, 2024. "Fast and simple inner-loop algorithms of static / dynamic BLP estimations," Papers 2404.04494, arXiv.org, revised Oct 2024.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jmvana:v:196:y:2023:i:c:s0047259x23000349. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/wps/find/journaldescription.cws_home/622892/description#description .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.