IDEAS home Printed from https://ideas.repec.org/a/eee/csdana/v127y2018icp160-171.html
   My bibliography  Save this article

Addressing overfitting and underfitting in Gaussian model-based clustering

Author

Listed:
  • Andrews, Jeffrey L.

Abstract

The expectation–maximization (EM) algorithm is a common approach for parameter estimation in the context of cluster analysis using finite mixture models. This approach suffers from the well-known issue of convergence to local maxima, but also the less obvious problem of overfitting. These combined, and competing, concerns are illustrated through simulation and then addressed by introducing an algorithm that augments the traditional EM with the nonparametric bootstrap. Further simulations and applications to real data lend support for the usage of this bootstrap augmented EM-style algorithm to avoid both overfitting and local maxima.

Suggested Citation

  • Andrews, Jeffrey L., 2018. "Addressing overfitting and underfitting in Gaussian model-based clustering," Computational Statistics & Data Analysis, Elsevier, vol. 127(C), pages 160-171.
    • Handle: RePEc:eee:csdana:v:127:y:2018:i:c:p:160-171
    DOI: 10.1016/j.csda.2018.05.015
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167947318301245
    Download Restriction: Full text for ScienceDirect subscribers only.
    File URL: https://libkey.io/10.1016/j.csda.2018.05.015?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Wei‐Chien Chang, 1983. "On Using Principal Components before Separating a Mixture of Two Multivariate Normal Distributions," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 32(3), pages 267-275, November.
    2. Bouveyron, Charles & Brunet-Saumard, Camille, 2014. "Model-based clustering of high-dimensional data: A review," Computational Statistics & Data Analysis, Elsevier, vol. 71(C), pages 52-78.
    3. Biernacki, Christophe & Chrétien, Stéphane, 2003. "Degeneracy in the maximum likelihood estimation of univariate Gaussian mixtures with EM," Statistics & Probability Letters, Elsevier, vol. 61(4), pages 373-382, February.
    4. Paul D. McNicholas, 2016. "Model-Based Clustering," Journal of Classification, Springer;The Classification Society, vol. 33(3), pages 331-373, October.
    5. Salvatore Ingrassia, 2004. "A likelihood-based constrained algorithm for multivariate normal mixture models," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 13(2), pages 151-166, September.
    6. Simon N. Wood, 2001. "Minimizing Model Fitting Objectives That Contain Spurious Local Minima by Bootstrap Restarting," Biometrics, The International Biometric Society, vol. 57(1), pages 240-244, March.
    7. Louis Guttman, 1954. "Some necessary conditions for common-factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 19(2), pages 149-161, June.
    8. McGrory, C.A. & Titterington, D.M., 2007. "Variational approximations in Bayesian model selection for finite mixture distributions," Computational Statistics & Data Analysis, Elsevier, vol. 51(11), pages 5352-5367, July.
    9. Bouveyron, C. & Girard, S. & Schmid, C., 2007. "High-dimensional data clustering," Computational Statistics & Data Analysis, Elsevier, vol. 52(1), pages 502-519, September.
    10. Donald Rubin, 1991. "EM and beyond," Psychometrika, Springer;The Psychometric Society, vol. 56(2), pages 241-254, June.
    11. Biernacki, Christophe & Celeux, Gilles & Govaert, Gerard, 2003. "Choosing starting values for the EM algorithm for getting the highest likelihood in multivariate Gaussian mixture models," Computational Statistics & Data Analysis, Elsevier, vol. 41(3-4), pages 561-575, January.
    12. Karlis, Dimitris & Xekalaki, Evdokia, 2003. "Choosing initial values for the EM algorithm for finite mixtures," Computational Statistics & Data Analysis, Elsevier, vol. 41(3-4), pages 577-590, January.
    13. Ingrassia, Salvatore & Rocci, Roberto, 2007. "Constrained monotone EM algorithms for finite mixture of multivariate Gaussians," Computational Statistics & Data Analysis, Elsevier, vol. 51(11), pages 5339-5351, July.
    14. Haiganoush K. Preisler & Lengyi Han & W. John Braun, 2014. "Dionysus: a stochastic fire growth scenario generator," Environmetrics, John Wiley & Sons, Ltd., vol. 25(6), pages 431-442, September.
    15. Ingrassia, Salvatore & Rocci, Roberto, 2011. "Degeneracy of the EM algorithm for the MLE of multivariate Gaussian mixtures and dynamic constraints," Computational Statistics & Data Analysis, Elsevier, vol. 55(4), pages 1715-1725, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ahmed, Walid M.A. & Sleem, Mohamed A.E., 2023. "Short- and long-run determinants of the price behavior of US clean energy stocks: A dynamic ARDL simulations approach," Energy Economics, Elsevier, vol. 124(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. C. Biernacki & J. Jacques & C. Keribin, 2023. "A Survey on Model-Based Co-Clustering: High Dimension and Estimation Challenges," Journal of Classification, Springer;The Classification Society, vol. 40(2), pages 332-381, July.
    2. Luis Angel García-Escudero & Alfonso Gordaliza & Francesca Greselin & Salvatore Ingrassia & Agustín Mayo-Iscar, 2018. "Eigenvalues and constraints in mixture modeling: geometric and computational issues," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 12(2), pages 203-233, June.
    3. Roberto Rocci & Stefano Antonio Gattone & Roberto Di Mari, 2018. "A data driven equivariant approach to constrained Gaussian mixture modeling," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 12(2), pages 235-260, June.
    4. Melnykov, Volodymyr & Melnykov, Igor, 2012. "Initializing the EM algorithm in Gaussian mixture models with an unknown number of components," Computational Statistics & Data Analysis, Elsevier, vol. 56(6), pages 1381-1395.
    5. Paul D. McNicholas, 2016. "Model-Based Clustering," Journal of Classification, Springer;The Classification Society, vol. 33(3), pages 331-373, October.
    6. Antonio Punzo & Paul. D. McNicholas, 2017. "Robust Clustering in Regression Analysis via the Contaminated Gaussian Cluster-Weighted Model," Journal of Classification, Springer;The Classification Society, vol. 34(2), pages 249-293, July.
    7. Alex Sharp & Glen Chalatov & Ryan P. Browne, 2023. "A dual subspace parsimonious mixture of matrix normal distributions," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 17(3), pages 801-822, September.
    8. Angelo Mazza & Antonio Punzo, 2020. "Mixtures of multivariate contaminated normal regression models," Statistical Papers, Springer, vol. 61(2), pages 787-822, April.
    9. Utkarsh J. Dang & Michael P.B. Gallaugher & Ryan P. Browne & Paul D. McNicholas, 2023. "Model-Based Clustering and Classification Using Mixtures of Multivariate Skewed Power Exponential Distributions," Journal of Classification, Springer;The Classification Society, vol. 40(1), pages 145-167, April.
    10. Derek S. Young & Xi Chen & Dilrukshi C. Hewage & Ricardo Nilo-Poyanco, 2019. "Finite mixture-of-gamma distributions: estimation, inference, and model-based clustering," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 13(4), pages 1053-1082, December.
    11. Salvatore Ingrassia & Antonio Punzo, 2020. "Cluster Validation for Mixtures of Regressions via the Total Sum of Squares Decomposition," Journal of Classification, Springer;The Classification Society, vol. 37(2), pages 526-547, July.
    12. Bouveyron, Charles & Brunet, Camille, 2012. "Theoretical and practical considerations on the convergence properties of the Fisher-EM algorithm," Journal of Multivariate Analysis, Elsevier, vol. 109(C), pages 29-41.
    13. Ingrassia, Salvatore & Rocci, Roberto, 2011. "Degeneracy of the EM algorithm for the MLE of multivariate Gaussian mixtures and dynamic constraints," Computational Statistics & Data Analysis, Elsevier, vol. 55(4), pages 1715-1725, April.
    14. Sahin, Özge & Czado, Claudia, 2022. "Vine copula mixture models and clustering for non-Gaussian data," Econometrics and Statistics, Elsevier, vol. 22(C), pages 136-158.
    15. Antonello Maruotti & Antonio Punzo, 2021. "Initialization of Hidden Markov and Semi‐Markov Models: A Critical Evaluation of Several Strategies," International Statistical Review, International Statistical Institute, vol. 89(3), pages 447-480, December.
    16. Sharon M. McNicholas & Paul D. McNicholas & Daniel A. Ashlock, 2021. "An Evolutionary Algorithm with Crossover and Mutation for Model-Based Clustering," Journal of Classification, Springer;The Classification Society, vol. 38(2), pages 264-279, July.
    17. M. P. B. Gallaugher & C. Biernacki & P. D. McNicholas, 2023. "Parameter-wise co-clustering for high-dimensional data," Computational Statistics, Springer, vol. 38(3), pages 1597-1619, September.
    18. Pietro Coretto & Christian Hennig, 2016. "Robust Improper Maximum Likelihood: Tuning, Computation, and a Comparison With Other Methods for Robust Gaussian Clustering," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1648-1659, October.
    19. Kasa, Siva Rajesh & Rajan, Vaibhav, 2022. "Improved Inference of Gaussian Mixture Copula Model for Clustering and Reproducibility Analysis using Automatic Differentiation," Econometrics and Statistics, Elsevier, vol. 22(C), pages 67-97.
    20. Sanjeena Subedi & Paul D. McNicholas, 2021. "A Variational Approximations-DIC Rubric for Parameter Estimation and Mixture Model Selection Within a Family Setting," Journal of Classification, Springer;The Classification Society, vol. 38(1), pages 89-108, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:csdana:v:127:y:2018:i:c:p:160-171. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/csda .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.