IDEAS home Printed from https://ideas.repec.org/a/spr/compst/v35y2020i3d10.1007_s00180-019-00950-7.html
   My bibliography  Save this article

Parallel computing in linear mixed models

Author

Listed:
  • Fulya Gokalp Yavuz

    (Purdue University
    Middle East Technical University)

  • Barret Schloerke

    (Purdue University)

Abstract

In this study, we propose a parallel programming method for linear mixed models (LMM) generated from big data. A commonly used algorithm, expectation maximization (EM), is preferred for its use of maximum likelihood estimations, as the estimations are stable and simple. However, EM has a high computation cost. In our proposed method, we use a divide and recombine to split the data into smaller subsets, running the algorithm steps in parallel on multiple local cores and combining the results. The proposed method is used to fit LMM with dense and sparse parameters and for large number of observations. It is faster than the classical approach and generalizes for big data. Supplementary sources for the proposed method are available in the R package lmmpar.

Suggested Citation

  • Fulya Gokalp Yavuz & Barret Schloerke, 2020. "Parallel computing in linear mixed models," Computational Statistics, Springer, vol. 35(3), pages 1273-1289, September.
    • Handle: RePEc:spr:compst:v:35:y:2020:i:3:d:10.1007_s00180-019-00950-7
    DOI: 10.1007/s00180-019-00950-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00180-019-00950-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00180-019-00950-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Fulya Gokalp Yavuz & Olcay Arslan, 2018. "Linear mixed model with Laplace distribution (LLMM)," Statistical Papers, Springer, vol. 59(1), pages 271-289, March.
    2. Wickham, Hadley, 2011. "The Split-Apply-Combine Strategy for Data Analysis," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 40(i01).
    3. Kane, Michael & Emerson, John W. & Weston, Stephen, 2013. "Scalable Strategies for Computing with Massive Data," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 55(i14).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ferraro, Maria Brigida, 2024. "Fuzzy k-Means: history and applications," Econometrics and Statistics, Elsevier, vol. 30(C), pages 110-123.
    2. Ravic Nijbroek & Kristin Piikki & Mats Söderström & Bas Kempen & Katrine G. Turner & Simeon Hengari & John Mutua, 2018. "Soil Organic Carbon Baselines for Land Degradation Neutrality: Map Accuracy and Cost Tradeoffs with Respect to Complexity in Otjozondjupa, Namibia," Sustainability, MDPI, vol. 10(5), pages 1-20, May.
    3. Miller, Christine M.F. & Waterhouse, Hannah & Harter, Thomas & Fadel, James G. & Meyer, Deanne, 2020. "Quantifying the uncertainty in nitrogen application and groundwater nitrate leaching in manure based cropping systems," Agricultural Systems, Elsevier, vol. 184(C).
    4. Aaron T L Lun & Hervé Pagès & Mike L Smith, 2018. "beachmat: A Bioconductor C++ API for accessing high-throughput biological data from a variety of R matrix types," PLOS Computational Biology, Public Library of Science, vol. 14(5), pages 1-15, May.
    5. Sean McKenzie & Hilary Parkinson & Jane Mangold & Mary Burrows & Selena Ahmed & Fabian Menalled, 2018. "Perceptions, Experiences, and Priorities Supporting Agroecosystem Management Decisions Differ among Agricultural Producers, Consultants, and Researchers," Sustainability, MDPI, vol. 10(11), pages 1-19, November.
    6. Ahmad Alsaber & Jiazhu Pan & Adeeba Al-Herz & Dhary S. Alkandary & Adeeba Al-Hurban & Parul Setiya & on behalf of the KRRD Group, 2020. "Influence of Ambient Air Pollution on Rheumatoid Arthritis Disease Activity Score Index," IJERPH, MDPI, vol. 17(2), pages 1-17, January.
    7. Rebecca Hong & Monica Perkins & Belinda J. Gabbe & Lincoln M. Tracy, 2022. "Comparing Peak Burn Injury Times and Characteristics in Australia and New Zealand," IJERPH, MDPI, vol. 19(15), pages 1-9, August.
    8. Yu, Lining & Härdle, Wolfgang Karl & Borke, Lukas & Benschop, Thijs, 2017. "FRM: A financial risk meter based on penalizing tail events occurrence," SFB 649 Discussion Papers 2017-003, Humboldt University Berlin, Collaborative Research Center 649: Economic Risk.
    9. Merl, Robert & Stöckl, Thomas & Palan, Stefan, 2023. "Insider trading regulation and shorting constraints. Evaluating the joint effects of two market interventions," Journal of Banking & Finance, Elsevier, vol. 154(C).
    10. Paul J McMurdie & Susan Holmes, 2014. "Waste Not, Want Not: Why Rarefying Microbiome Data Is Inadmissible," PLOS Computational Biology, Public Library of Science, vol. 10(4), pages 1-12, April.
    11. Debanjan Mukherjee & Ângelo Ferreira Chora & Jean-Christophe Lone & Ricardo S. Ramiro & Birte Blankenhaus & Karine Serre & Mário Ramirez & Isabel Gordo & Marc Veldhoen & Patrick Varga-Weisz & Maria M., 2022. "Host lung microbiota promotes malaria-associated acute respiratory distress syndrome," Nature Communications, Nature, vol. 13(1), pages 1-14, December.
    12. Eff, Ellis Anthon, 2013. "Settlers and surnames: An atlas illustrating the origins of settlers in 19th century America," MPRA Paper 56296, University Library of Munich, Germany.
    13. Hofert, Marius & Mächler, Martin, 2016. "Parallel and Other Simulations in R Made Easy: An End-to-End Study," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 69(i04).
    14. Lang Zhao & Yuan Zeng & Zhidong Wang & Yizheng Li & Dong Peng & Yao Wang & Xueying Wang, 2023. "Robust Optimal Scheduling of Integrated Energy Systems Considering the Uncertainty of Power Supply and Load in the Power Market," Energies, MDPI, vol. 16(14), pages 1-14, July.
    15. Nicole Smialek & Joachim Pander & Arne Heinrich & Juergen Geist, 2021. "Sneaker, Dweller and Commuter: Evaluating Fish Behavior in Net-Based Monitoring at Hydropower Plants—A Case Study on Brown Trout ( Salmo trutta )," Sustainability, MDPI, vol. 13(2), pages 1-20, January.
    16. Stefan LINGNER & Eiko THIESSEN & Kerrin MÜLLER & Eberhard HARTUNG, 2018. "Dry Biomass Estimation of Hedge Banks: Allometric Equation vs. Structure from Motion via Unmanned Aerial Vehicle," Journal of Forest Science, Czech Academy of Agricultural Sciences, vol. 64(4), pages 149-156.
    17. Guney, Yesim & Arslan, Olcay & Yavuz, Fulya Gokalp, 2022. "Robust estimation in multivariate heteroscedastic regression models with autoregressive covariance structures using EM algorithm," Journal of Multivariate Analysis, Elsevier, vol. 191(C).
    18. Wilson Lara & Stella Bogino & Felipe Bravo, 2018. "Multilevel analysis of dendroclimatic series with the R-package BIOdry," PLOS ONE, Public Library of Science, vol. 13(5), pages 1-23, May.
    19. Bischl, Bernd & Lang, Michel & Mersmann, Olaf & Rahnenführer, Jörg & Weihs, Claus, 2015. "BatchJobs and BatchExperiments: Abstraction Mechanisms for Using R in Batch Environments," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 64(i11).
    20. Mammen, Enno & Wilke, Ralf A. & Zapp, Kristina Maria, 2022. "Estimation of group structures in panel models with individual fixed effects," ZEW Discussion Papers 22-023, ZEW - Leibniz Centre for European Economic Research.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:compst:v:35:y:2020:i:3:d:10.1007_s00180-019-00950-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.