IDEAS home Printed from https://ideas.repec.org/a/spr/stpapr/v64y2023i4d10.1007_s00362-023-01446-9.html
   My bibliography  Save this article

A model robust subsampling approach for Generalised Linear Models in big data settings

Author

Listed:
  • Amalan Mahendran

    (Queensland University of Technology
    Queensland University of Technology)

  • Helen Thompson

    (Queensland University of Technology
    Queensland University of Technology)

  • James M. McGree

    (Queensland University of Technology
    Queensland University of Technology)

Abstract

In today’s modern era of big data, computationally efficient and scalable methods are needed to support timely insights and informed decision making. One such method is subsampling, where a subset of the big data is analysed and used as the basis for inference rather than considering the whole data set. A key question when applying subsampling approaches is how to select an informative subset based on the questions being asked of the data. A recent approach for this has been proposed based on determining subsampling probabilities for each data point, but a limitation of this approach is that the appropriate subsampling probabilities rely on an assumed model for the big data. In this article, to overcome this limitation, we propose a model robust approach where a set of models is considered, and the subsampling probabilities are evaluated based on the weighted average of probabilities that would be obtained if each model was considered singularly. Theoretical results are derived to inform such an approach. Our model robust subsampling approach is applied in a simulation study and in two real-world applications where performance is compared to current subsampling practices. The results show that our model robust approach outperforms alternative methods.

Suggested Citation

  • Amalan Mahendran & Helen Thompson & James M. McGree, 2023. "A model robust subsampling approach for Generalised Linear Models in big data settings," Statistical Papers, Springer, vol. 64(4), pages 1137-1157, August.
  • Handle: RePEc:spr:stpapr:v:64:y:2023:i:4:d:10.1007_s00362-023-01446-9
    DOI: 10.1007/s00362-023-01446-9
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00362-023-01446-9
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s00362-023-01446-9?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Diana Martinez-Mosquera & Rosa Navarrete & Sergio Lujan-Mora, 2020. "Modeling and Management Big Data in Databases—A Systematic Literature Review," Sustainability, MDPI, vol. 12(2), pages 1-41, January.
    2. HaiYing Wang & Min Yang & John Stufken, 2019. "Information-Based Optimal Subdata Selection for Big Data Linear Regression," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(525), pages 393-405, January.
    3. Ariel Kleiner & Ameet Talwalkar & Purnamrita Sarkar & Michael I. Jordan, 2014. "A scalable bootstrap for massive data," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(4), pages 795-816, September.
    4. Vaughan, Gregory, 2020. "Efficient big data model selection with applications to fraud detection," International Journal of Forecasting, Elsevier, vol. 36(3), pages 1116-1127.
    5. Jun Yu & HaiYing Wang, 2022. "Subdata selection algorithm for linear model discrimination," Statistical Papers, Springer, vol. 63(6), pages 1883-1906, December.
    6. Adewale, Adeniyi J. & Xu, Xiaojian, 2010. "Robust designs for generalized linear models with possible overdispersion and misspecified link functions," Computational Statistics & Data Analysis, Elsevier, vol. 54(4), pages 875-890, April.
    7. Yaqiong Yao & HaiYing Wang, 2019. "Optimal subsampling for softmax regression," Statistical Papers, Springer, vol. 60(2), pages 585-599, April.
    8. HaiYing Wang & Rong Zhu & Ping Ma, 2018. "Optimal Subsampling for Large Sample Logistic Regression," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 829-844, April.
    9. Yishu Xue & HaiYing Wang & Jun Yan & Elizabeth D. Schifano, 2020. "An online updating approach for testing the proportional hazards assumption with streams of survival data," Biometrics, The International Biometric Society, vol. 76(1), pages 171-182, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lee, JooChul & Wang, HaiYing & Schifano, Elizabeth D., 2020. "Online updating method to correct for measurement error in big data streams," Computational Statistics & Data Analysis, Elsevier, vol. 149(C).
    2. Su, Miaomiao & Wang, Ruoyu & Wang, Qihua, 2022. "A two-stage optimal subsampling estimation for missing data problems with large-scale data," Computational Statistics & Data Analysis, Elsevier, vol. 173(C).
    3. Jun Yu & Jiaqi Liu & HaiYing Wang, 2023. "Information-based optimal subdata selection for non-linear models," Statistical Papers, Springer, vol. 64(4), pages 1069-1093, August.
    4. Jun Yu & HaiYing Wang, 2022. "Subdata selection algorithm for linear model discrimination," Statistical Papers, Springer, vol. 63(6), pages 1883-1906, December.
    5. J. Lars Kirkby & Dang H. Nguyen & Duy Nguyen & Nhu N. Nguyen, 2022. "Inversion-free subsampling Newton’s method for large sample logistic regression," Statistical Papers, Springer, vol. 63(3), pages 943-963, June.
    6. Ziyang Wang & HaiYing Wang & Nalini Ravishanker, 2023. "Subsampling in Longitudinal Models," Methodology and Computing in Applied Probability, Springer, vol. 25(1), pages 1-29, March.
    7. Xiaohui Yuan & Yong Li & Xiaogang Dong & Tianqing Liu, 2022. "Optimal subsampling for composite quantile regression in big data," Statistical Papers, Springer, vol. 63(5), pages 1649-1676, October.
    8. Feifei Wang & Danyang Huang & Tianchen Gao & Shuyuan Wu & Hansheng Wang, 2022. "Sequential one‐step estimator by sub‐sampling for customer churn analysis with massive data sets," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(5), pages 1753-1786, November.
    9. Lee, JooChul & Schifano, Elizabeth D. & Wang, HaiYing, 2024. "Fast Optimal Subsampling Probability Approximation for Generalized Linear Models," Econometrics and Statistics, Elsevier, vol. 29(C), pages 224-237.
    10. Duarte, Belmiro P.M. & Atkinson, Anthony C. & Oliveira, Nuno M.C., 2024. "Using hierarchical information-theoretic criteria to optimize subsampling of extensive datasets," LSE Research Online Documents on Economics 121641, London School of Economics and Political Science, LSE Library.
    11. Tianzhen Wang & Haixiang Zhang, 2022. "Optimal subsampling for multiplicative regression with massive data," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 76(4), pages 418-449, November.
    12. Deng, Jiayi & Huang, Danyang & Ding, Yi & Zhu, Yingqiu & Jing, Bingyi & Zhang, Bo, 2024. "Subsampling spectral clustering for stochastic block models in large-scale networks," Computational Statistics & Data Analysis, Elsevier, vol. 189(C).
    13. Zhang, Haixiang & Wang, HaiYing, 2021. "Distributed subdata selection for big data via sampling-based approach," Computational Statistics & Data Analysis, Elsevier, vol. 153(C).
    14. Hector, Emily C. & Luo, Lan & Song, Peter X.-K., 2023. "Parallel-and-stream accelerator for computationally fast supervised learning," Computational Statistics & Data Analysis, Elsevier, vol. 177(C).
    15. Sokbae Lee & Serena Ng, 2020. "An Econometric Perspective on Algorithmic Subsampling," Annual Review of Economics, Annual Reviews, vol. 12(1), pages 45-80, August.
    16. Xuejun Ma & Shaochen Wang & Wang Zhou, 2022. "Statistical inference in massive datasets by empirical likelihood," Computational Statistics, Springer, vol. 37(3), pages 1143-1164, July.
    17. Lulu Zuo & Haixiang Zhang & HaiYing Wang & Liuquan Sun, 2021. "Optimal subsample selection for massive logistic regression with distributed data," Computational Statistics, Springer, vol. 36(4), pages 2535-2562, December.
    18. Ma, Xuejun & Wang, Shaochen & Zhou, Wang, 2021. "Testing multivariate quantile by empirical likelihood," Journal of Multivariate Analysis, Elsevier, vol. 182(C).
    19. Laura Deldossi & Elena Pesce & Chiara Tommasi, 2023. "Accounting for outliers in optimal subsampling methods," Statistical Papers, Springer, vol. 64(4), pages 1119-1135, August.
    20. Sanjoy Sinha, 2013. "Robust designs for multivariate logistic regression," METRON, Springer;Sapienza Università di Roma, vol. 71(2), pages 157-173, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:stpapr:v:64:y:2023:i:4:d:10.1007_s00362-023-01446-9. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.