IDEAS home Printed from https://ideas.repec.org/a/spr/stpapr/v61y2020i1d10.1007_s00362-017-0932-6.html
   My bibliography  Save this article

Block average quantile regression for massive dataset

Author

Listed:
  • Qifa Xu

    (Hefei University of Technology)

  • Chao Cai

    (Hefei University of Technology)

  • Cuixia Jiang

    (Hefei University of Technology)

  • Fang Sun

    (Queens College)

  • Xue Huang

    (Florida State University)

Abstract

Nowadays, researchers are frequently confronted with challenges from large-scale data computing. Quantile regression on massive dataset is challenging due to the limitations of computer primary memory. Our proposed block average quantile regression provides a simple and efficient way to implement quantile regression on massive dataset. The major novelty of this method is splitting the entire data into a few blocks, applying the convectional quantile regression onto the data within each block, and deriving final results through aggregating these quantile regression results via simple average approach. While our approach can significantly reduce the storage volume needed for estimation, the resulting estimator is theoretically as efficient as the traditional quantile regression on entire dataset. On the statistical side, asymptotic properties of the resulting estimator are investigated. We verify and illustrate our proposed method via extensive Monte Carlo simulation studies as well as a real-world application.

Suggested Citation

  • Qifa Xu & Chao Cai & Cuixia Jiang & Fang Sun & Xue Huang, 2020. "Block average quantile regression for massive dataset," Statistical Papers, Springer, vol. 61(1), pages 141-165, February.
    • Handle: RePEc:spr:stpapr:v:61:y:2020:i:1:d:10.1007_s00362-017-0932-6
    DOI: 10.1007/s00362-017-0932-6
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s00362-017-0932-6
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s00362-017-0932-6?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Zou, Hui, 2006. "The Adaptive Lasso and Its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1418-1429, December.
    2. Rong Jiang & Wei-Min Qian & Zhan-Gong Zhou, 2016. "Single-index composite quantile regression with heteroscedasticity and general error distributions," Statistical Papers, Springer, vol. 57(1), pages 185-203, March.
    3. Bantli, Faouzi El & Hallin, Marc, 1999. "L1-estimation in linear models with heterogeneous white noise," Statistics & Probability Letters, Elsevier, vol. 45(4), pages 305-315, December.
    4. Okada, Keisuke & Samreth, Sovannroeun, 2012. "The effect of foreign aid on corruption: A quantile regression approach," Economics Letters, Elsevier, vol. 115(2), pages 240-243.
    5. Runze Li & Dennis K.J. Lin & Bing Li, 2013. "Statistical inference in massive data sets," Applied Stochastic Models in Business and Industry, John Wiley & Sons, vol. 29(5), pages 399-409, September.
    6. Peng, Limin & Huang, Yijian, 2008. "Survival Analysis With Quantile Regression Models," Journal of the American Statistical Association, American Statistical Association, vol. 103, pages 637-649, June.
    7. Hu Yang & Huilan Liu, 2016. "Penalized weighted composite quantile estimators with missing covariates," Statistical Papers, Springer, vol. 57(1), pages 69-88, March.
    8. Wang, Huixia & He, Xuming, 2007. "Detecting Differential Expressions in GeneChip Microarray Studies: A Quantile Approach," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 104-112, March.
    9. Fan J. & Li R., 2001. "Variable Selection via Nonconcave Penalized Likelihood and its Oracle Properties," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1348-1360, December.
    10. Koenker R. & Geling O., 2001. "Reappraising Medfly Longevity: A Quantile Regression Survival Analysis," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 458-468, June.
    11. Koenker, Roger W & Bassett, Gilbert, Jr, 1978. "Regression Quantiles," Econometrica, Econometric Society, vol. 46(1), pages 33-50, January.
    12. David Powell & Joachim Wagner, 2021. "The Exporter Productivity Premium Along the Productivity Distribution: Evidence from Quantile Regression with Nonadditive Firm Fixed Effects," World Scientific Book Chapters, in: Joachim Wagner (ed.), MICROECONOMETRIC STUDIES OF FIRMS’ IMPORTS AND EXPORTS Advanced Methods of Analysis and Evidence from German Enterprises, chapter 9, pages 121-149, World Scientific Publishing Co. Pte. Ltd..
    13. Arcones, Miguel A., 1996. "The Bahadur-Kiefer Representation of Lp Regression Estimators," Econometric Theory, Cambridge University Press, vol. 12(2), pages 257-283, June.
    14. Ning, Zijun & Tang, Linjun, 2014. "Estimation and test procedures for composite quantile regression with covariates missing at random," Statistics & Probability Letters, Elsevier, vol. 95(C), pages 15-25.
    15. Xu, Qifa & Niu, Xufeng & Jiang, Cuixia & Huang, Xue, 2015. "The Phillips curve in the US: A nonlinear quantile regression approach," Economic Modelling, Elsevier, vol. 49(C), pages 186-197.
    16. Rahim Alhamzawi, 2015. "Model selection in quantile regression models," Journal of Applied Statistics, Taylor & Francis Journals, vol. 42(2), pages 445-458, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wang, Kangning & Li, Shaomin, 2021. "Robust distributed modal regression for massive data," Computational Statistics & Data Analysis, Elsevier, vol. 160(C).
    2. Daouia, Abdelaati & Padoan, Simone A. & Stupfler, Gilles, 2022. "Optimal weighted pooling for inference about the tail index and extreme quantiles," TSE Working Papers 22-1322, Toulouse School of Economics (TSE), revised 07 Jun 2023.
    3. Islam, M.S. & Das, Barun K. & Das, Pronob & Rahaman, Md Habibur, 2021. "Techno-economic optimization of a zero emission energy system for a coastal community in Newfoundland, Canada," Energy, Elsevier, vol. 220(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jiang, Rong & Qian, Weimin & Zhou, Zhangong, 2012. "Variable selection and coefficient estimation via composite quantile regression with randomly censored data," Statistics & Probability Letters, Elsevier, vol. 82(2), pages 308-317.
    2. Bang, Sungwan & Jhun, Myoungshic, 2012. "Simultaneous estimation and factor selection in quantile regression via adaptive sup-norm regularization," Computational Statistics & Data Analysis, Elsevier, vol. 56(4), pages 813-826.
    3. Xu, Qifa & Zhou, Yingying & Jiang, Cuixia & Yu, Keming & Niu, Xufeng, 2016. "A large CVaR-based portfolio selection model with weight constraints," Economic Modelling, Elsevier, vol. 59(C), pages 436-447.
    4. Alexander Aue & Rex C. Y. Cheung & Thomas C. M. Lee & Ming Zhong, 2014. "Segmented Model Selection in Quantile Regression Using the Minimum Description Length Principle," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(507), pages 1241-1256, September.
    5. Rahim Alhamzawi, 2016. "Bayesian Analysis of Composite Quantile Regression," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 8(2), pages 358-373, October.
    6. Narisetty, Naveen & Koenker, Roger, 2022. "Censored quantile regression survival models with a cure proportion," Journal of Econometrics, Elsevier, vol. 226(1), pages 192-203.
    7. De Backer, Mickael & El Ghouch, Anouar & Van Keilegom, Ingrid, 2017. "An Adapted Loss Function for Censored Quantile Regression," LIDAM Discussion Papers ISBA 2017003, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    8. Tang, Linjun & Zhou, Zhangong & Wu, Changchun, 2012. "Weighted composite quantile estimation and variable selection method for censored regression model," Statistics & Probability Letters, Elsevier, vol. 82(3), pages 653-663.
    9. Weihua Zhao & Riquan Zhang & Yazhao Lv & Jicai Liu, 2017. "Quantile regression and variable selection of single-index coefficient model," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 69(4), pages 761-789, August.
    10. Tang, Yanlin & Song, Xinyuan & Wang, Huixia Judy & Zhu, Zhongyi, 2013. "Variable selection in high-dimensional quantile varying coefficient models," Journal of Multivariate Analysis, Elsevier, vol. 122(C), pages 115-132.
    11. Sungwan Bang & Soo-Heang Eo & Yong Mee Cho & Myoungshic Jhun & HyungJun Cho, 2016. "Non-crossing weighted kernel quantile regression with right censored data," Lifetime Data Analysis: An International Journal Devoted to Statistical Methods and Applications for Time-to-Event Data, Springer, vol. 22(1), pages 100-121, January.
    12. Zhang, Ting & Wang, Lei, 2020. "Smoothed empirical likelihood inference and variable selection for quantile regression with nonignorable missing response," Computational Statistics & Data Analysis, Elsevier, vol. 144(C).
    13. Lan Wang & Yichao Wu & Runze Li, 2012. "Quantile Regression for Analyzing Heterogeneity in Ultra-High Dimension," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 107(497), pages 214-222, March.
    14. Zhangong Zhou & Rong Jiang & Weimin Qian, 2013. "LAD variable selection for linear models with randomly censored data," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 76(2), pages 287-300, February.
    15. Yu-Zhu Tian & Man-Lai Tang & Wai-Sum Chan & Mao-Zai Tian, 2021. "Bayesian bridge-randomized penalized quantile regression for ordinal longitudinal data, with application to firm’s bond ratings," Computational Statistics, Springer, vol. 36(2), pages 1289-1319, June.
    16. Fan, Zengyan & Lian, Heng, 2018. "Quantile regression for additive coefficient models in high dimensions," Journal of Multivariate Analysis, Elsevier, vol. 164(C), pages 54-64.
    17. Torri, Gabriele & Giacometti, Rosella & Tichý, Tomáš, 2021. "Network tail risk estimation in the European banking system," Journal of Economic Dynamics and Control, Elsevier, vol. 127(C).
    18. Lina Liao & Cheolwoo Park & Hosik Choi, 2019. "Penalized expectile regression: an alternative to penalized quantile regression," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 71(2), pages 409-438, April.
    19. Haowen Bao & Zongwu Cai & Yuying Sun & Shouyang Wang, 2023. "Penalized Model Averaging for High Dimensional Quantile Regressions," WORKING PAPERS SERIES IN THEORETICAL AND APPLIED ECONOMICS 202302, University of Kansas, Department of Economics, revised Jan 2023.
    20. Marcelo Cajias & Philipp Freudenreich & Anna Freudenreich, 2020. "Exploring the determinants of real estate liquidity from an alternative perspective: censored quantile regression in real estate research," Journal of Business Economics, Springer, vol. 90(7), pages 1057-1086, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:stpapr:v:61:y:2020:i:1:d:10.1007_s00362-017-0932-6. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.