IDEAS home Printed from https://ideas.repec.org/p/zbw/sfb475/200825.html
   My bibliography  Save this paper

Convergence analysis of generalized iteratively reweighted least squares algorithms on convex function spaces

Author

Listed:
  • Bissantz, Nicolai
  • Dümbgen, Lutz
  • Munk, Axel
  • Stratmann, Bernd

Abstract

The computation of robust regression estimates often relies on minimization of a convex functional on a convex set. In this paper we discuss a general technique for a large class of convex functionals to compute the minimizers iteratively which is closely related to majorization-minimization algorithms. Our approach is based on a quadratic approximation of the functional to be minimized and includes the iteratively reweighted least squares algorithm as a special case. We prove convergence on convex function spaces for general coercive and convex functionals F and derive geometric convergence in certain unconstrained settings. The algorithm is applied to TV penalized quantile regression and is compared with a step size corrected Newton-Raphson algorithm. It is found that typically in the first steps the iteratively reweighted least squares algorithm performs significantly better, whereas the Newton type method outpaces the former only after many iterations. Finally, in the setting of bivariate regression with unimodality constraints we illustrate how this algorithm allows to utilize highly efficient algorithms for special quadratic programs in more complex settings.

Suggested Citation

  • Bissantz, Nicolai & Dümbgen, Lutz & Munk, Axel & Stratmann, Bernd, 2008. "Convergence analysis of generalized iteratively reweighted least squares algorithms on convex function spaces," Technical Reports 2008,25, Technische Universität Dortmund, Sonderforschungsbereich 475: Komplexitätsreduktion in multivariaten Datenstrukturen.
    • Handle: RePEc:zbw:sfb475:200825
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/36620/1/600423409.PDF
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Dankmar Böhning & Bruce Lindsay, 1988. "Monotonicity of quadratic-approximation algorithms," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 40(4), pages 641-663, December.
    2. David E. Coleman & Paul W. Holland & Neil Kaden & Virginia Klema, 1977. "A System of Subroutines For Iteratively Reweighted Least Squares Computations," NBER Working Papers 0189, National Bureau of Economic Research, Inc.
    3. Hans Künsch, 1994. "Robust priors for smoothing and image restoration," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 46(1), pages 1-19, March.
    4. Hardle, W. & Marron, J., 1989. "Bootstrap Simultaneous Error Bars For Nonparametric Regression," LIDAM Discussion Papers CORE 1989023, Université catholique de Louvain, Center for Operations Research and Econometrics (CORE).
    5. Lejeune, Michel G. & Sarda, Pascal, 1988. "Quantile regression: a nonparametric approach," Computational Statistics & Data Analysis, Elsevier, vol. 6(3), pages 229-239, April.
    6. Brown, Bruce M. & Hall, Peter & Young, G. Alastair, 1997. "On the Effect of Inliers on the Spatial Median," Journal of Multivariate Analysis, Elsevier, vol. 63(1), pages 88-104, October.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Severance-Lossin, E. & Sperlich, S., 1995. "Estimation of Derivatives for Additive Separable Models," SFB 373 Discussion Papers 1995,60, Humboldt University of Berlin, Interdisciplinary Research Project 373: Quantification and Simulation of Economic Processes.
    2. Peter Hall & Joel L. Horowitz, 2012. "A simple bootstrap method for constructing nonparametric confidence bands for functions," CeMMAP working papers CWP14/12, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    3. Wang, Fa, 2017. "Maximum likelihood estimation and inference for high dimensional nonlinear factor models with application to factor-augmented regressions," MPRA Paper 93484, University Library of Munich, Germany, revised 19 May 2019.
    4. de Leeuw, Jan, 2006. "Principal component analysis of binary data by iterated singular value decomposition," Computational Statistics & Data Analysis, Elsevier, vol. 50(1), pages 21-39, January.
    5. Roussille, Nina & Scuderi, Benjamin, 2023. "Bidding for Talent: A Test of Conduct in a High-Wage Labor Market," IZA Discussion Papers 16352, Institute of Labor Economics (IZA).
    6. Kenneth Lange & Hua Zhou, 2022. "A Legacy of EM Algorithms," International Statistical Review, International Statistical Institute, vol. 90(S1), pages 52-66, December.
    7. Politis, Dimitris N, 2010. "Model-free Model-fitting and Predictive Distributions," University of California at San Diego, Economics Working Paper Series qt67j6s174, Department of Economics, UC San Diego.
    8. Wang, Fa, 2022. "Maximum likelihood estimation and inference for high dimensional generalized factor models with application to factor-augmented regressions," Journal of Econometrics, Elsevier, vol. 229(1), pages 180-200.
    9. Qian, Junhui & Wang, Le, 2012. "Estimating semiparametric panel data models by marginal integration," Journal of Econometrics, Elsevier, vol. 167(2), pages 483-493.
    10. Utkarsh J. Dang & Michael P.B. Gallaugher & Ryan P. Browne & Paul D. McNicholas, 2023. "Model-Based Clustering and Classification Using Mixtures of Multivariate Skewed Power Exponential Distributions," Journal of Classification, Springer;The Classification Society, vol. 40(1), pages 145-167, April.
    11. Kauermann, Göran & Müller, Marlene & Carroll, Raymond J., 1998. "The efficiency of bias-corrected estimators for nonparametric kernel estimation based on local estimating equations," Statistics & Probability Letters, Elsevier, vol. 37(1), pages 41-47, January.
    12. Tian, Guo-Liang & Tang, Man-Lai & Liu, Chunling, 2012. "Accelerating the quadratic lower-bound algorithm via optimizing the shrinkage parameter," Computational Statistics & Data Analysis, Elsevier, vol. 56(2), pages 255-265.
    13. Tian, Guo-Liang & Tang, Man-Lai & Fang, Hong-Bin & Tan, Ming, 2008. "Efficient methods for estimating constrained parameters with applications to regularized (lasso) logistic regression," Computational Statistics & Data Analysis, Elsevier, vol. 52(7), pages 3528-3542, March.
    14. Stefan Profit & Stefan Sperlich, 2004. "Non-uniformity of job-matching in a transition economy - A nonparametric analysis for the Czech Republic," Applied Economics, Taylor & Francis Journals, vol. 36(7), pages 695-714.
    15. Ollila, Esa & Oja, Hannu & Croux, Christophe, 2003. "The affine equivariant sign covariance matrix: asymptotic behavior and efficiencies," Journal of Multivariate Analysis, Elsevier, vol. 87(2), pages 328-355, November.
    16. Dankmar Böhning, 1992. "Multinomial logistic regression algorithm," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 44(1), pages 197-200, March.
    17. Bohning, Dankmar, 1999. "The lower bound method in probit regression," Computational Statistics & Data Analysis, Elsevier, vol. 30(1), pages 13-17, March.
    18. Lindgren, Anna, 1997. "Quantile regression with censored data using generalized L1 minimization," Computational Statistics & Data Analysis, Elsevier, vol. 23(4), pages 509-524, February.
    19. Oliver Linton & Pedro Gozalo, 1995. "Testing Additivity in Generalized Nonparametric Regression Models," Cowles Foundation Discussion Papers 1106, Cowles Foundation for Research in Economics, Yale University.
    20. Liu, Wenchen & Tang, Yincai & Wu, Xianyi, 2020. "Separating variables to accelerate non-convex regularized optimization," Computational Statistics & Data Analysis, Elsevier, vol. 147(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:sfb475:200825. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/isdorde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.