IDEAS home Printed from https://ideas.repec.org/a/spr/coopap/v61y2015i3p609-634.html
   My bibliography  Save this article

Path following in the exact penalty method of convex programming

Author

Listed:
  • Hua Zhou
  • Kenneth Lange

Abstract

Classical penalty methods solve a sequence of unconstrained problems that put greater and greater stress on meeting the constraints. In the limit as the penalty constant tends to $$\infty $$ ∞ , one recovers the constrained solution. In the exact penalty method, squared penalties are replaced by absolute value penalties, and the solution is recovered for a finite value of the penalty constant. In practice, the kinks in the penalty and the unknown magnitude of the penalty constant prevent wide application of the exact penalty method in nonlinear programming. In this article, we examine a strategy of path following consistent with the exact penalty method. Instead of performing optimization at a single penalty constant, we trace the solution as a continuous function of the penalty constant. Thus, path following starts at the unconstrained solution and follows the solution path as the penalty constant increases. In the process, the solution path hits, slides along, and exits from the various constraints. For quadratic programming, the solution path is piecewise linear and takes large jumps from constraint to constraint. For a general convex program, the solution path is piecewise smooth, and path following operates by numerically solving an ordinary differential equation segment by segment. Our diverse applications to (a) projection onto a convex set, (b) nonnegative least squares, (c) quadratically constrained quadratic programming, (d) geometric programming, and (e) semidefinite programming illustrate the mechanics and potential of path following. The final detour to image denoising demonstrates the relevance of path following to regularized estimation in inverse problems. In regularized estimation, one follows the solution path as the penalty constant decreases from a large value. Copyright Springer Science+Business Media New York 2015

Suggested Citation

  • Hua Zhou & Kenneth Lange, 2015. "Path following in the exact penalty method of convex programming," Computational Optimization and Applications, Springer, vol. 61(3), pages 609-634, July.
  • Handle: RePEc:spr:coopap:v:61:y:2015:i:3:p:609-634
    DOI: 10.1007/s10589-015-9732-x
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s10589-015-9732-x
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1007/s10589-015-9732-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Daniel D. Lee & H. Sebastian Seung, 1999. "Learning the parts of objects by non-negative matrix factorization," Nature, Nature, vol. 401(6755), pages 788-791, October.
    2. Berry, Michael W. & Browne, Murray & Langville, Amy N. & Pauca, V. Paul & Plemmons, Robert J., 2007. "Algorithms and applications for approximate nonnegative matrix factorization," Computational Statistics & Data Analysis, Elsevier, vol. 52(1), pages 155-173, September.
    3. Elmor L. Peterson, 1976. "Fenchel's Duality Thereom in Generalized Geometric Programming," Discussion Papers 252, Northwestern University, Center for Mathematical Studies in Economics and Management Science.
    4. Elmor L. Peterson, 1976. "Optimality Conditions in Generalized Geometric Programming," Discussion Papers 221, Northwestern University, Center for Mathematical Studies in Economics and Management Science.
    5. Stephen P. Boyd & Seung-Jean Kim & Dinesh D. Patil & Mark A. Horowitz, 2005. "Digital Circuit Optimization via Geometric Programming," Operations Research, INFORMS, vol. 53(6), pages 899-932, December.
    6. Hua Zhou & Yichao Wu, 2014. "A Generic Path Algorithm for Regularized Statistical Estimation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(506), pages 686-699, June.
    7. David G. Luenberger & Yinyu Ye, 2008. "Linear and Nonlinear Programming," International Series in Operations Research and Management Science, Springer, edition 0, number 978-0-387-74503-9, December.
    8. Willard I. Zangwill, 1967. "Non-Linear Programming Via Penalty Functions," Management Science, INFORMS, vol. 13(5), pages 344-358, January.
    9. Hua Zhou & Kenneth L. Lange, 2010. "On the Bumpy Road to the Dominant Mode," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 37(4), pages 612-631, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bennet Gebken & Katharina Bieker & Sebastian Peitz, 2023. "On the structure of regularization paths for piecewise differentiable regularization terms," Journal of Global Optimization, Springer, vol. 85(3), pages 709-741, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kenneth Lange & Eric C. Chi & Hua Zhou, 2014. "Rejoinder," International Statistical Review, International Statistical Institute, vol. 82(1), pages 81-89, April.
    2. Belleh Fontem, 2023. "Robust Chance-Constrained Geometric Programming with Application to Demand Risk Mitigation," Journal of Optimization Theory and Applications, Springer, vol. 197(2), pages 765-797, May.
    3. Jianfei Cao & Han Yang & Jianshu Lv & Quanyuan Wu & Baolei Zhang, 2023. "Estimating Soil Salinity with Different Levels of Vegetation Cover by Using Hyperspectral and Non-Negative Matrix Factorization Algorithm," IJERPH, MDPI, vol. 20(4), pages 1-15, February.
    4. Takehiro Sano & Tsuyoshi Migita & Norikazu Takahashi, 2022. "A novel update rule of HALS algorithm for nonnegative matrix factorization and Zangwill’s global convergence," Journal of Global Optimization, Springer, vol. 84(3), pages 755-781, November.
    5. Yoshi Fujiwara & Rubaiyat Islam, 2021. "Bitcoin's Crypto Flow Network," Papers 2106.11446, arXiv.org, revised Jul 2021.
    6. Yin Liu & Sam Davanloo Tajbakhsh, 2023. "Stochastic Composition Optimization of Functions Without Lipschitz Continuous Gradient," Journal of Optimization Theory and Applications, Springer, vol. 198(1), pages 239-289, July.
    7. Hiroyasu Abe & Hiroshi Yadohisa, 2019. "Orthogonal nonnegative matrix tri-factorization based on Tweedie distributions," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 13(4), pages 825-853, December.
    8. Guowei Yang & Lin Zhang & Minghua Wan, 2022. "Exponential Graph Regularized Non-Negative Low-Rank Factorization for Robust Latent Representation," Mathematics, MDPI, vol. 10(22), pages 1-20, November.
    9. Jingu Kim & Yunlong He & Haesun Park, 2014. "Algorithms for nonnegative matrix and tensor factorizations: a unified view based on block coordinate descent framework," Journal of Global Optimization, Springer, vol. 58(2), pages 285-319, February.
    10. C.H. Scott & T.R. Jefferson, 2003. "On Duality for a Class of Quasiconcave Multiplicative Programs," Journal of Optimization Theory and Applications, Springer, vol. 117(3), pages 575-583, June.
    11. Soodabeh Asadi & Janez Povh, 2021. "A Block Coordinate Descent-Based Projected Gradient Algorithm for Orthogonal Non-Negative Matrix Factorization," Mathematics, MDPI, vol. 9(5), pages 1-22, March.
    12. Thiel, Michel & Sauwen, Nicolas & Khamiakova, Tastian & Maes, Tor & Govaerts, Bernadette, 2021. "Comparison of chemometrics strategies for the spectroscopic monitoring of active pharmaceutical ingredients in chemical reactions," LIDAM Discussion Papers ISBA 2021009, Université catholique de Louvain, Institute of Statistics, Biostatistics and Actuarial Sciences (ISBA).
    13. Jason B Castro & Arvind Ramanathan & Chakra S Chennubhotla, 2013. "Categorical Dimensions of Human Odor Descriptor Space Revealed by Non-Negative Matrix Factorization," PLOS ONE, Public Library of Science, vol. 8(9), pages 1-16, September.
    14. M. Moghadam & K. Aminian & M. Asghari & M. Parnianpour, 2013. "How well do the muscular synergies extracted via non-negative matrix factorisation explain the variation of torque at shoulder joint?," Computer Methods in Biomechanics and Biomedical Engineering, Taylor & Francis Journals, vol. 16(3), pages 291-301.
    15. Duy Khuong Nguyen & Tu Bao Ho, 2017. "Accelerated parallel and distributed algorithm using limited internal memory for nonnegative matrix factorization," Journal of Global Optimization, Springer, vol. 68(2), pages 307-328, June.
    16. Chiu, Nan-Chieh & Fang, Shu-Cherng & Lavery, John E. & Lin, Jen-Yen & Wang, Yong, 2008. "Approximating term structure of interest rates using cubic L1 splines," European Journal of Operational Research, Elsevier, vol. 184(3), pages 990-1004, February.
    17. FUJIWARA Yoshi & INOUE Hiroyasu & YAMAGUCHI Takayuki & AOYAMA Hideaki & TANAKA Takuma & KIKUCHI Kentaro, 2021. "Money Flow Network Among Firms' Accounts in a Regional Bank of Japan," Discussion papers 21005, Research Institute of Economy, Trade and Industry (RIETI).
    18. Bastian Schaefermeier & Gerd Stumme & Tom Hanika, 2021. "Topic space trajectories," Scientometrics, Springer;Akadémiai Kiadó, vol. 126(7), pages 5759-5795, July.
    19. R. I. Boţ & S. M. Grad & G. Wanka, 2006. "Fenchel-Lagrange Duality Versus Geometric Duality in Convex Optimization," Journal of Optimization Theory and Applications, Springer, vol. 129(1), pages 33-54, April.
    20. Hiroyasu Abe & Hiroshi Yadohisa, 2017. "A non-negative matrix factorization model based on the zero-inflated Tweedie distribution," Computational Statistics, Springer, vol. 32(2), pages 475-499, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:coopap:v:61:y:2015:i:3:p:609-634. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.