IDEAS home Printed from https://ideas.repec.org/a/eee/ejores/v290y2021i3p807-828.html
   My bibliography  Save this article

Optimization problems for machine learning: A survey

Author

Listed:
  • Gambella, Claudio
  • Ghaddar, Bissan
  • Naoum-Sawaya, Joe

Abstract

This paper surveys the machine learning literature and presents in an optimization framework several commonly used machine learning approaches. Particularly, mathematical optimization models are presented for regression, classification, clustering, deep learning, and adversarial learning, as well as new emerging applications in machine teaching, empirical model learning, and Bayesian network structure learning. Such models can benefit from the advancement of numerical optimization techniques which have already played a distinctive role in several machine learning settings. The strengths and the shortcomings of these models are discussed and potential research directions and open problems are highlighted.

Suggested Citation

  • Gambella, Claudio & Ghaddar, Bissan & Naoum-Sawaya, Joe, 2021. "Optimization problems for machine learning: A survey," European Journal of Operational Research, Elsevier, vol. 290(3), pages 807-828.
  • Handle: RePEc:eee:ejores:v:290:y:2021:i:3:p:807-828
    DOI: 10.1016/j.ejor.2020.08.045
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S037722172030758X
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.ejor.2020.08.045?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Miyashiro, Ryuhei & Takano, Yuichi, 2015. "Mixed integer second-order cone programming formulations for variable selection in linear regression," European Journal of Operational Research, Elsevier, vol. 247(3), pages 721-731.
    2. Wang, Gang & Gunasekaran, Angappa & Ngai, Eric W.T. & Papadopoulos, Thanos, 2016. "Big data analytics in logistics and supply chain management: Certain investigations for research and applications," International Journal of Production Economics, Elsevier, vol. 176(C), pages 98-110.
    3. Carrizosa, Emilio & Guerrero, Vanesa, 2014. "Biobjective sparse principal component analysis," Journal of Multivariate Analysis, Elsevier, vol. 132(C), pages 151-159.
    4. Blanquero, Rafael & Carrizosa, Emilio & Molero-Río, Cristina & Romero Morales, Dolores, 2020. "Sparsity in optimal randomized classification trees," European Journal of Operational Research, Elsevier, vol. 284(1), pages 255-272.
    5. Ryuta Tamura & Ken Kobayashi & Yuichi Takano & Ryuhei Miyashiro & Kazuhide Nakata & Tomomi Matsui, 2019. "Mixed integer quadratic optimization formulations for eliminating multicollinearity based on variance inflation factor," Journal of Global Optimization, Springer, vol. 73(2), pages 431-446, February.
    6. Toriello, Alejandro & Vielma, Juan Pablo, 2012. "Fitting piecewise linear continuous functions," European Journal of Operational Research, Elsevier, vol. 219(1), pages 86-95.
    7. John M. Mulvey & Harlan P. Crowder, 1979. "Cluster Analysis: An Application of Lagrangian Relaxation," Management Science, INFORMS, vol. 25(4), pages 329-340, April.
    8. Bertsimas, Dimitris & Copenhaver, Martin S., 2018. "Characterization of the equivalence of robustification and regularization in linear and matrix regression," European Journal of Operational Research, Elsevier, vol. 270(3), pages 931-942.
    9. Paul Mielke & Kenneth Berry, 1997. "Permutation-based multivariate regression analysis: The case for least sum of absolute deviations regression," Annals of Operations Research, Springer, vol. 74(0), pages 259-268, November.
    10. Kawano, Shuichi & Fujisawa, Hironori & Takada, Toyoyuki & Shiroishi, Toshihiko, 2015. "Sparse principal component regression with adaptive loading," Computational Statistics & Data Analysis, Elsevier, vol. 89(C), pages 192-203.
    11. Carrizosa, Emilio & Mladenović, Nenad & Todosijević, Raca, 2013. "Variable neighborhood search for minimum sum-of-squares clustering on networks," European Journal of Operational Research, Elsevier, vol. 230(2), pages 356-363.
    12. Václavík, Roman & Novák, Antonín & Šůcha, Přemysl & Hanzálek, Zdeněk, 2018. "Accelerating the Branch-and-Price Algorithm Using Machine Learning," European Journal of Operational Research, Elsevier, vol. 271(3), pages 1055-1069.
    13. Dunbar, Michelle & Murray, John M. & Cysique, Lucette A. & Brew, Bruce J. & Jeyakumar, Vaithilingam, 2010. "Simultaneous classification and feature selection via convex quadratic programming with application to HIV-associated neurocognitive disorder assessment," European Journal of Operational Research, Elsevier, vol. 206(2), pages 470-478, October.
    14. Ghaddar, Bissan & Naoum-Sawaya, Joe, 2018. "High dimensional data classification and feature selection using support vector machines," European Journal of Operational Research, Elsevier, vol. 265(3), pages 993-1004.
    15. Baumann, P. & Hochbaum, D.S. & Yang, Y.T., 2019. "A comparative study of the leading machine learning techniques and two new optimization algorithms," European Journal of Operational Research, Elsevier, vol. 272(3), pages 1041-1057.
    16. Dimitris Bertsimas & Nathan Kallus, 2020. "From Predictive to Prescriptive Analytics," Management Science, INFORMS, vol. 66(3), pages 1025-1044, March.
    17. Corne, David & Dhaenens, Clarisse & Jourdan, Laetitia, 2012. "Synergies between operations research and data mining: The emerging use of multi-objective approaches," European Journal of Operational Research, Elsevier, vol. 221(3), pages 469-479.
    18. Emilio Carrizosa & Belen Martin-Barragan & Dolores Romero Morales, 2010. "Binarized Support Vector Machines," INFORMS Journal on Computing, INFORMS, vol. 22(1), pages 154-167, February.
    19. Juan Pablo Vielma & Shabbir Ahmed & George Nemhauser, 2010. "Mixed-Integer Models for Nonseparable Piecewise-Linear Optimization: Unifying Framework and Extensions," Operations Research, INFORMS, vol. 58(2), pages 303-315, April.
    20. Amaldi, Edoardo & Coniglio, Stefano, 2013. "A distance-based point-reassignment heuristic for the k-hyperplane clustering problem," European Journal of Operational Research, Elsevier, vol. 227(1), pages 22-29.
    21. Jan, Rong-Hong & Chern, Maw-Sheng, 1994. "Nonlinear integer bilevel programming," European Journal of Operational Research, Elsevier, vol. 72(3), pages 574-587, February.
    22. Santi, Éverton & Aloise, Daniel & Blanchard, Simon J., 2016. "A model for clustering data from heterogeneous dissimilarities," European Journal of Operational Research, Elsevier, vol. 253(3), pages 659-672.
    23. D’Ambrosio, Claudia & Lodi, Andrea & Wiese, Sven & Bragalli, Cristiana, 2015. "Mathematical programming techniques in water network optimization," European Journal of Operational Research, Elsevier, vol. 243(3), pages 774-788.
    24. Kraus, Mathias & Feuerriegel, Stefan & Oztekin, Asil, 2020. "Deep learning in business analytics and operations research: Models, applications and managerial implications," European Journal of Operational Research, Elsevier, vol. 281(3), pages 628-641.
    25. Scheuerer, Stephan & Wendolsky, Rolf, 2006. "A scatter search heuristic for the capacitated clustering problem," European Journal of Operational Research, Elsevier, vol. 169(2), pages 533-547, March.
    26. Piramuthu, Selwyn, 2004. "Evaluating feature selection methods for learning in data mining applications," European Journal of Operational Research, Elsevier, vol. 156(2), pages 483-494, July.
    27. Aytug, Haldun, 2015. "Feature selection for support vector machines using Generalized Benders Decomposition," European Journal of Operational Research, Elsevier, vol. 244(1), pages 210-218.
    28. Claassen, G.D.H. & Hendriks, Th.H.B., 2007. "An application of Special Ordered Sets to a periodic milk collection problem," European Journal of Operational Research, Elsevier, vol. 180(2), pages 754-769, July.
    29. Chikalov, Igor & Hussain, Shahid & Moshkov, Mikhail, 2018. "Bi-criteria optimization of decision trees with applications to data analysis," European Journal of Operational Research, Elsevier, vol. 266(2), pages 689-701.
    30. Mai, Feng & Fry, Michael J. & Ohlmann, Jeffrey W., 2018. "Model-based capacitated clustering with posterior regularization," European Journal of Operational Research, Elsevier, vol. 271(2), pages 594-605.
    31. T. D. Klastorin, 1985. "The p-Median Problem for Cluster Analysis: A Comparative Test Using the Mixture Model Approach," Management Science, INFORMS, vol. 31(1), pages 84-95, January.
    32. Hui Zou & Trevor Hastie, 2005. "Addendum: Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(5), pages 768-768, November.
    33. Saglam, Burcu & Salman, F. Sibel & Sayin, Serpil & Turkay, Metin, 2006. "A mixed-integer programming approach to the clustering problem with an application in customer segmentation," European Journal of Operational Research, Elsevier, vol. 173(3), pages 866-879, September.
    34. Dimitris Bertsimas & Romy Shioda, 2007. "Classification and Regression via Integer Optimization," Operations Research, INFORMS, vol. 55(2), pages 252-271, April.
    35. Mortenson, Michael J. & Doherty, Neil F. & Robinson, Stewart, 2015. "Operational research from Taylorism to Terabytes: A research agenda for the analytics age," European Journal of Operational Research, Elsevier, vol. 241(3), pages 583-595.
    36. Karmitsa, Napsu & Bagirov, Adil M. & Taheri, Sona, 2017. "New diagonal bundle method for clustering problems in large data sets," European Journal of Operational Research, Elsevier, vol. 263(2), pages 367-379.
    37. Sonia Cafieri & Alberto Costa & Pierre Hansen, 2014. "Reformulation of a model for hierarchical divisive graph modularity maximization," Annals of Operations Research, Springer, vol. 222(1), pages 213-226, November.
    38. Olafsson, Sigurdur & Li, Xiaonan & Wu, Shuning, 2008. "Operations research and data mining," European Journal of Operational Research, Elsevier, vol. 187(3), pages 1429-1448, June.
    39. Hui Zou & Trevor Hastie, 2005. "Regularization and variable selection via the elastic net," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 67(2), pages 301-320, April.
    40. Ganesh, K. & Narendran, T.T., 2007. "CLOVES: A cluster-and-search heuristic to solve the vehicle routing problem with delivery and pick-up," European Journal of Operational Research, Elsevier, vol. 178(3), pages 699-717, May.
    41. Rovatti, Riccardo & D’Ambrosio, Claudia & Lodi, Andrea & Martello, Silvano, 2014. "Optimistic MILP modeling of non-linear optimization problems," European Journal of Operational Research, Elsevier, vol. 239(1), pages 32-45.
    42. Carrizosa, Emilio & Martín-Barragán, Belén & Morales, Dolores Romero, 2011. "Detecting relevant variables and interactions in supervised classification," European Journal of Operational Research, Elsevier, vol. 213(1), pages 260-269, August.
    43. Bagirov, Adil M. & Yearwood, John, 2006. "A new nonsmooth optimization algorithm for minimum sum-of-squares clustering problems," European Journal of Operational Research, Elsevier, vol. 170(2), pages 578-596, April.
    44. Andrea Lodi & Giulia Zarpellon, 2017. "Rejoinder on: On learning and branching: a survey," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 25(2), pages 247-248, July.
    45. Bot, Radu Ioan & Lorenz, Nicole, 2011. "Optimization problems in statistical learning: Duality and optimality conditions," European Journal of Operational Research, Elsevier, vol. 213(2), pages 395-404, September.
    46. Andrea Lodi & Giulia Zarpellon, 2017. "On learning and branching: a survey," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 25(2), pages 207-236, July.
    47. Azad, Mohammad & Moshkov, Mikhail, 2017. "Multi-stage optimization of decision and inhibitory trees for decision tables with many-valued decisions," European Journal of Operational Research, Elsevier, vol. 263(3), pages 910-921.
    48. Diaz-Banez, J. M. & Mesa, J. A. & Schobel, A., 2004. "Continuous location of dimensional structures," European Journal of Operational Research, Elsevier, vol. 152(1), pages 22-44, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yang, Yu & Boland, Natashia & Dilkina, Bistra & Savelsbergh, Martin, 2022. "Learning generalized strong branching for set covering, set packing, and 0–1 knapsack problems," European Journal of Operational Research, Elsevier, vol. 301(3), pages 828-840.
    2. Navarro-García, Manuel & Guerrero, Vanesa & Durban, María, 2023. "On constrained smoothing and out-of-range prediction using P-splines: A conic optimization approach," Applied Mathematics and Computation, Elsevier, vol. 441(C).
    3. Benati, Stefano & Ponce, Diego & Puerto, Justo & Rodríguez-Chía, Antonio M., 2022. "A branch-and-price procedure for clustering data that are graph connected," European Journal of Operational Research, Elsevier, vol. 297(3), pages 817-830.
    4. Philippe Jardin, 2023. "Designing topological data to forecast bankruptcy using convolutional neural networks," Annals of Operations Research, Springer, vol. 325(2), pages 1291-1332, June.
    5. Ruomiao Yang & Tianfang Xie & Zhentao Liu, 2022. "The Application of Machine Learning Methods to Predict the Power Output of Internal Combustion Engines," Energies, MDPI, vol. 15(9), pages 1-16, April.
    6. Miguel Angel Ortíz-Barrios & Dayana Milena Coba-Blanco & Juan-José Alfaro-Saíz & Daniela Stand-González, 2021. "Process Improvement Approaches for Increasing the Response of Emergency Departments against the COVID-19 Pandemic: A Systematic Review," IJERPH, MDPI, vol. 18(16), pages 1-31, August.
    7. Doumpos, Michalis & Zopounidis, Constantin & Gounopoulos, Dimitrios & Platanakis, Emmanouil & Zhang, Wenke, 2023. "Operational research and artificial intelligence methods in banking," European Journal of Operational Research, Elsevier, vol. 306(1), pages 1-16.
    8. Emilio Carrizosa & Vanesa Guerrero & Dolores Romero Morales, 2023. "On mathematical optimization for clustering categories in contingency tables," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 17(2), pages 407-429, June.
    9. Ifaei, Pouya & Nazari-Heris, Morteza & Tayerani Charmchi, Amir Saman & Asadi, Somayeh & Yoo, ChangKyoo, 2023. "Sustainable energies and machine learning: An organized review of recent applications and challenges," Energy, Elsevier, vol. 266(C).
    10. Fang, Chao & Han, Zonglei & Wang, Wei & Zio, Enrico, 2023. "Routing UAVs in landslides Monitoring: A neural network heuristic for team orienteering with mandatory visits," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 175(C).
    11. Astorino, Annabella & Avolio, Matteo & Fuduli, Antonio, 2022. "A maximum-margin multisphere approach for binary Multiple Instance Learning," European Journal of Operational Research, Elsevier, vol. 299(2), pages 642-652.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Emilio Carrizosa & Vanesa Guerrero & Dolores Romero Morales, 2023. "On mathematical optimization for clustering categories in contingency tables," Advances in Data Analysis and Classification, Springer;German Classification Society - Gesellschaft für Klassifikation (GfKl);Japanese Classification Society (JCS);Classification and Data Analysis Group of the Italian Statistical Society (CLADAG);International Federation of Classification Societies (IFCS), vol. 17(2), pages 407-429, June.
    2. He Jiang, 2023. "Robust forecasting in spatial autoregressive model with total variation regularization," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 42(2), pages 195-211, March.
    3. Jiang, He & Luo, Shihua & Dong, Yao, 2021. "Simultaneous feature selection and clustering based on square root optimization," European Journal of Operational Research, Elsevier, vol. 289(1), pages 214-231.
    4. Filom, Siyavash & Amiri, Amir M. & Razavi, Saiedeh, 2022. "Applications of machine learning methods in port operations – A systematic literature review," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 161(C).
    5. Ghaddar, Bissan & Naoum-Sawaya, Joe, 2018. "High dimensional data classification and feature selection using support vector machines," European Journal of Operational Research, Elsevier, vol. 265(3), pages 993-1004.
    6. Erkip, Nesim Kohen, 2023. "Can accessing much data reshape the theory? Inventory theory under the challenge of data-driven systems," European Journal of Operational Research, Elsevier, vol. 308(3), pages 949-959.
    7. Hauser, Matthias & Flath, Christoph M. & Thiesse, Frédéric, 2021. "Catch me if you scan: Data-driven prescriptive modeling for smart store environments," European Journal of Operational Research, Elsevier, vol. 294(3), pages 860-873.
    8. Karmitsa, Napsu & Bagirov, Adil M. & Taheri, Sona, 2017. "New diagonal bundle method for clustering problems in large data sets," European Journal of Operational Research, Elsevier, vol. 263(2), pages 367-379.
    9. Raeesi, Ramin & Sahebjamnia, Navid & Mansouri, S. Afshin, 2023. "The synergistic effect of operational research and big data analytics in greening container terminal operations: A review and future directions," European Journal of Operational Research, Elsevier, vol. 310(3), pages 943-973.
    10. Tom Pape, 2020. "Prioritising data items for business analytics: Framework and application to human resources," Papers 2012.13813, arXiv.org.
    11. Blanquero, Rafael & Carrizosa, Emilio & Molero-Río, Cristina & Morales, Dolores Romero, 2022. "On sparse optimal regression trees," European Journal of Operational Research, Elsevier, vol. 299(3), pages 1045-1054.
    12. Steffen Rebennack & Vitaliy Krasko, 2020. "Piecewise Linear Function Fitting via Mixed-Integer Linear Programming," INFORMS Journal on Computing, INFORMS, vol. 32(2), pages 507-530, April.
    13. Lingxun Kong & Christos T. Maravelias, 2020. "On the Derivation of Continuous Piecewise Linear Approximating Functions," INFORMS Journal on Computing, INFORMS, vol. 32(3), pages 531-546, July.
    14. Emilio Carrizosa & Cristina Molero-Río & Dolores Romero Morales, 2021. "Mathematical optimization in classification and regression trees," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 29(1), pages 5-33, April.
    15. Kawano, Shuichi & Fujisawa, Hironori & Takada, Toyoyuki & Shiroishi, Toshihiko, 2018. "Sparse principal component regression for generalized linear models," Computational Statistics & Data Analysis, Elsevier, vol. 124(C), pages 180-196.
    16. De Bock, Koen W. & Coussement, Kristof & Caigny, Arno De & Słowiński, Roman & Baesens, Bart & Boute, Robert N. & Choi, Tsan-Ming & Delen, Dursun & Kraus, Mathias & Lessmann, Stefan & Maldonado, Sebast, 2024. "Explainable AI for Operational Research: A defining framework, methods, applications, and a research agenda," European Journal of Operational Research, Elsevier, vol. 317(2), pages 249-272.
    17. Koen W. de Bock & Kristof Coussement & Arno De Caigny & Roman Slowiński & Bart Baesens & Robert N Boute & Tsan-Ming Choi & Dursun Delen & Mathias Kraus & Stefan Lessmann & Sebastián Maldonado & David , 2023. "Explainable AI for Operational Research: A Defining Framework, Methods, Applications, and a Research Agenda," Post-Print hal-04219546, HAL.
    18. Shao, Yu & Zhou, Xinhong & Yu, Tingchao & Zhang, Tuqiao & Chu, Shipeng, 2024. "Pump scheduling optimization in water distribution system based on mixed integer linear programming," European Journal of Operational Research, Elsevier, vol. 313(3), pages 1140-1151.
    19. Marc Gürtler & Marvin Zöllner, 2023. "Heterogeneities among credit risk parameter distributions: the modality defines the best estimation method," OR Spectrum: Quantitative Approaches in Management, Springer;Gesellschaft für Operations Research e.V., vol. 45(1), pages 251-287, March.
    20. Yuichi Takano & Ryuhei Miyashiro, 2020. "Best subset selection via cross-validation criterion," TOP: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 28(2), pages 475-488, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ejores:v:290:y:2021:i:3:p:807-828. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eor .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.