IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v10y2022i24p4730-d1001814.html
   My bibliography  Save this article

Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era

Author

Listed:
  • Ke-Lin Du

    (Department of Electrical and Computer Engineering, Concordia University, Montreal, QC H3G 1M8, Canada)

  • Chi-Sing Leung

    (Department of Electrical Engineering, City University of Hong Kong, Hong Kong, China)

  • Wai Ho Mow

    (Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China)

  • M. N. S. Swamy

    (Department of Electrical and Computer Engineering, Concordia University, Montreal, QC H3G 1M8, Canada)

Abstract

The single-layer perceptron, introduced by Rosenblatt in 1958, is one of the earliest and simplest neural network models. However, it is incapable of classifying linearly inseparable patterns. A new era of neural network research started in 1986, when the backpropagation (BP) algorithm was rediscovered for training the multilayer perceptron (MLP) model. An MLP with a large number of hidden nodes can function as a universal approximator. To date, the MLP model is the most fundamental and important neural network model. It is also the most investigated neural network model. Even in this AI or deep learning era, the MLP is still among the few most investigated and used neural network models. Numerous new results have been obtained in the past three decades. This survey paper gives a comprehensive and state-of-the-art introduction to the perceptron model, with emphasis on learning, generalization, model selection and fault tolerance. The role of the perceptron model in the deep learning era is also described. This paper provides a concluding survey of perceptron learning, and it covers all the major achievements in the past seven decades. It also serves a tutorial for perceptron learning.

Suggested Citation

  • Ke-Lin Du & Chi-Sing Leung & Wai Ho Mow & M. N. S. Swamy, 2022. "Perceptron: Learning, Generalization, Model Selection, Fault Tolerance, and Role in the Deep Learning Era," Mathematics, MDPI, vol. 10(24), pages 1-46, December.
  • Handle: RePEc:gam:jmathe:v:10:y:2022:i:24:p:4730-:d:1001814
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/10/24/4730/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2227-7390/10/24/4730/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Pier Francesco Orrù & Andrea Zoccheddu & Lorenzo Sassu & Carmine Mattia & Riccardo Cozza & Simone Arena, 2020. "Machine Learning Approach Using MLP and SVM Algorithms for the Fault Prediction of a Centrifugal Pump in the Oil and Gas Industry," Sustainability, MDPI, vol. 12(11), pages 1-15, June.
    2. David F. Shanno, 1978. "Conjugate Gradient Methods with Inexact Searches," Mathematics of Operations Research, INFORMS, vol. 3(3), pages 244-256, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ali Najem Alkawaz & Jeevan Kanesan & Anis Salwa Mohd Khairuddin & Irfan Anjum Badruddin & Sarfaraz Kamangar & Mohamed Hussien & Maughal Ahmed Ali Baig & N. Ameer Ahammad, 2023. "Training Multilayer Neural Network Based on Optimal Control Theory for Limited Computational Resources," Mathematics, MDPI, vol. 11(3), pages 1-15, February.
    2. Ke-Lin Du & M. N. S. Swamy & Zhang-Quan Wang & Wai Ho Mow, 2023. "Matrix Factorization Techniques in Machine Learning, Signal Processing, and Statistics," Mathematics, MDPI, vol. 11(12), pages 1-50, June.
    3. Jianan Chi & Xiangxin Bu & Xiao Zhang & Lijun Wang & Nannan Zhang, 2023. "Insights into Cottonseed Cultivar Identification Using Raman Spectroscopy and Explainable Machine Learning," Agriculture, MDPI, vol. 13(4), pages 1-17, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yasushi Narushima & Shummin Nakayama & Masashi Takemura & Hiroshi Yabe, 2023. "Memoryless Quasi-Newton Methods Based on the Spectral-Scaling Broyden Family for Riemannian Optimization," Journal of Optimization Theory and Applications, Springer, vol. 197(2), pages 639-664, May.
    2. Mustufa Haider Abidi & Usama Umer & Muneer Khan Mohammed & Mohamed K. Aboudaif & Hisham Alkhalefah, 2020. "Automated Maintenance Data Classification Using Recurrent Neural Network: Enhancement by Spotted Hyena-Based Whale Optimization," Mathematics, MDPI, vol. 8(11), pages 1-33, November.
    3. Churlzu Lim & Hanif Sherali & Stan Uryasev, 2010. "Portfolio optimization by minimizing conditional value-at-risk via nondifferentiable optimization," Computational Optimization and Applications, Springer, vol. 46(3), pages 391-415, July.
    4. Chenhong Zhu & J. G. Wang & Na Xu & Wei Liang & Bowen Hu & Peibo Li, 2022. "A Combination Approach of the Numerical Simulation and Data-Driven Analysis for the Impacts of Refracturing Layout and Time on Shale Gas Production," Sustainability, MDPI, vol. 14(23), pages 1-30, December.
    5. Fischer, Manfred M. & Staufer, Petra, 1998. "Optimization in an Error Backpropagation Neural Network Environment with a Performance Test on a Pattern Classification Problem," MPRA Paper 77810, University Library of Munich, Germany.
    6. Hail Jung & Jinsu Jeon & Dahui Choi & Jung-Ywn Park, 2021. "Application of Machine Learning Techniques in Injection Molding Quality Prediction: Implications on Sustainable Manufacturing Industry," Sustainability, MDPI, vol. 13(8), pages 1-16, April.
    7. B. Sellami & Y. Chaib, 2016. "A new family of globally convergent conjugate gradient methods," Annals of Operations Research, Springer, vol. 241(1), pages 497-513, June.
    8. Neculai Andrei, 2013. "Another Conjugate Gradient Algorithm with Guaranteed Descent and Conjugacy Conditions for Large-scale Unconstrained Optimization," Journal of Optimization Theory and Applications, Springer, vol. 159(1), pages 159-182, October.
    9. Hengyang Zhao & Guobao Zhang & Xi Yang, 2022. "GIS Fault Prediction Approach Based on IPSO-LSSVM Algorithm," Sustainability, MDPI, vol. 15(1), pages 1-11, December.
    10. Shaheer Ansari & Afida Ayob & Molla Shahadat Hossain Lipu & Aini Hussain & Mohamad Hanif Md Saad, 2021. "Multi-Channel Profile Based Artificial Neural Network Approach for Remaining Useful Life Prediction of Electric Vehicle Lithium-Ion Batteries," Energies, MDPI, vol. 14(22), pages 1-22, November.
    11. Shummin Nakayama & Yasushi Narushima & Hiroshi Yabe, 2021. "Inexact proximal memoryless quasi-Newton methods based on the Broyden family for minimizing composite functions," Computational Optimization and Applications, Springer, vol. 79(1), pages 127-154, May.
    12. Vanderschueren, Toon & Boute, Robert & Verdonck, Tim & Baesens, Bart & Verbeke, Wouter, 2023. "Optimizing the preventive maintenance frequency with causal machine learning," International Journal of Production Economics, Elsevier, vol. 258(C).
    13. N. Mahdavi-Amiri & M. Shaeiri, 2020. "A conjugate gradient sampling method for nonsmooth optimization," 4OR, Springer, vol. 18(1), pages 73-90, March.
    14. do Carmo, Pedro R.X. & do Monte, João Victor L. & Filho, Assis T. de Oliveira & Freitas, Eduardo & Tigre, Matheus F.F.S.L. & Sadok, Djamel & Kelner, Judith, 2023. "A data-driven model for the optimization of energy consumption of an industrial production boiler in a fiber plant," Energy, Elsevier, vol. 284(C).
    15. Abubakar, Auwal Bala & Kumam, Poom & Malik, Maulana & Ibrahim, Abdulkarim Hassan, 2022. "A hybrid conjugate gradient based approach for solving unconstrained optimization and motion control problems," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 201(C), pages 640-657.
    16. Andrei, Neculai, 2010. "Accelerated scaled memoryless BFGS preconditioned conjugate gradient algorithm for unconstrained optimization," European Journal of Operational Research, Elsevier, vol. 204(3), pages 410-420, August.
    17. Hanif D. Sherali & Churlzu Lim, 2007. "Enhancing Lagrangian Dual Optimization for Linear Programs by Obviating Nondifferentiability," INFORMS Journal on Computing, INFORMS, vol. 19(1), pages 3-13, February.
    18. David Ek & Anders Forsgren, 2021. "Exact linesearch limited-memory quasi-Newton methods for minimizing a quadratic function," Computational Optimization and Applications, Springer, vol. 79(3), pages 789-816, July.
    19. Eduardo Machado & Tiago Pinto & Vanessa Guedes & Hugo Morais, 2021. "Electrical Load Demand Forecasting Using Feed-Forward Neural Networks," Energies, MDPI, vol. 14(22), pages 1-24, November.
    20. T. L. Jensen & M. Diehl, 2017. "An Approach for Analyzing the Global Rate of Convergence of Quasi-Newton and Truncated-Newton Methods," Journal of Optimization Theory and Applications, Springer, vol. 172(1), pages 206-221, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:10:y:2022:i:24:p:4730-:d:1001814. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.