IDEAS home Printed from https://ideas.repec.org/a/eee/chsofr/v178y2024ics0960077923012651.html
   My bibliography  Save this article

Optimized injection of noise in activation functions to improve generalization of neural networks

Author

Listed:
  • Duan, Fabing
  • Chapeau-Blondeau, François
  • Abbott, Derek

Abstract

This paper proposes a flexible probabilistic activation function that enhances the training and operation of artificial neural networks by intentionally injecting noise to gain additional control over the response of each neuron. During the learning phase, the level of injected noise is iteratively optimized by gradient-descent, realizing a form of adaptive stochastic resonance. From simple hard-threshold non-differentiable neuronal responses, controlled injection of noise gives access to a wide range of useful activation functions, with sufficient differentiability to enable gradient-descent learning for both the neuron and the injected-noise levels. Experimental results on function approximation demonstrate injected noise generally converging to non-vanishing optimal levels associated with improved generalization abilities in the neural networks. A theoretical explanation of the generalization improvement based on the path norm bound is presented. With injected noise in the deep neural network, experimental results on classifying images also obtain non-vanishing optimal noise levels to achieve better testing accuracies. The proposed probabilistic activation functions show the potential of adaptive stochastic resonance for useful applications in machine learning.

Suggested Citation

  • Duan, Fabing & Chapeau-Blondeau, François & Abbott, Derek, 2024. "Optimized injection of noise in activation functions to improve generalization of neural networks," Chaos, Solitons & Fractals, Elsevier, vol. 178(C).
  • Handle: RePEc:eee:chsofr:v:178:y:2024:i:c:s0960077923012651
    DOI: 10.1016/j.chaos.2023.114363
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0960077923012651
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.chaos.2023.114363?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yijie Peng & Li Xiao & Bernd Heidergott & L. Jeff Hong & Henry Lam, 2022. "A New Likelihood Ratio Method for Training Artificial Neural Networks," INFORMS Journal on Computing, INFORMS, vol. 34(1), pages 638-655, January.
    2. Duan, Lingling & Ren, Yuhao & Duan, Fabing, 2022. "Adaptive stochastic resonance based convolutional neural network for image classification," Chaos, Solitons & Fractals, Elsevier, vol. 162(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Suo, Jian & Wang, Haiyan & Lian, Wei & Dong, Haitao & Shen, Xiaohong & Yan, Yongsheng, 2023. "Feed-forward cascaded stochastic resonance and its application in ship radiated line signature extraction," Chaos, Solitons & Fractals, Elsevier, vol. 174(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:chsofr:v:178:y:2024:i:c:s0960077923012651. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Thayer, Thomas R. (email available below). General contact details of provider: https://www.journals.elsevier.com/chaos-solitons-and-fractals .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.