IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i2p316-d1028169.html
   My bibliography  Save this article

A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition

Author

Listed:
  • Xiaojun Zhou

    (School of Information Science and Engineering, Yunnan University, Kunming 650500, China)

  • Chunna Zhao

    (School of Information Science and Engineering, Yunnan University, Kunming 650500, China)

  • Yaqun Huang

    (School of Information Science and Engineering, Yunnan University, Kunming 650500, China)

Abstract

In this paper, a deep learning optimization algorithm is proposed, which is based on the Grünwald–Letnikov (G-L) fractional order definition. An optimizer fractional calculus gradient descent based on the G-L fractional order definition (FCGD_G-L) is designed. Using the short-memory effect of the G-L fractional order definition, the derivation only needs 10 time steps. At the same time, via the transforming formula of the G-L fractional order definition, the Gamma function is eliminated. Thereby, it can achieve the unification of the fractional order and integer order in FCGD_G-L. To prevent the parameters falling into local optimum, a small disturbance is added in the unfolding process. According to the stochastic gradient descent (SGD) and Adam, two optimizers’ fractional calculus stochastic gradient descent based on the G-L definition (FCSGD_G-L), and the fractional calculus Adam based on the G-L definition (FCAdam_G-L), are obtained. These optimizers are validated on two time series prediction tasks. With the analysis of train loss, related experiments show that FCGD_G-L has the faster convergence speed and better convergence accuracy than the conventional integer order optimizer. Because of the fractional order property, the optimizer exhibits stronger robustness and generalization ability. Through the test sets, using the saved optimal model to evaluate, FCGD_G-L also shows a better evaluation effect than the conventional integer order optimizer.

Suggested Citation

  • Xiaojun Zhou & Chunna Zhao & Yaqun Huang, 2023. "A Deep Learning Optimizer Based on Grünwald–Letnikov Fractional Order Definition," Mathematics, MDPI, vol. 11(2), pages 1-15, January.
    • Handle: RePEc:gam:jmathe:v:11:y:2023:i:2:p:316-:d:1028169
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/2/316/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/11/2/316/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Zeshan Aslam Khan & Naveed Ishtiaq Chaudhary & Syed Zubair, 2019. "Fractional stochastic gradient descent for recommender systems," Electronic Markets, Springer;IIM University of St. Gallen, vol. 29(2), pages 275-285, June.
    2. El Mehdi Lotfi & Houssine Zine & Delfim F. M. Torres & Noura Yousfi, 2022. "The Power Fractional Calculus: First Definitions and Properties with Applications to Power Fractional Differential Equations," Mathematics, MDPI, vol. 10(19), pages 1-10, October.
    3. Yurii Nesterov, 2018. "Lectures on Convex Optimization," Springer Optimization and Its Applications, Springer, edition 2, number 978-3-319-91578-4, June.
    4. Haoze Shi & Naisen Yang & Hong Tang & Xin Yang, 2022. "aSGD: Stochastic Gradient Descent with Adaptive Batch Size for Every Parameter," Mathematics, MDPI, vol. 10(6), pages 1-15, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Shota Takahashi & Mituhiro Fukuda & Mirai Tanaka, 2022. "New Bregman proximal type algorithms for solving DC optimization problems," Computational Optimization and Applications, Springer, vol. 83(3), pages 893-931, December.
    2. A. Scagliotti & P. Colli Franzone, 2022. "A piecewise conservative method for unconstrained convex optimization," Computational Optimization and Applications, Springer, vol. 81(1), pages 251-288, January.
    3. Xin Jiang & Lieven Vandenberghe, 2022. "Bregman primal–dual first-order method and application to sparse semidefinite programming," Computational Optimization and Applications, Springer, vol. 81(1), pages 127-159, January.
    4. Huiyi Cao & Kamil A. Khan, 2023. "General convex relaxations of implicit functions and inverse functions," Journal of Global Optimization, Springer, vol. 86(3), pages 545-572, July.
    5. Francisco García Riesgo & Sergio Luis Suárez Gómez & Enrique Díez Alonso & Carlos González-Gutiérrez & Jesús Daniel Santos, 2021. "Fully Convolutional Approaches for Numerical Approximation of Turbulent Phases in Solar Adaptive Optics," Mathematics, MDPI, vol. 9(14), pages 1-20, July.
    6. Chaudhary, Naveed Ishtiaq & Raja, Muhammad Asif Zahoor & Khan, Zeshan Aslam & Mehmood, Ammara & Shah, Syed Muslim, 2022. "Design of fractional hierarchical gradient descent algorithm for parameter estimation of nonlinear control autoregressive systems," Chaos, Solitons & Fractals, Elsevier, vol. 157(C).
    7. Pavel Shcherbakov & Mingyue Ding & Ming Yuchi, 2021. "Random Sampling Many-Dimensional Sets Arising in Control," Mathematics, MDPI, vol. 9(5), pages 1-16, March.
    8. Liam Madden & Stephen Becker & Emiliano Dall’Anese, 2021. "Bounds for the Tracking Error of First-Order Online Optimization Methods," Journal of Optimization Theory and Applications, Springer, vol. 189(2), pages 437-457, May.
    9. Shariat Torbaghan, Shahab & Madani, Mehdi & Sels, Peter & Virag, Ana & Le Cadre, Hélène & Kessels, Kris & Mou, Yuting, 2021. "Designing day-ahead multi-carrier markets for flexibility: Models and clearing algorithms," Applied Energy, Elsevier, vol. 285(C).
    10. Khan, Zeshan Aslam & Chaudhary, Naveed Ishtiaq & Khan, Taimoor Ali & Farooq, Umair & Pinto, Carla M.A. & Raja, Muhammad Asif Zahoor, 2023. "Enhanced fractional prediction scheme for effective matrix factorization in chaotic feedback recommender systems," Chaos, Solitons & Fractals, Elsevier, vol. 176(C).
    11. Paul R. Rosenbaum, 2023. "Sensitivity analyses informed by tests for bias in observational studies," Biometrics, The International Biometric Society, vol. 79(1), pages 475-487, March.
    12. Xue Gao & Xingju Cai & Deren Han, 2020. "A Gauss–Seidel type inertial proximal alternating linearized minimization for a class of nonconvex optimization problems," Journal of Global Optimization, Springer, vol. 76(4), pages 863-887, April.
    13. Alexander Kononov & Yulia Zakharova, 2022. "Speed scaling scheduling of multiprocessor jobs with energy constraint and makespan criterion," Journal of Global Optimization, Springer, vol. 83(3), pages 539-564, July.
    14. Jean-Jacques Forneron, 2023. "Noisy, Non-Smooth, Non-Convex Estimation of Moment Condition Models," Papers 2301.07196, arXiv.org, revised Feb 2023.
    15. Azimbek Khudoyberdiev & Shabir Ahmad & Israr Ullah & DoHyeun Kim, 2020. "An Optimization Scheme Based on Fuzzy Logic Control for Efficient Energy Consumption in Hydroponics Environment," Energies, MDPI, vol. 13(2), pages 1-27, January.
    16. David Müller & Vladimir Shikhman, 2022. "Network manipulation algorithm based on inexact alternating minimization," Computational Management Science, Springer, vol. 19(4), pages 627-664, October.
    17. Mehdi Karimi & Levent Tunçel, 2020. "Primal–Dual Interior-Point Methods for Domain-Driven Formulations," Mathematics of Operations Research, INFORMS, vol. 45(2), pages 591-621, May.
    18. Fosgerau, Mogens & Melo, Emerson & Shum, Matthew & Sørensen, Jesper R.-V., 2021. "Some remarks on CCP-based estimators of dynamic models," Economics Letters, Elsevier, vol. 204(C).
    19. Chaudhary, Naveed Ishtiaq & Khan, Zeshan Aslam & Kiani, Adiqa Kausar & Raja, Muhammad Asif Zahoor & Chaudhary, Iqra Ishtiaq & Pinto, Carla M.A., 2022. "Design of auxiliary model based normalized fractional gradient algorithm for nonlinear output-error systems," Chaos, Solitons & Fractals, Elsevier, vol. 163(C).
    20. Ramon de Punder & Timo Dimitriadis & Rutger-Jan Lange, 2024. "Kullback-Leibler-based characterizations of score-driven updates," Papers 2408.02391, arXiv.org, revised Sep 2024.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:2:p:316-:d:1028169. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.