IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v11y2023i6p1432-d1098541.html
   My bibliography  Save this article

Performance Analysis of Long Short-Term Memory Predictive Neural Networks on Time Series Data

Author

Listed:
  • Roland Bolboacă

    (The Faculty of Engineering and Information Technology, George Emil Palade University of Medicine, Pharmacy, Science, and Technology of Târgu Mureş, 540139 Târgu Mureş, Romania)

  • Piroska Haller

    (The Faculty of Engineering and Information Technology, George Emil Palade University of Medicine, Pharmacy, Science, and Technology of Târgu Mureş, 540139 Târgu Mureş, Romania)

Abstract

Long short-term memory neural networks have been proposed as a means of creating accurate models from large time series data originating from various fields. These models can further be utilized for prediction, control, or anomaly-detection algorithms. However, finding the optimal hyperparameters to maximize different performance criteria remains a challenge for both novice and experienced users. Hyperparameter optimization algorithms can often be a resource-intensive and time-consuming task, particularly when the impact of the hyperparameters on the performance of the neural network is not comprehended or known. Teacher forcing denotes a procedure that involves feeding the ground truth output from the previous time-step as input to the current time-step during training, while during testing feeding back the predicted values. This paper presents a comprehensive examination of the impact of hyperparameters on long short-term neural networks, with and without teacher forcing, on prediction performance. The study includes testing long short-term memory neural networks, with two variations of teacher forcing, in two prediction modes, using two configurations (i.e., multi-input single-output and multi-input multi-output) on a well-known chemical process simulation dataset. Furthermore, this paper demonstrates the applicability of a long short-term memory neural network with a modified teacher forcing approach in a process state monitoring system. Over 100,000 experiments were conducted with varying hyperparameters and in multiple neural network operation modes, revealing the direct impact of each tested hyperparameter on the training and testing procedures.

Suggested Citation

  • Roland Bolboacă & Piroska Haller, 2023. "Performance Analysis of Long Short-Term Memory Predictive Neural Networks on Time Series Data," Mathematics, MDPI, vol. 11(6), pages 1-35, March.
  • Handle: RePEc:gam:jmathe:v:11:y:2023:i:6:p:1432-:d:1098541
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/11/6/1432/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2227-7390/11/6/1432/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Adrian Sergiu Darabant & Diana Borza & Radu Danescu, 2021. "Recognizing Human Races through Machine Learning—A Multi-Network, Multi-Features Study," Mathematics, MDPI, vol. 9(2), pages 1-19, January.
    2. Jaime Buitrago & Shihab Asfour, 2017. "Short-Term Forecasting of Electric Loads Using Nonlinear Autoregressive Artificial Neural Networks with Exogenous Vector Inputs," Energies, MDPI, vol. 10(1), pages 1-24, January.
    3. Christopher Bennett & Rodney A. Stewart & Junwei Lu, 2014. "Autoregressive with Exogenous Variables and Neural Network Short-Term Load Forecast Models for Residential Low Voltage Distribution Networks," Energies, MDPI, vol. 7(5), pages 1-23, April.
    4. Mohamed Abdel-Basset & Hossam Hawash & Khalid Abdulaziz Alnowibet & Ali Wagdy Mohamed & Karam M. Sallam, 2022. "Interpretable Deep Learning for Discriminating Pneumonia from Lung Ultrasounds," Mathematics, MDPI, vol. 10(21), pages 1-17, November.
    5. João Antunes Rodrigues & José Torres Farinha & Mateus Mendes & Ricardo J. G. Mateus & António J. Marques Cardoso, 2022. "Comparison of Different Features and Neural Networks for Predicting Industrial Paper Press Condition," Energies, MDPI, vol. 15(17), pages 1-16, August.
    6. Xiaojia Wang & Ting Huang & Keyu Zhu & Xibin Zhao, 2022. "LSTM-Based Broad Learning System for Remaining Useful Life Prediction," Mathematics, MDPI, vol. 10(12), pages 1-13, June.
    7. Sanda-Maria Avram & Mihai Oltean, 2022. "A Comparison of Several AI Techniques for Authorship Attribution on Romanian Texts," Mathematics, MDPI, vol. 10(23), pages 1-35, December.
    8. Fabio Henrique Pereira & Francisco Elânio Bezerra & Shigueru Junior & Josemir Santos & Ivan Chabu & Gilberto Francisco Martha de Souza & Fábio Micerino & Silvio Ikuyo Nabeta, 2018. "Nonlinear Autoregressive Neural Network Models for Prediction of Transformer Oil-Dissolved Gas Concentrations," Energies, MDPI, vol. 11(7), pages 1-12, June.
    9. Zina Boussaada & Octavian Curea & Ahmed Remaci & Haritza Camblong & Najiba Mrabet Bellaaj, 2018. "A Nonlinear Autoregressive Exogenous (NARX) Neural Network Model for the Prediction of the Daily Direct Solar Radiation," Energies, MDPI, vol. 11(3), pages 1-21, March.
    10. Amin Ullah & Khalid Mahmood Malik & Abdul Khader Jilani Saudagar & Muhammad Badruddin Khan & Mozaherul Hoque Abul Hasanat & Abdullah AlTameem & Mohammed AlKhathami & Muhammad Sajjad, 2022. "COVID-19 Genome Sequence Analysis for New Variant Prediction and Generation," Mathematics, MDPI, vol. 10(22), pages 1-16, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sen Zheng & Chongshi Gu & Chenfei Shao & Yating Hu & Yanxin Xu & Xiaoyu Huang, 2023. "A Novel Prediction Model for Seawall Deformation Based on CPSO-WNN-LSTM," Mathematics, MDPI, vol. 11(17), pages 1-22, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Timothy Praditia & Thilo Walser & Sergey Oladyshkin & Wolfgang Nowak, 2020. "Improving Thermochemical Energy Storage Dynamics Forecast with Physics-Inspired Neural Network Architecture," Energies, MDPI, vol. 13(15), pages 1-26, July.
    2. Fjelkestam Frederiksen, Cornelia A. & Cai, Zuansi, 2022. "Novel machine learning approach for solar photovoltaic energy output forecast using extra-terrestrial solar irradiance," Applied Energy, Elsevier, vol. 306(PB).
    3. Salma Hamad Almuhaini & Nahid Sultana, 2023. "Forecasting Long-Term Electricity Consumption in Saudi Arabia Based on Statistical and Machine Learning Algorithms to Enhance Electric Power Supply Management," Energies, MDPI, vol. 16(4), pages 1-28, February.
    4. Bin Li & Mingzhen Lu & Yiyi Zhang & Jia Huang, 2019. "A Weekend Load Forecasting Model Based on Semi-Parametric Regression Analysis Considering Weather and Load Interaction," Energies, MDPI, vol. 12(20), pages 1-19, October.
    5. Jee-Heon Kim & Nam-Chul Seong & Wonchang Choi, 2019. "Cooling Load Forecasting via Predictive Optimization of a Nonlinear Autoregressive Exogenous (NARX) Neural Network Model," Sustainability, MDPI, vol. 11(23), pages 1-13, November.
    6. Karodine Chreng & Han Soo Lee & Soklin Tuy, 2022. "A Hybrid Model for Electricity Demand Forecast Using Improved Ensemble Empirical Mode Decomposition and Recurrent Neural Networks with ERA5 Climate Variables," Energies, MDPI, vol. 15(19), pages 1-26, October.
    7. SeyedAli Ghahari & Cesar Queiroz & Samuel Labi & Sue McNeil, 2021. "Cluster Forecasting of Corruption Using Nonlinear Autoregressive Models with Exogenous Variables (NARX)—An Artificial Neural Network Analysis," Sustainability, MDPI, vol. 13(20), pages 1-20, October.
    8. Vladimir J. Alarcon, 2021. "Hindcasting and Forecasting Total Suspended Sediment Concentrations Using a NARX Neural Network," Sustainability, MDPI, vol. 13(1), pages 1-18, January.
    9. Nahid Sultana & S. M. Zakir Hossain & Salma Hamad Almuhaini & Dilek Düştegör, 2022. "Bayesian Optimization Algorithm-Based Statistical and Machine Learning Approaches for Forecasting Short-Term Electricity Demand," Energies, MDPI, vol. 15(9), pages 1-26, May.
    10. Alexandru Pîrjan & Simona-Vasilica Oprea & George Căruțașu & Dana-Mihaela Petroșanu & Adela Bâra & Cristina Coculescu, 2017. "Devising Hourly Forecasting Solutions Regarding Electricity Consumption in the Case of Commercial Center Type Consumers," Energies, MDPI, vol. 10(11), pages 1-36, October.
    11. Anshuman Satapathy & Niranjan Nayak & Tanmoy Parida, 2022. "Real-Time Power Quality Enhancement in a Hybrid Micro-Grid Using Nonlinear Autoregressive Neural Network," Energies, MDPI, vol. 15(23), pages 1-35, November.
    12. Ahmad, Tanveer & Chen, Huanxin, 2018. "Potential of three variant machine-learning models for forecasting district level medium-term and long-term energy demand in smart grid environment," Energy, Elsevier, vol. 160(C), pages 1008-1020.
    13. Alhamwi, Alaa & Medjroubi, Wided & Vogt, Thomas & Agert, Carsten, 2018. "Modelling urban energy requirements using open source data and models," Applied Energy, Elsevier, vol. 231(C), pages 1100-1108.
    14. Liu, Xiufeng & Nielsen, Per Sieverts, 2016. "A hybrid ICT-solution for smart meter data analytics," Energy, Elsevier, vol. 115(P3), pages 1710-1722.
    15. Oksana Mandrikova & Yuryi Polozov & Nataly Zhukova & Yulia Shichkina, 2022. "Approximation and Analysis of Natural Data Based on NARX Neural Networks Involving Wavelet Filtering," Mathematics, MDPI, vol. 10(22), pages 1-16, November.
    16. Kayode Ayankoya & Andre P. Calitz & Jean H. Greyling, 2016. "Real-Time Grain Commodities Price Predictions in South Africa: A Big Data and Neural Networks Approach," Agrekon, Taylor & Francis Journals, vol. 55(4), pages 483-508, October.
    17. Guilherme Henrique Alves & Geraldo Caixeta Guimarães & Fabricio Augusto Matheus Moura, 2023. "Battery Storage Systems Control Strategies with Intelligent Algorithms in Microgrids with Dynamic Pricing," Energies, MDPI, vol. 16(14), pages 1-30, July.
    18. Sufyan Samara & Emad Natsheh, 2020. "Intelligent PV Panels Fault Diagnosis Method Based on NARX Network and Linguistic Fuzzy Rule-Based Systems," Sustainability, MDPI, vol. 12(5), pages 1-20, March.
    19. Jee-Heon Kim & Nam-Chul Seong & Wonchang Choi, 2020. "Forecasting the Energy Consumption of an Actual Air Handling Unit and Absorption Chiller Using ANN Models," Energies, MDPI, vol. 13(17), pages 1-12, August.
    20. Shengyang Lu & Yu Zhu & Lihu Dong & Guangyu Na & Yan Hao & Guanfeng Zhang & Wuyang Zhang & Shanshan Cheng & Junyou Yang & Yuqiu Sui, 2022. "Small-Signal Stability Research of Grid-Connected Virtual Synchronous Generators," Energies, MDPI, vol. 15(19), pages 1-17, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:11:y:2023:i:6:p:1432-:d:1098541. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.