IDEAS home Printed from https://ideas.repec.org/a/gam/jeners/v13y2020i18p4722-d411730.html
   My bibliography  Save this article

Time Series Forecasting with Multi-Headed Attention-Based Deep Learning for Residential Energy Consumption

Author

Listed:
  • Seok-Jun Bu

    (Department of Computer Science, Graduate School of Artificial Intelligence, Yonsei University, Seoul 03722, Korea)

  • Sung-Bae Cho

    (Department of Computer Science, Graduate School of Artificial Intelligence, Yonsei University, Seoul 03722, Korea)

Abstract

Predicting residential energy consumption is tantamount to forecasting a multivariate time series. A specific window for several sensor signals can induce various features extracted to forecast the energy consumption by using a prediction model. However, it is still a challenging task because of irregular patterns inside including hidden correlations between power attributes. In order to extract the complicated irregular energy patterns and selectively learn the spatiotemporal features to reduce the translational variance between energy attributes, we propose a deep learning model based on the multi-headed attention with the convolutional recurrent neural network. It exploits the attention scores calculated with softmax and dot product operation in the network to model the transient and impulsive nature of energy demand. Experiments with the dataset of University of California, Irvine (UCI) household electric power consumption consisting of a total 2,075,259 time-series show that the proposed model reduces the prediction error by 31.01% compared to the state-of-the-art deep learning model. Especially, the multi-headed attention improves the prediction performance even more by up to 27.91% than the single-attention.

Suggested Citation

  • Seok-Jun Bu & Sung-Bae Cho, 2020. "Time Series Forecasting with Multi-Headed Attention-Based Deep Learning for Residential Energy Consumption," Energies, MDPI, vol. 13(18), pages 1-16, September.
    • Handle: RePEc:gam:jeners:v:13:y:2020:i:18:p:4722-:d:411730
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/1996-1073/13/18/4722/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/1996-1073/13/18/4722/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Tso, Geoffrey K.F. & Yau, Kelvin K.W., 2007. "Predicting electricity energy consumption: A comparison of regression analysis, decision tree and neural networks," Energy, Elsevier, vol. 32(9), pages 1761-1768.
    2. Arghira, Nicoleta & Hawarah, Lamis & Ploix, Stéphane & Jacomino, Mireille, 2012. "Prediction of appliances energy use in smart homes," Energy, Elsevier, vol. 48(1), pages 128-134.
    3. Fan, Cheng & Sun, Yongjun & Zhao, Yang & Song, Mengjie & Wang, Jiayuan, 2019. "Deep learning-based feature engineering methods for improved building energy prediction," Applied Energy, Elsevier, vol. 240(C), pages 35-45.
    4. Wang, Yi & Gan, Dahua & Sun, Mingyang & Zhang, Ning & Lu, Zongxiang & Kang, Chongqing, 2019. "Probabilistic individual load forecasting using pinball loss guided LSTM," Applied Energy, Elsevier, vol. 235(C), pages 10-20.
    5. Kim, Tae-Young & Cho, Sung-Bae, 2019. "Predicting residential energy consumption using CNN-LSTM neural networks," Energy, Elsevier, vol. 182(C), pages 72-81.
    6. Rahman, Aowabin & Srikumar, Vivek & Smith, Amanda D., 2018. "Predicting electricity consumption for commercial and residential buildings using deep recurrent neural networks," Applied Energy, Elsevier, vol. 212(C), pages 372-385.
    7. Weide Li & Xuan Yang & Hao Li & Lili Su, 2017. "Hybrid Forecasting Approach Based on GRNN Neural Network and SVR Machine for Electricity Demand Forecasting," Energies, MDPI, vol. 10(1), pages 1-17, January.
    8. Ekonomou, L., 2010. "Greek long-term energy consumption prediction using artificial neural networks," Energy, Elsevier, vol. 35(2), pages 512-517.
    9. Guo, Zhifeng & Zhou, Kaile & Zhang, Xiaoling & Yang, Shanlin, 2018. "A deep learning model for short-term power load and probability density forecasting," Energy, Elsevier, vol. 160(C), pages 1186-1200.
    10. Lago, Jesus & De Ridder, Fjo & De Schutter, Bart, 2018. "Forecasting spot electricity prices: Deep learning approaches and empirical comparison of traditional algorithms," Applied Energy, Elsevier, vol. 221(C), pages 386-405.
    11. Wang, Kejun & Qi, Xiaoxia & Liu, Hongda, 2019. "Photovoltaic power forecasting based LSTM-Convolutional Network," Energy, Elsevier, vol. 189(C).
    12. Qing, Xiangyun & Niu, Yugang, 2018. "Hourly day-ahead solar irradiance prediction using weather forecasts by LSTM," Energy, Elsevier, vol. 148(C), pages 461-468.
    13. Deb, Chirag & Zhang, Fan & Yang, Junjing & Lee, Siew Eang & Shah, Kwok Wei, 2017. "A review on time series forecasting techniques for building energy consumption," Renewable and Sustainable Energy Reviews, Elsevier, vol. 74(C), pages 902-924.
    14. Chengdong Li & Zixiang Ding & Dongbin Zhao & Jianqiang Yi & Guiqing Zhang, 2017. "Building Energy Consumption Prediction: An Extreme Deep Learning Approach," Energies, MDPI, vol. 10(10), pages 1-20, October.
    15. Jin-Young Kim & Sung-Bae Cho, 2019. "Electric Energy Consumption Prediction by Deep Learning with State Explainable Autoencoder," Energies, MDPI, vol. 12(4), pages 1-14, February.
    16. Salah Bouktif & Ali Fiaz & Ali Ouni & Mohamed Adel Serhani, 2018. "Optimal Deep Learning LSTM Model for Electric Load Forecasting using Feature Selection and Genetic Algorithm: Comparison with Machine Learning Approaches †," Energies, MDPI, vol. 11(7), pages 1-20, June.
    17. Alessandro Burgio & Daniele Menniti & Nicola Sorrentino & Anna Pinnarelli & Zbigniew Leonowicz, 2020. "Influence and Impact of Data Averaging and Temporal Resolution on the Assessment of Energetic, Economic and Technical Issues of Hybrid Photovoltaic-Battery Systems," Energies, MDPI, vol. 13(2), pages 1-26, January.
    18. Spyros Makridakis & Evangelos Spiliotis & Vassilios Assimakopoulos, 2018. "Statistical and Machine Learning forecasting methods: Concerns and ways forward," PLOS ONE, Public Library of Science, vol. 13(3), pages 1-26, March.
    19. Ismail Shah & Hasnain Iftikhar & Sajid Ali & Depeng Wang, 2019. "Short-Term Electricity Demand Forecasting Using Components Estimation Technique," Energies, MDPI, vol. 12(13), pages 1-17, July.
    20. Nejat, Payam & Jomehzadeh, Fatemeh & Taheri, Mohammad Mahdi & Gohari, Mohammad & Abd. Majid, Muhd Zaimi, 2015. "A global review of energy consumption, CO2 emissions and policy in the residential sector (with an overview of the top ten CO2 emitting countries)," Renewable and Sustainable Energy Reviews, Elsevier, vol. 43(C), pages 843-862.
    21. Zhao, G.Y. & Liu, Z.Y. & He, Y. & Cao, H.J. & Guo, Y.B., 2017. "Energy consumption in machining: Classification, prediction, and reduction strategy," Energy, Elsevier, vol. 133(C), pages 142-157.
    22. Fan, Cheng & Wang, Jiayuan & Gang, Wenjie & Li, Shenghan, 2019. "Assessment of deep recurrent neural network-based strategies for short-term building energy predictions," Applied Energy, Elsevier, vol. 236(C), pages 700-710.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. M. C. Pegalajar & L. G. B. Ruiz, 2022. "Time Series Forecasting for Energy Consumption," Energies, MDPI, vol. 15(3), pages 1-3, January.
    2. Hanif, M.F. & Mi, J., 2024. "Harnessing AI for solar energy: Emergence of transformer models," Applied Energy, Elsevier, vol. 369(C).
    3. Jinyuan Liu & Shouxi Wang & Nan Wei & Yi Yang & Yihao Lv & Xu Wang & Fanhua Zeng, 2023. "An Enhancement Method Based on Long Short-Term Memory Neural Network for Short-Term Natural Gas Consumption Forecasting," Energies, MDPI, vol. 16(3), pages 1-14, January.
    4. Joanna Henzel & Łukasz Wróbel & Marcin Fice & Marek Sikora, 2022. "Energy Consumption Forecasting for the Digital-Twin Model of the Building," Energies, MDPI, vol. 15(12), pages 1-21, June.
    5. Khan, Zulfiqar Ahmad & Hussain, Tanveer & Baik, Sung Wook, 2023. "Dual stream network with attention mechanism for photovoltaic power forecasting," Applied Energy, Elsevier, vol. 338(C).
    6. Jicheng Liu & Yu Yin, 2022. "Power Load Forecasting Considering Climate Factors Based on IPSO-Elman Method in China," Energies, MDPI, vol. 15(3), pages 1-23, February.
    7. Tingting Hou & Rengcun Fang & Jinrui Tang & Ganheng Ge & Dongjun Yang & Jianchao Liu & Wei Zhang, 2021. "A Novel Short-Term Residential Electric Load Forecasting Method Based on Adaptive Load Aggregation and Deep Learning Algorithms," Energies, MDPI, vol. 14(22), pages 1-21, November.
    8. Nahid Sultana & S. M. Zakir Hossain & Salma Hamad Almuhaini & Dilek Düştegör, 2022. "Bayesian Optimization Algorithm-Based Statistical and Machine Learning Approaches for Forecasting Short-Term Electricity Demand," Energies, MDPI, vol. 15(9), pages 1-26, May.
    9. Manuel R. Arahal & Manuel G. Ortega & Manuel G. Satué, 2021. "Chiller Load Forecasting Using Hyper-Gaussian Nets," Energies, MDPI, vol. 14(12), pages 1-15, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jason Runge & Radu Zmeureanu, 2021. "A Review of Deep Learning Techniques for Forecasting Energy Use in Buildings," Energies, MDPI, vol. 14(3), pages 1-26, January.
    2. Gonçalves, Rui & Ribeiro, Vitor Miguel & Pereira, Fernando Lobo, 2023. "Variable Split Convolutional Attention: A novel Deep Learning model applied to the household electric power consumption," Energy, Elsevier, vol. 274(C).
    3. Salam, Abdulwahed & El Hibaoui, Abdelaaziz, 2021. "Energy consumption prediction model with deep inception residual network inspiration and LSTM," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 190(C), pages 97-109.
    4. Chaturvedi, Shobhit & Rajasekar, Elangovan & Natarajan, Sukumar & McCullen, Nick, 2022. "A comparative assessment of SARIMA, LSTM RNN and Fb Prophet models to forecast total and peak monthly energy demand for India," Energy Policy, Elsevier, vol. 168(C).
    5. Saima Akhtar & Sulman Shahzad & Asad Zaheer & Hafiz Sami Ullah & Heybet Kilic & Radomir Gono & Michał Jasiński & Zbigniew Leonowicz, 2023. "Short-Term Load Forecasting Models: A Review of Challenges, Progress, and the Road Ahead," Energies, MDPI, vol. 16(10), pages 1-29, May.
    6. Hwang, Jun Kwon & Yun, Geun Young & Lee, Sukho & Seo, Hyeongjoon & Santamouris, Mat, 2020. "Using deep learning approaches with variable selection process to predict the energy performance of a heating and cooling system," Renewable Energy, Elsevier, vol. 149(C), pages 1227-1245.
    7. Li, Guannan & Li, Fan & Ahmad, Tanveer & Liu, Jiangyan & Li, Tao & Fang, Xi & Wu, Yubei, 2022. "Performance evaluation of sequence-to-sequence-Attention model for short-term multi-step ahead building energy predictions," Energy, Elsevier, vol. 259(C).
    8. Zang, Haixiang & Xu, Ruiqi & Cheng, Lilin & Ding, Tao & Liu, Ling & Wei, Zhinong & Sun, Guoqiang, 2021. "Residential load forecasting based on LSTM fusing self-attention mechanism with pooling," Energy, Elsevier, vol. 229(C).
    9. Ding, Zhikun & Chen, Weilin & Hu, Ting & Xu, Xiaoxiao, 2021. "Evolutionary double attention-based long short-term memory model for building energy prediction: Case study of a green building," Applied Energy, Elsevier, vol. 288(C).
    10. Prince Waqas Khan & Yongjun Kim & Yung-Cheol Byun & Sang-Joon Lee, 2021. "Influencing Factors Evaluation of Machine Learning-Based Energy Consumption Prediction," Energies, MDPI, vol. 14(21), pages 1-22, November.
    11. Bedi, Jatin & Toshniwal, Durga, 2019. "Deep learning framework to forecast electricity demand," Applied Energy, Elsevier, vol. 238(C), pages 1312-1326.
    12. Barja-Martinez, Sara & Aragüés-Peñalba, Mònica & Munné-Collado, Íngrid & Lloret-Gallego, Pau & Bullich-Massagué, Eduard & Villafafila-Robles, Roberto, 2021. "Artificial intelligence techniques for enabling Big Data services in distribution networks: A review," Renewable and Sustainable Energy Reviews, Elsevier, vol. 150(C).
    13. Lu, Yakai & Tian, Zhe & Zhou, Ruoyu & Liu, Wenjing, 2021. "A general transfer learning-based framework for thermal load prediction in regional energy system," Energy, Elsevier, vol. 217(C).
    14. Hyunsoo Kim & Jiseok Jeong & Changwan Kim, 2022. "Daily Peak-Electricity-Demand Forecasting Based on Residual Long Short-Term Network," Mathematics, MDPI, vol. 10(23), pages 1-17, November.
    15. Ajith, Meenu & Martínez-Ramón, Manel, 2023. "Deep learning algorithms for very short term solar irradiance forecasting: A survey," Renewable and Sustainable Energy Reviews, Elsevier, vol. 182(C).
    16. Khan, Zulfiqar Ahmad & Khan, Shabbir Ahmad & Hussain, Tanveer & Baik, Sung Wook, 2024. "DSPM: Dual sequence prediction model for efficient energy management in micro-grid," Applied Energy, Elsevier, vol. 356(C).
    17. Lu, Shixiang & Xu, Qifa & Jiang, Cuixia & Liu, Yezheng & Kusiak, Andrew, 2022. "Probabilistic load forecasting with a non-crossing sparse-group Lasso-quantile regression deep neural network," Energy, Elsevier, vol. 242(C).
    18. Li, Ao & Xiao, Fu & Zhang, Chong & Fan, Cheng, 2021. "Attention-based interpretable neural network for building cooling load prediction," Applied Energy, Elsevier, vol. 299(C).
    19. Hao Zhen & Dongxiao Niu & Min Yu & Keke Wang & Yi Liang & Xiaomin Xu, 2020. "A Hybrid Deep Learning Model and Comparison for Wind Power Forecasting Considering Temporal-Spatial Feature Extraction," Sustainability, MDPI, vol. 12(22), pages 1-24, November.
    20. Andreea Valeria Vesa & Tudor Cioara & Ionut Anghel & Marcel Antal & Claudia Pop & Bogdan Iancu & Ioan Salomie & Vasile Teodor Dadarlat, 2020. "Energy Flexibility Prediction for Data Center Engagement in Demand Response Programs," Sustainability, MDPI, vol. 12(4), pages 1-23, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jeners:v:13:y:2020:i:18:p:4722-:d:411730. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.