IDEAS home Printed from https://ideas.repec.org/a/gam/jsusta/v13y2021i24p13599-d698410.html
   My bibliography  Save this article

On Comparing Cross-Validated Forecasting Models with a Novel Fuzzy-TOPSIS Metric: A COVID-19 Case Study

Author

Listed:
  • Dalton Garcia Borges de Souza

    (Institute of Science and Technology, Fluminense Federal University, Rio das Ostras 28890-000, Brazil
    Division of Computer Science, Aeronautics Institute of Technology, São José dos Campos 12228-900, Brazil
    Institute of Science and Technology, Federal University of Sao Paulo, São José dos Campos 12247-014, Brazil)

  • Erivelton Antonio dos Santos

    (Institute of Industrial Engineering and Management, Federal University of Itajubá, Itajubá 37500-903, Brazil
    Department of Administration Course, José do Rosário Vellano University, Alfenas 37132-440, Brazil)

  • Francisco Tarcísio Alves Júnior

    (Collegiate of Industrial Engineering, University of Amapa State, Macapá 68900-070, Brazil
    Post-Graduate Program in Intellectual Property and Technology Transfer for Innovation, Federal University of Macapá, Macapá 68903-419, Brazil)

  • Mariá Cristina Vasconcelos Nascimento

    (Division of Computer Science, Aeronautics Institute of Technology, São José dos Campos 12228-900, Brazil
    Institute of Science and Technology, Federal University of Sao Paulo, São José dos Campos 12247-014, Brazil)

Abstract

Time series cross-validation is a technique to select forecasting models. Despite the sophistication of cross-validation over single test/training splits, traditional and independent metrics, such as Mean Absolute Error (MAE) and Root Mean Square Error (RMSE), are commonly used to assess the model’s accuracy. However, what if decision-makers have different models fitting expectations to each moment of a time series? What if the precision of the forecasted values is also important? This is the case of predicting COVID-19 in Amapá, a Brazilian state in the Amazon rainforest. Due to the lack of hospital capacities, a model that promptly and precisely responds to notable ups and downs in the number of cases may be more desired than average models that only have good performances in more frequent and calm circumstances. In line with this, this paper proposes a hybridization of the Technique for Order of Preference by Similarity to Ideal Solution (TOPSIS) and fuzzy sets to create a similarity metric, the closeness coefficient (CC), that enables relative comparisons of forecasting models under heterogeneous fitting expectations and also considers volatility in the predictions. We present a case study using three parametric and three machine learning models commonly used to forecast COVID-19 numbers. The results indicate that the introduced fuzzy similarity metric is a more informative performance assessment metric, especially when using time series cross-validation.

Suggested Citation

  • Dalton Garcia Borges de Souza & Erivelton Antonio dos Santos & Francisco Tarcísio Alves Júnior & Mariá Cristina Vasconcelos Nascimento, 2021. "On Comparing Cross-Validated Forecasting Models with a Novel Fuzzy-TOPSIS Metric: A COVID-19 Case Study," Sustainability, MDPI, vol. 13(24), pages 1-25, December.
  • Handle: RePEc:gam:jsusta:v:13:y:2021:i:24:p:13599-:d:698410
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2071-1050/13/24/13599/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2071-1050/13/24/13599/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Makridakis, Spyros & Hibon, Michele, 2000. "The M3-Competition: results, conclusions and implications," International Journal of Forecasting, Elsevier, vol. 16(4), pages 451-476.
    2. Peter R. Winters, 1960. "Forecasting Sales by Exponentially Weighted Moving Averages," Management Science, INFORMS, vol. 6(3), pages 324-342, April.
    3. Chen, Chen-Tung & Lin, Ching-Torng & Huang, Sue-Fn, 2006. "A fuzzy approach for supplier evaluation and selection in supply chain management," International Journal of Production Economics, Elsevier, vol. 102(2), pages 289-301, August.
    4. James Sears & J. Miguel Villas-Boas & Sofia Berto Villas-Boas & Vasco Villas-Boas, 2023. "Are We #Stayinghome to Flatten the Curve?," American Journal of Health Economics, University of Chicago Press, vol. 9(1), pages 71-95.
    5. Holt, Charles C., 2004. "Author's retrospective on 'Forecasting seasonals and trends by exponentially weighted moving averages'," International Journal of Forecasting, Elsevier, vol. 20(1), pages 11-13.
    6. Donald Douglas Atsa'am & Ruth Wario, 2021. "Classifier Selection for the Prediction of Dominant Transmission Mode of Coronavirus Within Localities: Predicting COVID-19 Transmission Mode," International Journal of E-Health and Medical Communications (IJEHMC), IGI Global, vol. 12(6), pages 1-12, November.
    7. Qiu-Ping Bi & Yu-Cheng Li & Cheng Shen, 2021. "Screening of Evaluation Index and Construction of Evaluation Index System for Mine Ventilation System," Sustainability, MDPI, vol. 13(21), pages 1-15, October.
    8. Marcos A Capistran & Antonio Capella & J Andrés Christen, 2021. "Forecasting hospital demand in metropolitan areas during the current COVID-19 pandemic and estimates of lockdown-induced 2nd waves," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-16, January.
    9. Canova, Fabio & Hansen, Bruce E, 1995. "Are Seasonal Patterns Constant over Time? A Test for Seasonal Stability," Journal of Business & Economic Statistics, American Statistical Association, vol. 13(3), pages 237-252, July.
    10. Gholamreza Dehdasht & M Salim Ferwati & Rosli Mohamad Zin & Nazirah Zainul Abidin, 2020. "A hybrid approach using entropy and TOPSIS to select key drivers for a successful and sustainable lean construction implementation," PLOS ONE, Public Library of Science, vol. 15(2), pages 1-32, February.
    11. Holt, Charles C., 2004. "Forecasting seasonals and trends by exponentially weighted moving averages," International Journal of Forecasting, Elsevier, vol. 20(1), pages 5-10.
    12. Fotios Petropoulos & Spyros Makridakis, 2020. "Forecasting the novel coronavirus COVID-19," PLOS ONE, Public Library of Science, vol. 15(3), pages 1-8, March.
    13. Kim, Sungil & Kim, Heeyoung, 2016. "A new metric of absolute percentage error for intermittent demand forecasts," International Journal of Forecasting, Elsevier, vol. 32(3), pages 669-679.
    14. Marcel Goic & Mirko S Bozanic-Leal & Magdalena Badal & Leonardo J Basso, 2021. "COVID-19: Short-term forecast of ICU beds in times of crisis," PLOS ONE, Public Library of Science, vol. 16(1), pages 1-24, January.
    15. Chakraborty, Tanujit & Ghosh, Indrajit, 2020. "Real-time forecasts and risk assessment of novel coronavirus (COVID-19) cases: A data-driven analysis," Chaos, Solitons & Fractals, Elsevier, vol. 135(C).
    16. Zhang, Xiaolei & Ma, Renjun & Wang, Lin, 2020. "Predicting turning point, duration and attack rate of COVID-19 outbreaks in major Western countries," Chaos, Solitons & Fractals, Elsevier, vol. 135(C).
    17. Chao Chen & Jamie Twycross & Jonathan M Garibaldi, 2017. "A new accuracy measure based on bounded relative error for time series forecasting," PLOS ONE, Public Library of Science, vol. 12(3), pages 1-23, March.
    18. Dalton Garcia Borges de Souza & Erivelton Antonio dos Santos & Nei Yoshihiro Soma & Carlos Eduardo Sanches da Silva, 2021. "MCDM-Based R&D Project Selection: A Systematic Literature Review," Sustainability, MDPI, vol. 13(21), pages 1-34, October.
    19. da Silva, Ramon Gomes & Ribeiro, Matheus Henrique Dal Molin & Mariani, Viviana Cocco & Coelho, Leandro dos Santos, 2020. "Forecasting Brazilian and American COVID-19 cases based on artificial intelligence coupled with climatic exogenous variables," Chaos, Solitons & Fractals, Elsevier, vol. 139(C).
    20. Hillard Kaplan & Benjamin C. Trumble & Jonathan Stieglitz & Roberta Mendez Mamany & Maguin Gutierrez Cayuba & Leonardina Maito Moye & Sarah Alami & Thomas S. Kraft & Raul Quispe Gutierrez & Juan Copaj, 2020. "Voluntary collective isolation as a best response to COVID-19 for indigenous populations? A case study and protocol from the Bolivian Amazon," Post-Print hal-02924018, HAL.
    21. Wang, Peipei & Zheng, Xinqi & Li, Jiayang & Zhu, Bangren, 2020. "Prediction of epidemic trends in COVID-19 with logistic model and machine learning technics," Chaos, Solitons & Fractals, Elsevier, vol. 139(C).
    22. Ribeiro, Matheus Henrique Dal Molin & da Silva, Ramon Gomes & Mariani, Viviana Cocco & Coelho, Leandro dos Santos, 2020. "Short-term forecasting COVID-19 cumulative confirmed cases: Perspectives for Brazil," Chaos, Solitons & Fractals, Elsevier, vol. 135(C).
    23. Makridakis, Spyros & Spiliotis, Evangelos & Assimakopoulos, Vassilios, 2020. "The M4 Competition: 100,000 time series and 61 forecasting methods," International Journal of Forecasting, Elsevier, vol. 36(1), pages 54-74.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tien-Hsiang Chang & Kuei-Ying Hsu & Hsin-Pin Fu & Ying-Hua Teng & Yi-Jhen Li, 2022. "Integrating FSE and AHP to Identify Valuable Customer Needs by Service Quality Analysis," Sustainability, MDPI, vol. 14(3), pages 1-15, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    2. Meira, Erick & Cyrino Oliveira, Fernando Luiz & de Menezes, Lilian M., 2022. "Forecasting natural gas consumption using Bagging and modified regularization techniques," Energy Economics, Elsevier, vol. 106(C).
    3. Hill, Arthur V. & Zhang, Weiyong & Burch, Gerald F., 2015. "Forecasting the forecastability quotient for inventory management," International Journal of Forecasting, Elsevier, vol. 31(3), pages 651-663.
    4. Zhen Zeng & Rachneet Kaur & Suchetha Siddagangappa & Saba Rahimi & Tucker Balch & Manuela Veloso, 2023. "Financial Time Series Forecasting using CNN and Transformer," Papers 2304.04912, arXiv.org.
    5. Jennifer L. Castle & Jurgen A. Doornik & David F. Hendry, 2021. "Forecasting Principles from Experience with Forecasting Competitions," Forecasting, MDPI, vol. 3(1), pages 1-28, February.
    6. Pantelis Agathangelou & Demetris Trihinas & Ioannis Katakis, 2020. "A Multi-Factor Analysis of Forecasting Methods: A Study on the M4 Competition," Data, MDPI, vol. 5(2), pages 1-24, April.
    7. Rice, William L. & Park, So Young & Pan, Bing & Newman, Peter, 2019. "Forecasting campground demand in US national parks," Annals of Tourism Research, Elsevier, vol. 75(C), pages 424-438.
    8. Theodosiou, Marina, 2011. "Forecasting monthly and quarterly time series using STL decomposition," International Journal of Forecasting, Elsevier, vol. 27(4), pages 1178-1195, October.
    9. Steinbuks, Jevgenijs, 2019. "Assessing the accuracy of electricity production forecasts in developing countries," International Journal of Forecasting, Elsevier, vol. 35(3), pages 1175-1185.
    10. Jan G. De Gooijer & Rob J. Hyndman, 2005. "25 Years of IIF Time Series Forecasting: A Selective Review," Monash Econometrics and Business Statistics Working Papers 12/05, Monash University, Department of Econometrics and Business Statistics.
    11. McKenzie, Eddie & Gardner Jr., Everette S., 2010. "Damped trend exponential smoothing: A modelling viewpoint," International Journal of Forecasting, Elsevier, vol. 26(4), pages 661-665, October.
    12. Filelis - Papadopoulos, Christos K. & Kyziropoulos, Panagiotis E. & Morrison, John P. & O‘Reilly, Philip, 2022. "Modelling and forecasting based on recursive incomplete pseudoinverse matrices," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 197(C), pages 358-376.
    13. Gardner, Everette Jr., 2006. "Exponential smoothing: The state of the art--Part II," International Journal of Forecasting, Elsevier, vol. 22(4), pages 637-666.
    14. Posch, Konstantin & Truden, Christian & Hungerländer, Philipp & Pilz, Jürgen, 2022. "A Bayesian approach for predicting food and beverage sales in staff canteens and restaurants," International Journal of Forecasting, Elsevier, vol. 38(1), pages 321-338.
    15. Tayarani N., Mohammad-H., 2021. "Applications of artificial intelligence in battling against covid-19: A literature review," Chaos, Solitons & Fractals, Elsevier, vol. 142(C).
    16. Fieger, Peter & Rice, John, 2016. "Modelling Chinese Inbound Tourism Arrivals into Christchurch," MPRA Paper 75468, University Library of Munich, Germany.
    17. Nghia Chu & Binh Dao & Nga Pham & Huy Nguyen & Hien Tran, 2022. "Predicting Mutual Funds' Performance using Deep Learning and Ensemble Techniques," Papers 2209.09649, arXiv.org, revised Jul 2023.
    18. Albrecht, Tobias & Rausch, Theresa Maria & Derra, Nicholas Daniel, 2021. "Call me maybe: Methods and practical implementation of artificial intelligence in call center arrivals’ forecasting," Journal of Business Research, Elsevier, vol. 123(C), pages 267-278.
    19. Sprangers, Olivier & Schelter, Sebastian & de Rijke, Maarten, 2023. "Parameter-efficient deep probabilistic forecasting," International Journal of Forecasting, Elsevier, vol. 39(1), pages 332-345.
    20. Kosuke Kawakami & Hirokazu Kobayashi & Kazuhide Nakata, 2021. "Seasonal Inventory Management Model for Raw Materials in Steel Industry," Interfaces, INFORMS, vol. 51(4), pages 312-324, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jsusta:v:13:y:2021:i:24:p:13599-:d:698410. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.