IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v14y2023i1d10.1038_s41467-023-42680-x.html
   My bibliography  Save this article

Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty

Author

Listed:
  • Emily Howerton

    (The Pennsylvania State University)

  • Lucie Contamin

    (University of Pittsburgh)

  • Luke C. Mullany

    (Johns Hopkins University Applied Physics Lab)

  • Michelle Qin

    (Harvard University)

  • Nicholas G. Reich

    (University of Massachusetts Amherst)

  • Samantha Bents

    (National Institutes of Health Fogarty International Center)

  • Rebecca K. Borchering

    (The Pennsylvania State University
    Centers for Disease Control and Prevention)

  • Sung-mok Jung

    (University of North Carolina at Chapel Hill)

  • Sara L. Loo

    (Johns Hopkins University)

  • Claire P. Smith

    (Johns Hopkins University)

  • John Levander

    (University of Pittsburgh)

  • Jessica Kerr

    (University of Pittsburgh)

  • J. Espino

    (University of Pittsburgh)

  • Willem G. Panhuis

    (National Institute of Allergy and Infectious Diseases)

  • Harry Hochheiser

    (University of Pittsburgh)

  • Marta Galanti

    (Columbia University)

  • Teresa Yamana

    (Columbia University)

  • Sen Pei

    (Columbia University)

  • Jeffrey Shaman

    (Columbia University)

  • Kaitlin Rainwater-Lovett

    (Johns Hopkins University Applied Physics Lab)

  • Matt Kinsey

    (Johns Hopkins University Applied Physics Lab)

  • Kate Tallaksen

    (Johns Hopkins University Applied Physics Lab)

  • Shelby Wilson

    (Johns Hopkins University Applied Physics Lab)

  • Lauren Shin

    (Johns Hopkins University Applied Physics Lab)

  • Joseph C. Lemaitre

    (University of North Carolina at Chapel Hill)

  • Joshua Kaminsky

    (Johns Hopkins University)

  • Juan Dent Hulse

    (Johns Hopkins University)

  • Elizabeth C. Lee

    (Johns Hopkins University)

  • Clifton D. McKee

    (Johns Hopkins University)

  • Alison Hill

    (Johns Hopkins University)

  • Dean Karlen

    (University of Victoria)

  • Matteo Chinazzi

    (Northeastern University)

  • Jessica T. Davis

    (Northeastern University)

  • Kunpeng Mu

    (Northeastern University)

  • Xinyue Xiong

    (Northeastern University)

  • Ana Pastore y Piontti

    (Northeastern University)

  • Alessandro Vespignani

    (Northeastern University)

  • Erik T. Rosenstrom

    (North Carolina State University)

  • Julie S. Ivy

    (North Carolina State University)

  • Maria E. Mayorga

    (North Carolina State University)

  • Julie L. Swann

    (North Carolina State University)

  • Guido España

    (University of Notre Dame)

  • Sean Cavany

    (University of Notre Dame)

  • Sean Moore

    (University of Notre Dame)

  • Alex Perkins

    (University of Notre Dame)

  • Thomas Hladish

    (University of Florida)

  • Alexander Pillai

    (University of Florida)

  • Kok Toh

    (Northwestern University)

  • Ira Longini

    (University of Florida)

  • Shi Chen

    (University of North Carolina at Charlotte)

  • Rajib Paul

    (University of North Carolina at Charlotte)

  • Daniel Janies

    (University of North Carolina at Charlotte)

  • Jean-Claude Thill

    (University of North Carolina at Charlotte)

  • Anass Bouchnita

    (University of Texas at El Paso)

  • Kaiming Bi

    (University of Texas at Austin)

  • Michael Lachmann

    (Santa Fe Institute)

  • Spencer J. Fox

    (University of Georgia)

  • Lauren Ancel Meyers

    (University of Texas at Austin)

  • Ajitesh Srivastava

    (University of Southern California)

  • Przemyslaw Porebski

    (University of Virginia)

  • Srini Venkatramanan

    (University of Virginia)

  • Aniruddha Adiga

    (University of Virginia)

  • Bryan Lewis

    (University of Virginia)

  • Brian Klahn

    (University of Virginia)

  • Joseph Outten

    (University of Virginia)

  • Benjamin Hurt

    (University of Virginia)

  • Jiangzhuo Chen

    (University of Virginia)

  • Henning Mortveit

    (University of Virginia)

  • Amanda Wilson

    (University of Virginia)

  • Madhav Marathe

    (University of Virginia)

  • Stefan Hoops

    (University of Virginia)

  • Parantapa Bhattacharya

    (University of Virginia)

  • Dustin Machi

    (University of Virginia)

  • Betsy L. Cadwell

    (Centers for Disease Control and Prevention)

  • Jessica M. Healy

    (Centers for Disease Control and Prevention)

  • Rachel B. Slayton

    (Centers for Disease Control and Prevention)

  • Michael A. Johansson

    (Centers for Disease Control and Prevention)

  • Matthew Biggerstaff

    (Centers for Disease Control and Prevention)

  • Shaun Truelove

    (Johns Hopkins University)

  • Michael C. Runge

    (U.S. Geological Survey Eastern Ecological Science Center)

  • Katriona Shea

    (The Pennsylvania State University)

  • Cécile Viboud

    (National Institutes of Health Fogarty International Center)

  • Justin Lessler

    (University of North Carolina at Chapel Hill
    Johns Hopkins University)

Abstract

Our ability to forecast epidemics far into the future is constrained by the many complexities of disease systems. Realistic longer-term projections may, however, be possible under well-defined scenarios that specify the future state of critical epidemic drivers. Since December 2020, the U.S. COVID-19 Scenario Modeling Hub (SMH) has convened multiple modeling teams to make months ahead projections of SARS-CoV-2 burden, totaling nearly 1.8 million national and state-level projections. Here, we find SMH performance varied widely as a function of both scenario validity and model calibration. We show scenarios remained close to reality for 22 weeks on average before the arrival of unanticipated SARS-CoV-2 variants invalidated key assumptions. An ensemble of participating models that preserved variation between models (using the linear opinion pool method) was consistently more reliable than any single model in periods of valid scenario assumptions, while projection interval coverage was near target levels. SMH projections were used to guide pandemic response, illustrating the value of collaborative hubs for longer-term scenario projections.

Suggested Citation

  • Emily Howerton & Lucie Contamin & Luke C. Mullany & Michelle Qin & Nicholas G. Reich & Samantha Bents & Rebecca K. Borchering & Sung-mok Jung & Sara L. Loo & Claire P. Smith & John Levander & Jessica , 2023. "Evaluation of the US COVID-19 Scenario Modeling Hub for informing pandemic response under uncertainty," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    • Handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-42680-x
    DOI: 10.1038/s41467-023-42680-x
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-023-42680-x
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-023-42680-x?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Haoxiang Yang & Özge Sürer & Daniel Duque & David P. Morton & Bismark Singh & Spencer J. Fox & Remy Pasco & Kelly Pierce & Paul Rathouz & Victoria Valencia & Zhanwei Du & Michael Pignone & Mark E. Esc, 2021. "Design of COVID-19 staged alert systems to ensure healthcare capacity with minimal closures," Nature Communications, Nature, vol. 12(1), pages 1-7, December.
    2. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    3. G. Elliott & C. Granger & A. Timmermann (ed.), 2006. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 1, number 1.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sarabeth M. Mathis & Alexander E. Webber & Tomás M. León & Erin L. Murray & Monica Sun & Lauren A. White & Logan C. Brooks & Alden Green & Addison J. Hu & Roni Rosenfeld & Dmitry Shemetov & Ryan J. Ti, 2024. "Evaluation of FluSight influenza forecasting in the 2021–22 and 2022–23 seasons with a new target laboratory-confirmed influenza hospitalizations," Nature Communications, Nature, vol. 15(1), pages 1-13, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kim, Hyun Hak & Swanson, Norman R., 2018. "Mining big data using parsimonious factor, machine learning, variable selection and shrinkage methods," International Journal of Forecasting, Elsevier, vol. 34(2), pages 339-354.
    2. Fernando M. Duarte & Carlo Rosa, 2015. "The equity risk premium: a review of models," Economic Policy Review, Federal Reserve Bank of New York, issue 2, pages 39-57.
    3. Kourentzes, Nikolaos & Petropoulos, Fotios & Trapero, Juan R., 2014. "Improving forecasting by estimating time series structural components across multiple frequencies," International Journal of Forecasting, Elsevier, vol. 30(2), pages 291-302.
    4. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    5. Mauro Costantini & Ulrich Gunter & Robert M. Kunst, 2017. "Forecast Combinations in a DSGE‐VAR Lab," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 36(3), pages 305-324, April.
    6. Fantazzini, Dean, 2020. "Short-term forecasting of the COVID-19 pandemic using Google Trends data: Evidence from 158 countries," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 59, pages 33-54.
    7. Conflitti, Cristina & De Mol, Christine & Giannone, Domenico, 2015. "Optimal combination of survey forecasts," International Journal of Forecasting, Elsevier, vol. 31(4), pages 1096-1103.
    8. Tae-Hwy Lee & Ekaterina Seregina, 2020. "Learning from Forecast Errors: A New Approach to Forecast Combination," Working Papers 202024, University of California at Riverside, Department of Economics.
    9. Di Bu & Simone Kelly & Yin Liao & Qing Zhou, 2018. "A hybrid information approach to predict corporate credit risk," Journal of Futures Markets, John Wiley & Sons, Ltd., vol. 38(9), pages 1062-1078, September.
    10. Samuels, Jon D. & Sekkel, Rodrigo M., 2017. "Model Confidence Sets and forecast combination," International Journal of Forecasting, Elsevier, vol. 33(1), pages 48-60.
    11. Clements, Michael P. & Harvey, David I., 2011. "Combining probability forecasts," International Journal of Forecasting, Elsevier, vol. 27(2), pages 208-223.
    12. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    13. Mehmet Pinar & Thanasis Stengos & M. Ege Yazgan, 2012. "Is there an Optimal Forecast Combination? A Stochastic Dominance Approach to Forecast Combination Puzzle," Working Paper series 17_12, Rimini Centre for Economic Analysis.
    14. Sebastian M. Blanc & Thomas Setzer, 2020. "Bias–Variance Trade-Off and Shrinkage of Weights in Forecast Combination," Management Science, INFORMS, vol. 66(12), pages 5720-5737, December.
    15. Wilson, Kevin J., 2017. "An investigation of dependence in expert judgement studies with multiple experts," International Journal of Forecasting, Elsevier, vol. 33(1), pages 325-336.
    16. Carlo Altavilla & Matteo Ciccarelli, 2006. "Inflation Forecasts, Monetary Policy and Unemployment Dynamics: Evidence from the US and the Euro Area," Discussion Papers 7_2006, D.E.S. (Department of Economic Studies), University of Naples "Parthenope", Italy.
    17. Dean Fantazzini & Julia Pushchelenko & Alexey Mironenkov & Alexey Kurbatskii, 2021. "Forecasting Internal Migration in Russia Using Google Trends: Evidence from Moscow and Saint Petersburg," Forecasting, MDPI, vol. 3(4), pages 1-30, October.
    18. Wagner Piazza Gaglianone & Luiz Renato Lima, 2014. "Constructing Optimal Density Forecasts From Point Forecast Combinations," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(5), pages 736-757, August.
    19. Gang Cheng & Sicong Wang & Yuhong Yang, 2015. "Forecast Combination under Heavy-Tailed Errors," Econometrics, MDPI, vol. 3(4), pages 1-28, November.
    20. KOMINE Takao & BAN Kanemi & KAWAGOE Masaaki & YOSHIDA Hiroshi, 2009. "What Have We Learned from a Survey of Japanese Professional Forecasters? Taking Stock of Four Years of ESP Forecast Experience," ESRI Discussion paper series 214, Economic and Social Research Institute (ESRI).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:14:y:2023:i:1:d:10.1038_s41467-023-42680-x. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.