IDEAS home Printed from https://ideas.repec.org/a/plo/pcbi00/1007893.html
   My bibliography  Save this article

Calibration of individual-based models to epidemiological data: A systematic review

Author

Listed:
  • C Marijn Hazelbag
  • Jonathan Dushoff
  • Emanuel M Dominic
  • Zinhle E Mthombothi
  • Wim Delva

Abstract

Individual-based models (IBMs) informing public health policy should be calibrated to data and provide estimates of uncertainty. Two main components of model-calibration methods are the parameter-search strategy and the goodness-of-fit (GOF) measure; many options exist for each of these. This review provides an overview of calibration methods used in IBMs modelling infectious disease spread. We identified articles on PubMed employing simulation-based methods to calibrate IBMs informing public health policy in HIV, tuberculosis, and malaria epidemiology published between 1 January 2013 and 31 December 2018. Articles were included if models stored individual-specific information, and calibration involved comparing model output to population-level targets. We extracted information on parameter-search strategies, GOF measures, and model validation. The PubMed search identified 653 candidate articles, of which 84 met the review criteria. Of the included articles, 40 (48%) combined a quantitative GOF measure with an algorithmic parameter-search strategy–either an optimisation algorithm (14/40) or a sampling algorithm (26/40). These 40 articles varied widely in their choices of parameter-search strategies and GOF measures. For the remaining 44 (52%) articles, the parameter-search strategy could either not be identified (32/44) or was described as an informal, non-reproducible method (12/44). Of these 44 articles, the majority (25/44) were unclear about the GOF measure used; of the rest, only five quantitatively evaluated GOF. Only a minority of the included articles, 14 (17%) provided a rationale for their choice of model-calibration method. Model validation was reported in 31 (37%) articles. Reporting on calibration methods is far from optimal in epidemiological modelling studies of HIV, malaria and TB transmission dynamics. The adoption of better documented, algorithmic calibration methods could improve both reproducibility and the quality of inference in model-based epidemiology. There is a need for research comparing the performance of calibration methods to inform decisions about the parameter-search strategies and GOF measures.Author summary: Calibration—that is, “fitting” the model to data—is a crucial part of using mathematical models to better forecast and control the population-level spread of infectious diseases. Evidence that the mathematical model is well-calibrated improves confidence that the model provides a realistic picture of the consequences of health policy decisions. To make informed decisions, Policymakers need information about uncertainty: i.e., what is the range of likely outcomes (rather than just a single prediction). Thus, modellers should also strive to provide accurate measurements of uncertainty, both for their model parameters and for their predictions. This systematic review provides an overview of the methods used to calibrate individual-based models (IBMs) of the spread of HIV, malaria, and tuberculosis. We found that less than half of the reviewed articles used reproducible, non-subjective calibration methods. For the remaining articles, the method could either not be identified or was described as an informal, non-reproducible method. Only one-third of the articles obtained estimates of parameter uncertainty. We conclude that the adoption of better-documented, algorithmic calibration methods could improve both reproducibility and the quality of inference in model-based epidemiology.

Suggested Citation

  • C Marijn Hazelbag & Jonathan Dushoff & Emanuel M Dominic & Zinhle E Mthombothi & Wim Delva, 2020. "Calibration of individual-based models to epidemiological data: A systematic review," PLOS Computational Biology, Public Library of Science, vol. 16(5), pages 1-17, May.
  • Handle: RePEc:plo:pcbi00:1007893
    DOI: 10.1371/journal.pcbi.1007893
    as

    Download full text from publisher

    File URL: https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1007893
    Download Restriction: no

    File URL: https://journals.plos.org/ploscompbiol/article/file?id=10.1371/journal.pcbi.1007893&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pcbi.1007893?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Yuki Furuse, 2019. "Analysis of research intensity on infectious disease by disease burden reveals which infectious diseases are neglected by researchers," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 116(2), pages 478-483, January.
    2. Nadia N Abuelezam & Kathryn Rough & George R Seage III, 2013. "Individual-Based Simulation Models of HIV Transmission: Reporting Quality and Recommendations," PLOS ONE, Public Library of Science, vol. 8(9), pages 1-1, September.
    3. Natasha Stout & Amy Knudsen & Chung Kong & Pamela McMahon & G. Gazelle, 2009. "Calibration Methods Used in Cancer Simulation Models and Suggested Reporting Guidelines," PharmacoEconomics, Springer, vol. 27(7), pages 533-545, July.
    4. Eleanor J. Murray & James M. Robins & George R. Seage III & Sara Lodi & Emily P. Hyle & Krishna P. Reddy & Kenneth A. Freedberg & Miguel A. Hernán, 2018. "Using Observational Data to Calibrate Simulation Models," Medical Decision Making, , vol. 38(2), pages 212-224, February.
    5. Tazio Vanni & Jonathan Karnon & Jason Madan & Richard White & W. Edmunds & Anna Foss & Rosa Legood, 2011. "Calibrating Models in Economic Evaluation," PharmacoEconomics, Springer, vol. 29(1), pages 35-49, January.
    6. Satyajith Amaran & Nikolaos V. Sahinidis & Bikram Sharda & Scott J. Bury, 2016. "Simulation optimization: a review of algorithms and applications," Annals of Operations Research, Springer, vol. 240(1), pages 351-380, May.
    7. Marc C. Kennedy & Anthony O'Hagan, 2001. "Bayesian calibration of computer models," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 63(3), pages 425-464.
    8. Nicolas A. Menzies & Djøra I. Soeteman & Ankur Pandya & Jane J. Kim, 2017. "Bayesian Methods for Calibrating Health Policy Models: A Tutorial," PharmacoEconomics, Springer, vol. 35(6), pages 613-624, June.
    9. David Moher & Alessandro Liberati & Jennifer Tetzlaff & Douglas G Altman & The PRISMA Group, 2009. "Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement," PLOS Medicine, Public Library of Science, vol. 6(7), pages 1-6, July.
    10. James S. Hodges, 1991. "Six (Or So) Things You Can Do with a Bad Model," Operations Research, INFORMS, vol. 39(3), pages 355-365, June.
    11. Leombruni, Roberto & Richiardi, Matteo, 2005. "Why are economists sceptical about agent-based simulations?," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 355(1), pages 103-109.
    12. Douglas Taylor & Vivek Pawar & Denise Kruzikas & Kristen Gilmore & Ankur Pandya & Rowan Iskandar & Milton Weinstein, 2010. "Methods of Model Calibration," PharmacoEconomics, Springer, vol. 28(11), pages 995-1000, November.
    13. Lee, Ju-Sung & Filatova, Tatiana & Ligmann-Zielinska, Arika & Hassani-Mahmooei, Behrooz & Stonedahl, Forrest & Lorscheid, Iris & Voinov, Alexey & Polhill, J. Gareth & Sun, Zhanli & Parker, Dawn C., 2015. "The complexities of agent-based modeling output analysis," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 18(4).
    14. Jonathan Karnon & Tazio Vanni, 2011. "Calibrating Models in Economic Evaluation," PharmacoEconomics, Springer, vol. 29(1), pages 51-62, January.
    15. Andrew H. Briggs & Milton C. Weinstein & Elisabeth A. L. Fenwick & Jonathan Karnon & Mark J. Sculpher & A. David Paltiel, 2012. "Model Parameter Estimation and Uncertainty Analysis," Medical Decision Making, , vol. 32(5), pages 722-732, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Theresa Reiker & Monica Golumbeanu & Andrew Shattock & Lydia Burgert & Thomas A. Smith & Sarah Filippi & Ewan Cameron & Melissa A. Penny, 2021. "Emulator-based Bayesian optimization for efficient multi-objective calibration of an individual-based model of malaria," Nature Communications, Nature, vol. 12(1), pages 1-11, December.
    2. Penny R. Breeze & Hazel Squires & Kate Ennis & Petra Meier & Kate Hayes & Nik Lomax & Alan Shiell & Frank Kee & Frank de Vocht & Martin O’Flaherty & Nigel Gilbert & Robin Purshouse & Stewart Robinson , 2023. "Guidance on the use of complex systems models for economic evaluations of public health interventions," Health Economics, John Wiley & Sons, Ltd., vol. 32(7), pages 1603-1625, July.
    3. David Niyukuri & Trust Chibawara & Peter Suwirakwenda Nyasulu & Wim Delva, 2021. "Inferring HIV Transmission Network Determinants Using Agent-Based Models Calibrated to Multi-Data Sources," Mathematics, MDPI, vol. 9(21), pages 1-33, October.
    4. Vahab Vahdat & Oguzhan Alagoz & Jing Voon Chen & Leila Saoud & Bijan J. Borah & Paul J. Limburg, 2023. "Calibration and Validation of the Colorectal Cancer and Adenoma Incidence and Mortality (CRC-AIM) Microsimulation Model Using Deep Neural Networks," Medical Decision Making, , vol. 43(6), pages 719-736, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jing Voon Chen & Julia L. Higle & Michael Hintlian, 2018. "A systematic approach for examining the impact of calibration uncertainty in disease modeling," Computational Management Science, Springer, vol. 15(3), pages 541-561, October.
    2. Douglas Taylor & Vivek Pawar & Denise Kruzikas & Kristen Gilmore & Myrlene Sanon & Milton Weinstein, 2012. "Incorporating Calibrated Model Parameters into Sensitivity Analyses," PharmacoEconomics, Springer, vol. 30(2), pages 119-126, February.
    3. Penny R. Breeze & Hazel Squires & Kate Ennis & Petra Meier & Kate Hayes & Nik Lomax & Alan Shiell & Frank Kee & Frank de Vocht & Martin O’Flaherty & Nigel Gilbert & Robin Purshouse & Stewart Robinson , 2023. "Guidance on the use of complex systems models for economic evaluations of public health interventions," Health Economics, John Wiley & Sons, Ltd., vol. 32(7), pages 1603-1625, July.
    4. Xiuxian Wang & Na Geng & Jianxin Qiu & Zhibin Jiang & Liping Zhou, 2020. "Markov model and meta-heuristics combined method for cost-effectiveness analysis," Flexible Services and Manufacturing Journal, Springer, vol. 32(1), pages 213-235, March.
    5. Meimei Wang & Steffen Flessa, 2020. "Modelling Covid-19 under uncertainty: what can we expect?," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 21(5), pages 665-668, July.
    6. Jon Duan & G. Cornelis van Kooten & A. T. M. Hasibul Islam, 2023. "Calibration of Grid Models for Analyzing Energy Policies," Energies, MDPI, vol. 16(3), pages 1-21, January.
    7. Ioannis Andrianakis & Ian R Vernon & Nicky McCreesh & Trevelyan J McKinley & Jeremy E Oakley & Rebecca N Nsubuga & Michael Goldstein & Richard G White, 2015. "Bayesian History Matching of Complex Infectious Disease Models Using Emulation: A Tutorial and a Case Study on HIV in Uganda," PLOS Computational Biology, Public Library of Science, vol. 11(1), pages 1-18, January.
    8. Matthew R Behrend & María-Gloria Basáñez & Jonathan I D Hamley & Travis C Porco & Wilma A Stolk & Martin Walker & Sake J de Vlas & for the NTD Modelling Consortium, 2020. "Modelling for policy: The five principles of the Neglected Tropical Diseases Modelling Consortium," PLOS Neglected Tropical Diseases, Public Library of Science, vol. 14(4), pages 1-17, April.
    9. Steffen Flessa & Dominik Dietz & Elisabete Weiderpass, 2016. "Health policy support under extreme uncertainty: the case of cervical cancer in Cambodia," EURO Journal on Decision Processes, Springer;EURO - The Association of European Operational Research Societies, vol. 4(3), pages 183-218, November.
    10. Brown, Vicki & Diomedi, Belen Zapata & Moodie, Marj & Veerman, J. Lennert & Carter, Rob, 2016. "A systematic review of economic analyses of active transport interventions that include physical activity benefits," Transport Policy, Elsevier, vol. 45(C), pages 190-208.
    11. Utomo, Dhanan Sarwo & Onggo, Bhakti Stephan & Eldridge, Stephen, 2018. "Applications of agent-based modelling and simulation in the agri-food supply chains," European Journal of Operational Research, Elsevier, vol. 269(3), pages 794-805.
    12. Candio, Paolo & Meads, David & Hill, Andrew J. & Bojke, Laura, 2020. "Modelling the impact of physical activity on public health: A review and critique," Health Policy, Elsevier, vol. 124(10), pages 1155-1164.
    13. Ghaderi, Mohammad, 2022. "Public health interventions in the face of pandemics: Network structure, social distancing, and heterogeneity," European Journal of Operational Research, Elsevier, vol. 298(3), pages 1016-1031.
    14. Matthew Oldham, 2019. "Understanding How Short-Termism and a Dynamic Investor Network Affects Investor Returns: An Agent-Based Perspective," Complexity, Hindawi, vol. 2019, pages 1-21, July.
    15. Mohamed Hussein & Abdelrahman E. E. Eltoukhy & Amos Darko & Amr Eltawil, 2021. "Simulation-Optimization for the Planning of Off-Site Construction Projects: A Comparative Study of Recent Swarm Intelligence Metaheuristics," Sustainability, MDPI, vol. 13(24), pages 1-41, December.
    16. Tazio Vanni & Jonathan Karnon & Jason Madan & Richard White & W. Edmunds & Anna Foss & Rosa Legood, 2011. "Calibrating Models in Economic Evaluation," PharmacoEconomics, Springer, vol. 29(1), pages 35-49, January.
    17. İlkay Unay-Gailhard & Mark A. Brennen, 2022. "How digital communications contribute to shaping the career paths of youth: a review study focused on farming as a career option," Agriculture and Human Values, Springer;The Agriculture, Food, & Human Values Society (AFHVS), vol. 39(4), pages 1491-1508, December.
    18. Vanslette, Kevin & Tohme, Tony & Youcef-Toumi, Kamal, 2020. "A general model validation and testing tool," Reliability Engineering and System Safety, Elsevier, vol. 195(C).
    19. Mahin Ghafari & Vali Baigi & Zahra Cheraghi & Amin Doosti-Irani, 2016. "The Prevalence of Asymptomatic Bacteriuria in Iranian Pregnant Women: A Systematic Review and Meta-Analysis," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-10, June.
    20. Elizabeth T Cafiero-Fonseca & Andrew Stawasz & Sydney T Johnson & Reiko Sato & David E Bloom, 2017. "The full benefits of adult pneumococcal vaccination: A systematic review," PLOS ONE, Public Library of Science, vol. 12(10), pages 1-23, October.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pcbi00:1007893. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ploscompbiol (email available below). General contact details of provider: https://journals.plos.org/ploscompbiol/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.