IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0149817.html
   My bibliography  Save this article

Evaluation of a Performance-Based Expert Elicitation: WHO Global Attribution of Foodborne Diseases

Author

Listed:
  • W P Aspinall
  • R M Cooke
  • A H Havelaar
  • S Hoffmann
  • T Hald

Abstract

For many societally important science-based decisions, data are inadequate, unreliable or non-existent, and expert advice is sought. In such cases, procedures for eliciting structured expert judgments (SEJ) are increasingly used. This raises questions regarding validity and reproducibility. This paper presents new findings from a large-scale international SEJ study intended to estimate the global burden of foodborne disease on behalf of WHO. The study involved 72 experts distributed over 134 expert panels, with panels comprising thirteen experts on average. Elicitations were conducted in five languages. Performance-based weighted solutions for target questions of interest were formed for each panel. These weights were based on individual expert’s statistical accuracy and informativeness, determined using between ten and fifteen calibration variables from the experts' field with known values. Equal weights combinations were also calculated. The main conclusions on expert performance are: (1) SEJ does provide a science-based method for attribution of the global burden of foodborne diseases; (2) equal weighting of experts per panel increased statistical accuracy to acceptable levels, but at the cost of informativeness; (3) performance-based weighting increased informativeness, while retaining accuracy; (4) due to study constraints individual experts’ accuracies were generally lower than in other SEJ studies, and (5) there was a negative correlation between experts' informativeness and statistical accuracy which attenuated as accuracy improved, revealing that the least accurate experts drive the negative correlation. It is shown, however, that performance-based weighting has the ability to yield statistically accurate and informative combinations of experts' judgments, thereby offsetting this contrary influence. The present findings suggest that application of SEJ on a large scale is feasible, and motivate the development of enhanced training and tools for remote elicitation of multiple, internationally-dispersed panels.

Suggested Citation

  • W P Aspinall & R M Cooke & A H Havelaar & S Hoffmann & T Hald, 2016. "Evaluation of a Performance-Based Expert Elicitation: WHO Global Attribution of Foodborne Diseases," PLOS ONE, Public Library of Science, vol. 11(3), pages 1-14, March.
    • Handle: RePEc:plo:pone00:0149817
    DOI: 10.1371/journal.pone.0149817
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0149817
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0149817&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0149817?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Roger M. Cooke, 2015. "Messaging climate change uncertainty," Nature Climate Change, Nature, vol. 5(1), pages 8-10, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    2. Colson, Abigail R. & Cooke, Roger M., 2017. "Cross validation for the classical model of structured expert judgment," Reliability Engineering and System Safety, Elsevier, vol. 163(C), pages 109-120.
    3. Abigail R Colson & Roger M Cooke, 2018. "Expert Elicitation: Using the Classical Model to Validate Experts’ Judgments," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 12(1), pages 113-132.
    4. Abigail R Colson & Itamar Megiddo & Gerardo Alvarez-Uria & Sumanth Gandra & Tim Bedford & Alec Morton & Roger M Cooke & Ramanan Laxminarayan, 2019. "Quantifying uncertainty about future antimicrobial resistance: Comparing structured expert judgment and statistical forecasting methods," PLOS ONE, Public Library of Science, vol. 14(7), pages 1-18, July.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Zhang, Xi & Geng, Yong & Shao, Shuai & Wilson, Jeffrey & Song, Xiaoqian & You, Wei, 2020. "China’s non-fossil energy development and its 2030 CO2 reduction targets: The role of urbanization," Applied Energy, Elsevier, vol. 261(C).
    2. Laura Diaz Anadon & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2016. "Expert views - and disagreements - about the potential of energy technology R&D," Climatic Change, Springer, vol. 136(3), pages 677-691, June.
    3. Zhang, Xi & Geng, Yong & Shao, Shuai & Dong, Huijuan & Wu, Rui & Yao, Tianli & Song, Jiekun, 2020. "How to achieve China’s CO2 emission reduction targets by provincial efforts? – An analysis based on generalized Divisia index and dynamic scenario simulation," Renewable and Sustainable Energy Reviews, Elsevier, vol. 127(C).
    4. Colson, Abigail R. & Cooke, Roger M., 2017. "Cross validation for the classical model of structured expert judgment," Reliability Engineering and System Safety, Elsevier, vol. 163(C), pages 109-120.
    5. Simon French & Nikolaos Argyris & Stephanie M. Haywood & Matthew C. Hort & Jim Q. Smith, 2019. "Communicating Geographical Risks in Crisis Management: The Need for Research," Risk Analysis, John Wiley & Sons, vol. 39(1), pages 9-16, January.
    6. JL Bamber & WP Aspinall & RM Cooke, 2016. "A commentary on “how to interpret expert judgment assessments of twenty-first century sea-level rise” by Hylke de Vries and Roderik SW van de Wal," Climatic Change, Springer, vol. 137(3), pages 321-328, August.
    7. Céline Guivarch & Julie Rozenberg & Vanessa Schweizer, 2016. "The diversity of socio-economic pathways and CO2 emissions scenarios: Insights from the investigation of a scenarios database," Post-Print halshs-01292901, HAL.
    8. Thomas David Pol & Jochen Hinkel, 2019. "Uncertainty representations of mean sea-level change: a telephone game?," Climatic Change, Springer, vol. 152(3), pages 393-411, March.
    9. Charlie Wilson & Céline Guivarch & Elmar Kriegler & Bas Ruijven & Detlef P. Vuuren & Volker Krey & Valeria Jana Schwanitz & Erica L. Thompson, 2021. "Evaluating process-based integrated assessment models of climate change mitigation," Climatic Change, Springer, vol. 166(1), pages 1-22, May.
    10. Abigail R Colson & Itamar Megiddo & Gerardo Alvarez-Uria & Sumanth Gandra & Tim Bedford & Alec Morton & Roger M Cooke & Ramanan Laxminarayan, 2019. "Quantifying uncertainty about future antimicrobial resistance: Comparing structured expert judgment and statistical forecasting methods," PLOS ONE, Public Library of Science, vol. 14(7), pages 1-18, July.
    11. Durante Fabrizio & Puccetti Giovanni & Scherer Matthias & Vanduffel Steven, 2017. "The Vine Philosopher: An interview with Roger Cooke," Dependence Modeling, De Gruyter, vol. 5(1), pages 256-267, December.
    12. Fang, Kai & Li, Chenglin & Tang, Yiqi & He, Jianjian & Song, Junnian, 2022. "China’s pathways to peak carbon emissions: New insights from various industrial sectors," Applied Energy, Elsevier, vol. 306(PA).
    13. Frank Stadler & Luke Houghton, 2020. "Breathing life into climate change adaptation," Journal of Industrial Ecology, Yale University, vol. 24(2), pages 400-409, April.
    14. Cooke, Roger M. & Marti, Deniz & Mazzuchi, Thomas, 2021. "Expert forecasting with and without uncertainty quantification and weighting: What do the data say?," International Journal of Forecasting, Elsevier, vol. 37(1), pages 378-387.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0149817. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.