IDEAS home Printed from https://ideas.repec.org/a/gam/jforec/v5y2023i3p29-535d1223595.html
   My bibliography  Save this article

Shrinking the Variance in Experts’ “Classical” Weights Used in Expert Judgment Aggregation

Author

Listed:
  • Gayan Dharmarathne

    (Department of Statistics, University of Colombo, Colombo 00300, Sri Lanka)

  • Gabriela F. Nane

    (Delft Institute of Applied Mathematics, Delft University of Technology, 2628 CD Delft, The Netherlands)

  • Andrew Robinson

    (Centre of Excellence for Biosecurity Risk Analysis, School of BioSciences, The University of Melbourne, Melbourne, VIC 3010, Australia)

  • Anca M. Hanea

    (Centre of Excellence for Biosecurity Risk Analysis, School of BioSciences, The University of Melbourne, Melbourne, VIC 3010, Australia)

Abstract

Mathematical aggregation of probabilistic expert judgments often involves weighted linear combinations of experts’ elicited probability distributions of uncertain quantities. Experts’ weights are commonly derived from calibration experiments based on the experts’ performance scores, where performance is evaluated in terms of the calibration and the informativeness of the elicited distributions. This is referred to as Cooke’s method, or the classical model (CM), for aggregating probabilistic expert judgments. The performance scores are derived from experiments, so they are uncertain and, therefore, can be represented by random variables. As a consequence, the experts’ weights are also random variables. We focus on addressing the underlying uncertainty when calculating experts’ weights to be used in a mathematical aggregation of expert elicited distributions. This paper investigates the potential of applying an empirical Bayes development of the James–Stein shrinkage estimation technique on the CM’s weights to derive shrinkage weights with reduced mean squared errors. We analyze 51 professional CM expert elicitation studies. We investigate the differences between the classical and the (new) shrinkage CM weights and the benefits of using the new weights. In theory, the outcome of a probabilistic model using the shrinkage weights should be better than that obtained when using the classical weights because shrinkage estimation techniques reduce the mean squared errors of estimators in general. In particular, the empirical Bayes shrinkage method used here reduces the assigned weights for those experts with larger variances in the corresponding sampling distributions of weights in the experiment. We measure improvement of the aggregated judgments in a cross-validation setting using two studies that can afford such an approach. Contrary to expectations, the results are inconclusive. However, in practice, we can use the proposed shrinkage weights to increase the reliability of derived weights when only small-sized experiments are available. We demonstrate the latter on 49 post-2006 professional CM expert elicitation studies.

Suggested Citation

  • Gayan Dharmarathne & Gabriela F. Nane & Andrew Robinson & Anca M. Hanea, 2023. "Shrinking the Variance in Experts’ “Classical” Weights Used in Expert Judgment Aggregation," Forecasting, MDPI, vol. 5(3), pages 1-14, August.
  • Handle: RePEc:gam:jforec:v:5:y:2023:i:3:p:29-535:d:1223595
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2571-9394/5/3/29/pdf
    Download Restriction: no

    File URL: https://www.mdpi.com/2571-9394/5/3/29/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. John Quigley & Abigail Colson & Willy Aspinall & Roger M. Cooke, 2018. "Elicitation in the Classical Model," International Series in Operations Research & Management Science, in: Luis C. Dias & Alec Morton & John Quigley (ed.), Elicitation, chapter 0, pages 15-36, Springer.
    2. Cooke, Roger M. & Marti, Deniz & Mazzuchi, Thomas, 2021. "Expert forecasting with and without uncertainty quantification and weighting: What do the data say?," International Journal of Forecasting, Elsevier, vol. 37(1), pages 378-387.
    3. Cooke, Roger M. & Goossens, Louis L.H.J., 2008. "TU Delft expert judgment data base," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 657-674.
    4. Bing-Yi Jing & Zhouping Li & Guangming Pan & Wang Zhou, 2016. "On SURE-Type Double Shrinkage Estimation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1696-1704, October.
    5. Deniz Marti & Thomas A. Mazzuchi & Roger M. Cooke, 2021. "Are Performance Weights Beneficial? Investigating the Random Expert Hypothesis," International Series in Operations Research & Management Science, in: Anca M. Hanea & Gabriela F. Nane & Tim Bedford & Simon French (ed.), Expert Judgement in Risk and Decision Analysis, chapter 0, pages 53-82, Springer.
    6. Anca M. Hanea & Gabriela F. Nane, 2021. "An In-Depth Perspective on the Classical Model," International Series in Operations Research & Management Science, in: Anca M. Hanea & Gabriela F. Nane & Tim Bedford & Simon French (ed.), Expert Judgement in Risk and Decision Analysis, chapter 0, pages 225-256, Springer.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ren, Xin & Nane, Gabriela F. & Terwel, Karel C. & van Gelder, Pieter H.A.J.M., 2024. "Measuring the impacts of human and organizational factors on human errors in the Dutch construction industry using structured expert judgement," Reliability Engineering and System Safety, Elsevier, vol. 244(C).
    2. Jeremy Rohmer & Eric Chojnacki, 2021. "Forecast of environment systems using expert judgements: performance comparison between the possibilistic and the classical model," Environment Systems and Decisions, Springer, vol. 41(1), pages 131-146, March.
    3. Elena Verdolini & Laura Díaz Anadón & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2018. "Future Prospects for Energy Technologies: Insights from Expert Elicitations," Review of Environmental Economics and Policy, Association of Environmental and Resource Economists, vol. 12(1), pages 133-153.
    4. Misuri, Alessio & Landucci, Gabriele & Cozzani, Valerio, 2020. "Assessment of safety barrier performance in Natech scenarios," Reliability Engineering and System Safety, Elsevier, vol. 193(C).
    5. Peter Harrison Howard & Derek Sylvan, 2020. "Wisdom of the experts: Using survey responses to address positive and normative uncertainties in climate-economic models," Climatic Change, Springer, vol. 162(2), pages 213-232, September.
    6. Hathout, Michel & Vuillet, Marc & Carvajal, Claudio & Peyras, Laurent & Diab, Youssef, 2019. "Expert judgments calibration and combination for assessment of river levee failure probability," Reliability Engineering and System Safety, Elsevier, vol. 188(C), pages 377-392.
    7. Mohammad Yazdi, 2019. "A review paper to examine the validity of Bayesian network to build rational consensus in subjective probabilistic failure analysis," International Journal of System Assurance Engineering and Management, Springer;The Society for Reliability, Engineering Quality and Operations Management (SREQOM),India, and Division of Operation and Maintenance, Lulea University of Technology, Sweden, vol. 10(1), pages 1-18, February.
    8. Geovanna Hinojoza-Castro & Montserrat Gómez-Delgado & Wenseslao Plata-Rocha, 2022. "Real Estate Developers as Agents in the Simulation of Urban Sprawl," Sustainability, MDPI, vol. 14(15), pages 1-12, July.
    9. Fei Xiong & Yun Liu & Zhenjiang Zhang, 2011. "Dynamics With Co-Evolution Of Individual Inclination And Opinion," International Journal of Modern Physics C (IJMPC), World Scientific Publishing Co. Pte. Ltd., vol. 22(01), pages 51-62.
    10. Karner, Alex & Niemeier, Deb, 2013. "Civil rights guidance and equity analysis methods for regional transportation plans: a critical review of literature and practice," Journal of Transport Geography, Elsevier, vol. 33(C), pages 126-134.
    11. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    12. Hanea, A.M. & McBride, M.F. & Burgman, M.A. & Wintle, B.C. & Fidler, F. & Flander, L. & Twardy, C.R. & Manning, B. & Mascaro, S., 2017. "I nvestigate D iscuss E stimate A ggregate for structured expert judgement," International Journal of Forecasting, Elsevier, vol. 33(1), pages 267-279.
    13. Laura Diaz Anadon & Erin Baker & Valentina Bosetti & Lara Aleluia Reis, 2016. "Expert views - and disagreements - about the potential of energy technology R&D," Climatic Change, Springer, vol. 136(3), pages 677-691, June.
    14. Hanea, D.M. & Jagtman, H.M. & van Alphen, L.L.M.M. & Ale, B.J.M., 2010. "Quantitative and qualitative analysis of the expert and non-expert opinion in fire risk in buildings," Reliability Engineering and System Safety, Elsevier, vol. 95(7), pages 729-741.
    15. Wang, Fan & Li, Heng & Dong, Chao & Ding, Lieyun, 2019. "Knowledge representation using non-parametric Bayesian networks for tunneling risk analysis," Reliability Engineering and System Safety, Elsevier, vol. 191(C).
    16. Martine J. Barons & Lael E. Walsh & Edward E. Salakpi & Linda Nichols, 2024. "A Decision Support System for Sustainable Agriculture and Food Loss Reduction under Uncertain Agricultural Policy Frameworks," Agriculture, MDPI, vol. 14(3), pages 1-21, March.
    17. Colson, Abigail R. & Cooke, Roger M., 2017. "Cross validation for the classical model of structured expert judgment," Reliability Engineering and System Safety, Elsevier, vol. 163(C), pages 109-120.
    18. Nogal, Maria & Morales Nápoles, Oswaldo & O’Connor, Alan, 2019. "Structured expert judgement to understand the intrinsic vulnerability of traffic networks," Transportation Research Part A: Policy and Practice, Elsevier, vol. 127(C), pages 136-152.
    19. Fabio Boschetti & Elizabeth A. Fulton & Nicola J. Grigg, 2014. "Citizens’ Views of Australia’s Future to 2050," Sustainability, MDPI, vol. 7(1), pages 1-26, December.
    20. Schoch-Spana, Monica & Ravi, Sanjana J. & Martin, Elena K., 2022. "Modeling epidemic recovery: An expert elicitation on issues and approaches," Social Science & Medicine, Elsevier, vol. 292(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jforec:v:5:y:2023:i:3:p:29-535:d:1223595. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.