IDEAS home Printed from https://ideas.repec.org/a/eee/reensy/v93y2008i5p711-721.html
   My bibliography  Save this article

A study of expert overconfidence

Author

Listed:
  • Lin, Shi-Woei
  • Bier, Vicki M.

Abstract

Overconfidence is one of the most common (and potentially severe) problems in expert judgment. To assess the extent of expert overconfidence, we analyzed a large data set on expert opinion compiled by Cooke and colleagues at the Technical University of Delft and elsewhere. This data set contains roughly five thousand 90% confidence intervals of uncertain quantities for which the true values are now known. Our analysis assesses the overall extent of overconfidence in the data set. Significant differences in the extent of overconfidence were found among studies, among experts, and among questions within a study. Moreover, replications (multiple realizations for the same question) allowed a preliminary assessment of whether the question effect is due largely to question difficulty, or merely to random noise in the realizations of the uncertain quantities. The results of this analysis suggest that much of the apparent question effect may be due to noise rather than systematic differences in the difficulty of achieving good calibration for different questions. The results support the differential weighting of experts, since there are significant differences in expert calibration within studies.

Suggested Citation

  • Lin, Shi-Woei & Bier, Vicki M., 2008. "A study of expert overconfidence," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 711-721.
    • Handle: RePEc:eee:reensy:v:93:y:2008:i:5:p:711-721
    DOI: 10.1016/j.ress.2007.03.014
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0951832007000981
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ress.2007.03.014?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Klayman, Joshua & Soll, Jack B. & Gonzalez-Vallejo, Claudia & Barlas, Sema, 1999. "Overconfidence: It Depends on How, What, and Whom You Ask, , , , , , , , ," Organizational Behavior and Human Decision Processes, Elsevier, vol. 79(3), pages 216-247, September.
    2. L.H.J. Goossens & R.M. Cooke & F Woudenberg & P Van Der Torn, 1998. "Expert judgement and lethal toxicity of inhaled chemicals," Journal of Risk Research, Taylor & Francis Journals, vol. 1(2), pages 117-133, April.
    3. Shlyakhter, Alexander I. & Kammen, Daniel M. & Broido, Claire L. & Wilson, Richard, 1994. "Quantifying the credibility of energy projections from trends in past data : The US energy sector," Energy Policy, Elsevier, vol. 22(2), pages 119-130, February.
    4. Roger Cooke & Eric Jager, 1998. "A Probabilistic Model for the Failure Frequency of Underground Gas Pipelines," Risk Analysis, John Wiley & Sons, vol. 18(4), pages 511-527, August.
    5. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    6. Alexander I. Shlyakhter, 1994. "An Improved Framework for Uncertainty Analysis: Accounting for Unsuspected Errors," Risk Analysis, John Wiley & Sons, vol. 14(4), pages 441-447, August.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. World Health Organization, Foodborne Epidemiology Reference Group, Source Attribution Task Force, 2016. "Research Synthesis Methods in an Age of Globalized Risks: Lessons from the Global Burden of Foodborne Disease Expert Elicitation," Risk Analysis, John Wiley & Sons, vol. 36(2), pages 191-202, February.
    2. James K. Hammitt & Yifan Zhang, 2013. "Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data," Risk Analysis, John Wiley & Sons, vol. 33(1), pages 109-120, January.
    3. Bolger, Fergus & Wright, George, 2017. "Use of expert knowledge to anticipate the future: Issues, analysis and directions," International Journal of Forecasting, Elsevier, vol. 33(1), pages 230-243.
    4. Kash Barker & Kaycee J. Wilson, 2012. "Decision Trees with Single and Multiple Interval-Valued Objectives," Decision Analysis, INFORMS, vol. 9(4), pages 348-358, December.
    5. David R. Mandel & Daniel Irwin, 2021. "Tracking accuracy of strategic intelligence forecasts: Findings from a long‐term Canadian study," Futures & Foresight Science, John Wiley & Sons, vol. 3(3-4), September.
    6. Compernolle, Tine & Welkenhuysen, Kris & Petitclerc, Estelle & Maes, Dries & Piessens, Kris, 2019. "The impact of policy measures on profitability and risk in geothermal energy investments," Energy Economics, Elsevier, vol. 84(C).
    7. Barker, Kash & Haimes, Yacov Y., 2009. "Assessing uncertainty in extreme events: Applications to risk-based decision making in interdependent infrastructure sectors," Reliability Engineering and System Safety, Elsevier, vol. 94(4), pages 819-829.
    8. Strigini, Lorenzo & Wright, David, 2014. "Bounds on survival probability given mean probability of failure per demand; and the paradoxical advantages of uncertainty," Reliability Engineering and System Safety, Elsevier, vol. 128(C), pages 66-83.
    9. Morais, Caroline & Estrada-Lugo, Hector Diego & Tolo, Silvia & Jacques, Tiago & Moura, Raphael & Beer, Michael & Patelli, Edoardo, 2022. "Robust data-driven human reliability analysis using credal networks," Reliability Engineering and System Safety, Elsevier, vol. 218(PA).
    10. Fergus Bolger & Gene Rowe, 2015. "There is Data, and then there is Data: Only Experimental Evidence will Determine the Utility of Differential Weighting of Expert Judgment," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 21-26, January.
    11. Baker, Erin & Bosetti, Valentina & Anadon, Laura Diaz & Henrion, Max & Aleluia Reis, Lara, 2015. "Future costs of key low-carbon energy technologies: Harmonization and aggregation of energy technology expert elicitation data," Energy Policy, Elsevier, vol. 80(C), pages 219-232.
    12. Eggstaff, Justin W. & Mazzuchi, Thomas A. & Sarkani, Shahram, 2014. "The effect of the number of seed variables on the performance of Cooke′s classical model," Reliability Engineering and System Safety, Elsevier, vol. 121(C), pages 72-82.
    13. Hathout, Michel & Vuillet, Marc & Carvajal, Claudio & Peyras, Laurent & Diab, Youssef, 2019. "Expert judgments calibration and combination for assessment of river levee failure probability," Reliability Engineering and System Safety, Elsevier, vol. 188(C), pages 377-392.
    14. Bonnie C. Wintle & Ann Nicholson, 2014. "Exploring Risk Judgments in a Trade Dispute Using Bayesian Networks," Risk Analysis, John Wiley & Sons, vol. 34(6), pages 1095-1111, June.
    15. Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.
    16. Das Sourish & Yang Hongxia & Banks David, 2012. "Synthetic Priors that Merge Opinion from Multiple Experts," Statistics, Politics and Policy, De Gruyter, vol. 4(1), pages 1-26, December.
    17. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    18. Jeremy Rohmer & Eric Chojnacki, 2021. "Forecast of environment systems using expert judgements: performance comparison between the possibilistic and the classical model," Environment Systems and Decisions, Springer, vol. 41(1), pages 131-146, March.
    19. Cooke, Roger M., 2014. "Deep and Shallow Uncertainty in Messaging Climate Change," RFF Working Paper Series dp-14-11, Resources for the Future.
    20. Morgan, Horatio M. & Sui, Sui & Baum, Matthias, 2018. "Are SMEs with immigrant owners exceptional exporters?," Journal of Business Venturing, Elsevier, vol. 33(3), pages 241-260.
    21. Roger M. Cooke, 2015. "The Aggregation of Expert Judgment: Do Good Things Come to Those Who Weight?," Risk Analysis, John Wiley & Sons, vol. 35(1), pages 12-15, January.
    22. Hanea, D.M. & Jagtman, H.M. & van Alphen, L.L.M.M. & Ale, B.J.M., 2010. "Quantitative and qualitative analysis of the expert and non-expert opinion in fire risk in buildings," Reliability Engineering and System Safety, Elsevier, vol. 95(7), pages 729-741.
    23. Erin Baker & Olaitan Olaleye & Lara Aleluia Reis, 2015. "Decision Frameworks and the Investment in R&D," Working Papers 2015.42, Fondazione Eni Enrico Mattei.
    24. Sulian Wang & Chen Wang, 2021. "Quantile Judgments of Lognormal Losses: An Experimental Investigation," Decision Analysis, INFORMS, vol. 18(1), pages 78-99, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sulian Wang & Chen Wang, 2021. "Quantile Judgments of Lognormal Losses: An Experimental Investigation," Decision Analysis, INFORMS, vol. 18(1), pages 78-99, March.
    2. Robert T. Clemen & Robert L. Winkler, 1999. "Combining Probability Distributions From Experts in Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 19(2), pages 187-203, April.
    3. James K. Hammitt & Alexander I. Shlyakhter, 1999. "The Expected Value of Information and the Probability of Surprise," Risk Analysis, John Wiley & Sons, vol. 19(1), pages 135-152, February.
    4. Thunström, Linda & Nordström, Jonas & Shogren, Jason F., 2015. "Certainty and overconfidence in future preferences for food," Journal of Economic Psychology, Elsevier, vol. 51(C), pages 101-113.
    5. Tuan Pham, Michel & Meyvis, Tom & Zhou, Rongrong, 2001. "Beyond the Obvious: Chronic Vividness of Imagery and the Use of Information in Decision Making," Organizational Behavior and Human Decision Processes, Elsevier, vol. 84(2), pages 226-253, March.
    6. Brenner, Lyle & Griffin, Dale & Koehler, Derek J., 2005. "Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment," Organizational Behavior and Human Decision Processes, Elsevier, vol. 97(1), pages 64-81, May.
    7. Hall, Stephen G. & Mitchell, James, 2007. "Combining density forecasts," International Journal of Forecasting, Elsevier, vol. 23(1), pages 1-13.
    8. Victor R. Fuchs & Alan B. Krueger & James M. Poterba, 1997. "Why do Economists Disagree About Policy?," NBER Working Papers 6151, National Bureau of Economic Research, Inc.
    9. Stephen C. Hora, 2013. "A Note on the Aggregation of Event Probabilities," Risk Analysis, John Wiley & Sons, vol. 33(5), pages 909-914, May.
    10. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    11. Ali Mosleh & George Apostolakis, 1986. "The Assessment of Probability Distributions from Expert Opinions with an Application to Seismic Fragility Curves," Risk Analysis, John Wiley & Sons, vol. 6(4), pages 447-461, December.
    12. Fellner-Röhling, Gerlinde & Krügel, Sebastian, 2014. "Judgmental overconfidence and trading activity," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PB), pages 827-842.
    13. Auffhammer, Maximilian, 2005. "The Rationality of EIA Forecasts under Symmetric and Asymmetric Loss," Department of Agricultural & Resource Economics, UC Berkeley, Working Paper Series qt2ts415ts, Department of Agricultural & Resource Economics, UC Berkeley.
    14. Craig R. Fox & Robert T. Clemen, 2005. "Subjective Probability Assessment in Decision Analysis: Partition Dependence and Bias Toward the Ignorance Prior," Management Science, INFORMS, vol. 51(9), pages 1417-1432, September.
    15. Jiatao Li & Yi Tang, 2013. "The Social Influence of Executive Hubris," Management International Review, Springer, vol. 53(1), pages 83-107, February.
    16. Andersson, Patric, 2005. "Overconfident but yet well-calibrated and underconfident : a research not on judgmental miscalibration and flawed self-assessment," Papers 05-37, Sonderforschungsbreich 504.
    17. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    18. Ine H. J. Van Der Fels‐Klerx & Louis H. J. Goossens & Helmut W. Saatkamp & Suzan H. S. Horst, 2002. "Elicitation of Quantitative Data from a Heterogeneous Expert Panel: Formal Process and Application in Animal Health," Risk Analysis, John Wiley & Sons, vol. 22(1), pages 67-81, February.
    19. Bonaccio, Silvia & Dalal, Reeshad S., 2006. "Advice taking and decision-making: An integrative literature review, and implications for the organizational sciences," Organizational Behavior and Human Decision Processes, Elsevier, vol. 101(2), pages 127-151, November.
    20. Christian Kascha & Francesco Ravazzolo, 2010. "Combining inflation density forecasts," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 29(1-2), pages 231-250.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:reensy:v:93:y:2008:i:5:p:711-721. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: https://www.journals.elsevier.com/reliability-engineering-and-system-safety .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.