IDEAS home Printed from https://ideas.repec.org/a/inm/ordeca/v18y2021i1p78-99.html
   My bibliography  Save this article

Quantile Judgments of Lognormal Losses: An Experimental Investigation

Author

Listed:
  • Sulian Wang

    (Department of Industrial Engineering, Tsinghua University, 100084 Beijing, China)

  • Chen Wang

    (Department of Industrial Engineering, Tsinghua University, 100084 Beijing, China)

Abstract

The present study aims to investigate the quality of quantile judgments on a quantity of interest that follows the lognormal distribution, which is skewed and bounded from below with a long right tail. We conduct controlled experiments in which subjects predict the losses from a future typhoon based on losses from past typhoons. Our experiments find underconfidence of the 50% prediction intervals, which is primarily driven by overestimation of the 75th percentiles. We further perform exploratory analyses to disentangle sampling errors and judgmental biases in the overall miscalibration. Finally, we show that the correlations of log-transformed judgments between subjects are smaller than is justified by the information overlapping structure. It leads to overconfident aggregate predictions using the Bayes rule if we treat the low correlations as an indicator for independent information.

Suggested Citation

  • Sulian Wang & Chen Wang, 2021. "Quantile Judgments of Lognormal Losses: An Experimental Investigation," Decision Analysis, INFORMS, vol. 18(1), pages 78-99, March.
    • Handle: RePEc:inm:ordeca:v:18:y:2021:i:1:p:78-99
    DOI: 10.1287/deca.2020.0423
    as

    Download full text from publisher

    File URL: https://doi.org/10.1287/deca.2020.0423
    Download Restriction: no
    File URL: https://libkey.io/10.1287/deca.2020.0423?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Klayman, Joshua & Soll, Jack B. & Gonzalez-Vallejo, Claudia & Barlas, Sema, 1999. "Overconfidence: It Depends on How, What, and Whom You Ask, , , , , , , , ," Organizational Behavior and Human Decision Processes, Elsevier, vol. 79(3), pages 216-247, September.
    2. Kriti Jain & Kanchan Mukherjee & J. Neil Bearden & Anil Gaba, 2013. "Unpacking the Future: A Nudge Toward Wider Subjective Confidence Intervals," Management Science, INFORMS, vol. 59(9), pages 1970-1987, September.
    3. David Tannenbaum & Craig R. Fox & Gülden Ülkümen, 2017. "Judgment Extremity and Accuracy Under Epistemic vs. Aleatory Uncertainty," Management Science, INFORMS, vol. 63(2), pages 497-518, February.
    4. Willy Aspinall, 2010. "A route to more tractable expert advice," Nature, Nature, vol. 463(7279), pages 294-295, January.
    5. Shlyakhter, Alexander I. & Kammen, Daniel M. & Broido, Claire L. & Wilson, Richard, 1994. "Quantifying the credibility of energy projections from trends in past data : The US energy sector," Energy Policy, Elsevier, vol. 22(2), pages 119-130, February.
    6. Saurabh Bansal & Genaro J. Gutierrez & John R. Keiser, 2017. "Using Experts’ Noisy Quantile Judgments to Quantify Risks: Theory and Application to Agribusiness," Operations Research, INFORMS, vol. 65(5), pages 1115-1130, October.
    7. Julius Pahlke & Sebastian Strasser & Ferdinand Vieider, 2015. "Responsibility effects in decision making under risk," Journal of Risk and Uncertainty, Springer, vol. 51(2), pages 125-146, October.
    8. Soll, Jack B., 1996. "Determinants of Overconfidence and Miscalibration: The Roles of Random Error and Ecological Structure," Organizational Behavior and Human Decision Processes, Elsevier, vol. 65(2), pages 117-137, February.
    9. Robert T. Clemen & Robert L. Winkler, 1985. "Limits for the Precision and Value of Information from Dependent Sources," Operations Research, INFORMS, vol. 33(2), pages 427-442, April.
    10. Block, Richard A. & Harper, David R., 1991. "Overconfidence in estimation: Testing the anchoring-and-adjustment hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 49(2), pages 188-207, August.
    11. Juslin, Peter & Winman, Anders & Olsson, Henrik, 2003. "Calibration, additivity, and source independence of probability judgments in general knowledge and sensory discrimination tasks," Organizational Behavior and Human Decision Processes, Elsevier, vol. 92(1-2), pages 34-51.
    12. Stephen C. Hora, 2004. "Probability Judgments for Continuous Quantities: Linear Combinations and Calibration," Management Science, INFORMS, vol. 50(5), pages 597-604, May.
    13. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    14. David Tannenbaum & Craig R. Fox & Gülden Ülkümen, 2017. "Judgment Extremity and Accuracy Under Epistemic vs. Aleatory Uncertainty," Management Science, INFORMS, vol. 63(2), pages 497-518, February.
    15. Asa B. Palley & Jack B. Soll, 2019. "Extracting the Wisdom of Crowds When Information Is Shared," Management Science, INFORMS, vol. 67(5), pages 2291-2309, May.
    16. Victor Richmond R. Jose & Robert L. Winkler, 2009. "Evaluating Quantile Assessments," Operations Research, INFORMS, vol. 57(5), pages 1287-1297, October.
    17. Lin, Shi-Woei & Bier, Vicki M., 2008. "A study of expert overconfidence," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 711-721.
    18. David V. Budescu & Ning Du, 2007. "Coherence and Consistency of Investors' Probability Judgments," Management Science, INFORMS, vol. 53(11), pages 1731-1744, November.
    19. Nicholas Barberis, 2013. "The Psychology of Tail Events: Progress and Challenges," American Economic Review, American Economic Association, vol. 103(3), pages 611-616, May.
    20. Yaakov Kareev & Sharon Arnon & Reut Horwitz-Zeliger, 2002. "On the Misperception of Variability," Discussion Paper Series dp285, The Federmann Center for the Study of Rationality, the Hebrew University, Jerusalem.
    21. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Anil Gaba & Ilia Tsetlin & Robert L. Winkler, 2017. "Combining Interval Forecasts," Decision Analysis, INFORMS, vol. 14(1), pages 1-20, March.
    2. Gilberto Montibeller & Detlof von Winterfeldt, 2015. "Cognitive and Motivational Biases in Decision and Risk Analysis," Risk Analysis, John Wiley & Sons, vol. 35(7), pages 1230-1251, July.
    3. Steffen Keck & Wenjie Tang, 2018. "Gender Composition and Group Confidence Judgment: The Perils of All-Male Groups," Management Science, INFORMS, vol. 64(12), pages 5877-5898, December.
    4. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    5. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    6. Saemi Park & David V. Budescu, 2015. "Aggregating multiple probability intervals to improve calibration," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(2), pages 130-143, March.
    7. Daniel J. Walters & Philip M. Fernbach & Craig R. Fox & Steven A. Sloman, 2017. "Known Unknowns: A Critical Determinant of Confidence and Calibration," Management Science, INFORMS, vol. 63(12), pages 4298-4307, December.
    8. repec:cup:judgdm:v:10:y:2015:i:2:p:130-143 is not listed on IDEAS
    9. repec:cup:judgdm:v:12:y:2017:i:1:p:29-41 is not listed on IDEAS
    10. Julia P. Prims & Don A. Moore, 2017. "Overconfidence over the lifespan," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 12(1), pages 29-41, January.
    11. Ferretti, Valentina & Montibeller, Gilberto & von Winterfeldt, Detlof, 2023. "Testing the effectiveness of debiasing techniques to reduce overprecision in the elicitation of subjective continuous probability distributions," European Journal of Operational Research, Elsevier, vol. 304(2), pages 661-675.
    12. Jordan Tong & Daniel Feiler, 2017. "A Behavioral Model of Forecasting: Naive Statistics on Mental Samples," Management Science, INFORMS, vol. 63(11), pages 3609-3627, November.
    13. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    14. Jaspersen, Johannes G., 2022. "Convex combinations in judgment aggregation," European Journal of Operational Research, Elsevier, vol. 299(2), pages 780-794.
    15. Lin, Shi-Woei & Bier, Vicki M., 2008. "A study of expert overconfidence," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 711-721.
    16. Ying Han & David Budescu, 2019. "A universal method for evaluating the quality of aggregators," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(4), pages 395-411, July.
    17. Eggstaff, Justin W. & Mazzuchi, Thomas A. & Sarkani, Shahram, 2014. "The effect of the number of seed variables on the performance of Cooke′s classical model," Reliability Engineering and System Safety, Elsevier, vol. 121(C), pages 72-82.
    18. James K. Hammitt & Yifan Zhang, 2013. "Combining Experts’ Judgments: Comparison of Algorithmic Methods Using Synthetic Data," Risk Analysis, John Wiley & Sons, vol. 33(1), pages 109-120, January.
    19. Yael Grushka-Cockayne & Victor Richmond R. Jose & Kenneth C. Lichtendahl Jr., 2017. "Ensembles of Overfit and Overconfident Forecasts," Management Science, INFORMS, vol. 63(4), pages 1110-1130, April.
    20. Bonaccorsi, Andrea & Apreda, Riccardo & Fantoni, Gualtiero, 2020. "Expert biases in technology foresight. Why they are a problem and how to mitigate them," Technological Forecasting and Social Change, Elsevier, vol. 151(C).
    21. repec:cup:judgdm:v:15:y:2020:i:5:p:783-797 is not listed on IDEAS
    22. Knut Are Aastveit & James Mitchell & Francesco Ravazzolo & Herman van Dijk, 2018. "The Evolution of Forecast Density Combinations in Economics," Tinbergen Institute Discussion Papers 18-069/III, Tinbergen Institute.
    23. Anil Gaba & Dana G. Popescu & Zhi Chen, 2019. "Assessing Uncertainty from Point Forecasts," Management Science, INFORMS, vol. 65(1), pages 90-106, January.
    24. David R. Mandel & Robert N. Collins & Evan F. Risko & Jonathan A. Fugelsang, 2020. "Effect of confidence interval construction on judgment accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(5), pages 783-797, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ordeca:v:18:y:2021:i:1:p:78-99. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.