IDEAS home Printed from https://ideas.repec.org/a/inm/oropre/v68y2020i2p363-380.html
   My bibliography  Save this article

Estimating Uncertainties Using Judgmental Forecasts with Expert Heterogeneity

Author

Listed:
  • Saurabh Bansal

    (Pennsylvania State University, State College, Pennsylvania 16803)

  • Genaro J. Gutierrez

    (University of Texas at Austin, Austin, Texas 78712)

Abstract

In this paper, we develop a new characterization of multiple-point forecasts provided by experts and use it in an optimization framework to deduce actionable signals, including the mean, standard deviation, or a combination of the two for underlying probability distributions. This framework consists of three steps: (1) calibrate experts’ point forecasts using historical data to determine which quantile they provide, on average, when asked for forecasts, (2) quantify the precision in the experts’ forecasts around their average quantile, and (3) use this calibration information in an optimization framework to deduce the signals of interest. We also show that precision and accuracy in expert judgments are complementary in terms of their informativeness. We also discuss implementation of the development and the realized benefits at a large government project in the agribusiness domain.

Suggested Citation

  • Saurabh Bansal & Genaro J. Gutierrez, 2020. "Estimating Uncertainties Using Judgmental Forecasts with Expert Heterogeneity," Operations Research, INFORMS, vol. 68(2), pages 363-380, March.
    • Handle: RePEc:inm:oropre:v:68:y:2020:i:2:p:363-380
    DOI: 10.1287/opre.2019.1938
    as

    Download full text from publisher

    File URL: https://doi.org/10.1287/opre.2019.1938
    Download Restriction: no
    File URL: https://libkey.io/10.1287/opre.2019.1938?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sidney W. Hess, 1993. "Swinging on the Branch of a Tree: Project Selection Applications," Interfaces, INFORMS, vol. 23(6), pages 5-12, December.
    2. Saurabh Bansal & Genaro J. Gutierrez & John R. Keiser, 2016. "Quantifying Uncertainties and Risks Using Managerial Judgments in a Dynamic New Product Development Environment," Production and Operations Management, Production and Operations Management Society, vol. 25(12), pages 2010-2013, December.
    3. H. V. Ravinder & Don N. Kleinmuntz & James S. Dyer, 1988. "The Reliability of Subjective Probabilities Obtained Through Decomposition," Management Science, INFORMS, vol. 34(2), pages 186-199, February.
    4. Christoph H. Loch & Stylianos Kavadias, 2002. "Dynamic Portfolio Selection of NPD Programs Using Marginal Returns," Management Science, INFORMS, vol. 48(10), pages 1227-1241, October.
    5. Saurabh Bansal & Mahesh Nagarajan, 2017. "Product Portfolio Management with Production Flexibility in Agribusiness," Operations Research, INFORMS, vol. 65(4), pages 914-930, August.
    6. Batur, D. & Choobineh, F., 2010. "A quantile-based approach to system selection," European Journal of Operational Research, Elsevier, vol. 202(3), pages 764-772, May.
    7. Vishal Gaur & Saravanan Kesavan & Ananth Raman & Marshall L. Fisher, 2007. "Estimating Demand Uncertainty Using Judgmental Forecasts," Manufacturing & Service Operations Management, INFORMS, vol. 9(4), pages 480-491, April.
    8. Geweke, John & Amisano, Gianni, 2011. "Optimal prediction pools," Journal of Econometrics, Elsevier, vol. 164(1), pages 130-141, September.
    9. Saurabh Bansal & Genaro J. Gutierrez & John R. Keiser, 2017. "Using Experts’ Noisy Quantile Judgments to Quantify Risks: Theory and Application to Agribusiness," Operations Research, INFORMS, vol. 65(5), pages 1115-1130, October.
    10. John D. Sterman, 1989. "Modeling Managerial Behavior: Misperceptions of Feedback in a Dynamic Decision Making Experiment," Management Science, INFORMS, vol. 35(3), pages 321-339, March.
    11. Andrés Weintraub & Carlos Romero, 2006. "Operations Research Models and the Management of Agricultural and Forestry Resources: A Review and Comparison," Interfaces, INFORMS, vol. 36(5), pages 446-457, October.
    12. Donald L. Keefer & Samuel E. Bodily, 1983. "Three-Point Approximations for Continuous Random Variables," Management Science, INFORMS, vol. 29(5), pages 595-609, May.
    13. Donald L. Keefer & William A. Verdini, 1993. "Better Estimation of PERT Activity Time Parameters," Management Science, INFORMS, vol. 39(9), pages 1086-1091, September.
    14. Robert L. Winkler, 1981. "Combining Probability Distributions from Dependent Information Sources," Management Science, INFORMS, vol. 27(4), pages 479-488, April.
    15. Jason R. W. Merrick & J. Rene van Dorp & Amita Singh, 2005. "Analysis of Correlated Expert Judgments from Extended Pairwise Comparisons," Decision Analysis, INFORMS, vol. 2(1), pages 17-29, March.
    16. Thomas S. Wallsten & David V. Budescu, 1983. "State of the Art---Encoding Subjective Probabilities: A Psychological and Psychometric Review," Management Science, INFORMS, vol. 29(2), pages 151-173, February.
    17. James E. Smith & Robert L. Winkler, 2006. "The Optimizer's Curse: Skepticism and Postdecision Surprise in Decision Analysis," Management Science, INFORMS, vol. 52(3), pages 311-322, March.
    18. Janne Kettunen & Ahti Salo, 2017. "Estimation of Downside Risks in Project Portfolio Selection," Production and Operations Management, Production and Operations Management Society, vol. 26(10), pages 1839-1853, October.
    19. Agnew, Julie R., 2006. "Do Behavioral Biases Vary across Individuals? Evidence from Individual Level 401(k) Data," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 41(4), pages 939-962, December.
    20. repec:bla:jfinan:v:59:y:2004:i:5:p:1957-1978 is not listed on IDEAS
    21. Clintin P. Davis-Stober & David V. Budescu & Stephen B. Broomell & Jason Dana, 2015. "The Composition of Optimally Wise Crowds," Decision Analysis, INFORMS, vol. 12(3), pages 130-143.
    22. Sterman, John D., 1989. "Misperceptions of feedback in dynamic decision making," Organizational Behavior and Human Decision Processes, Elsevier, vol. 43(3), pages 301-335, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lance W. Saunders & J. Paul Brooks & Jason R. W. Merrick & Chad W. Autry, 2020. "Addressing Economic/Environmental Sustainability Trade‐offs in Procurement Episodes with Industrial Suppliers," Production and Operations Management, Production and Operations Management Society, vol. 29(5), pages 1256-1269, May.
    2. Jiang, Zhong-Zhong & He, Na & Huang, Song, 2021. "Government penalty provision and contracting with asymmetric quality information in a bioenergy supply chain," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 154(C).
    3. Chidambaram Subbiah & Andrea C. Hupman & Haitao Li & Joseph Simonis, 2023. "Improving Software Development Effort Estimation with a Novel Design Pattern Model," Interfaces, INFORMS, vol. 53(3), pages 192-206, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Anil Gaba & Dana G. Popescu & Zhi Chen, 2019. "Assessing Uncertainty from Point Forecasts," Management Science, INFORMS, vol. 65(1), pages 90-106, January.
    2. Jason R. W. Merrick & Claire A. Dorsey & Bo Wang & Martha Grabowski & John R. Harrald, 2022. "Measuring Prediction Accuracy in a Maritime Accident Warning System," Production and Operations Management, Production and Operations Management Society, vol. 31(2), pages 819-827, February.
    3. Erin Baker & Olaitan Olaleye, 2013. "Combining Experts: Decomposition and Aggregation Order," Risk Analysis, John Wiley & Sons, vol. 33(6), pages 1116-1127, June.
    4. Saurabh Bansal & Genaro J. Gutierrez & Mahesh Nagarajan, 2021. "Theory-Driven Practical Approach to Integrate R&D and Production Planning for Portfolio Management in Agribusiness," Interfaces, INFORMS, vol. 51(5), pages 332-346, September.
    5. James E. Smith & Detlof von Winterfeldt, 2004. "Anniversary Article: Decision Analysis in Management Science," Management Science, INFORMS, vol. 50(5), pages 561-574, May.
    6. Yufei Ren & Rachel Croson, 2013. "Overconfidence in Newsvendor Orders: An Experimental Study," Management Science, INFORMS, vol. 59(11), pages 2502-2517, November.
    7. Lance W. Saunders & J. Paul Brooks & Jason R. W. Merrick & Chad W. Autry, 2020. "Addressing Economic/Environmental Sustainability Trade‐offs in Procurement Episodes with Industrial Suppliers," Production and Operations Management, Production and Operations Management Society, vol. 29(5), pages 1256-1269, May.
    8. Mehrez, A. & Yuan, Y. & Gafni, A., 1995. "The search for information -- A patient perspective on multiple opinions," European Journal of Operational Research, Elsevier, vol. 85(2), pages 244-262, September.
    9. Saurabh Bansal & Genaro J. Gutierrez & John R. Keiser, 2017. "Using Experts’ Noisy Quantile Judgments to Quantify Risks: Theory and Application to Agribusiness," Operations Research, INFORMS, vol. 65(5), pages 1115-1130, October.
    10. Pastore, Erica & Alfieri, Arianna & Zotteri, Giulio, 2019. "An empirical investigation on the antecedents of the bullwhip effect: Evidence from the spare parts industry," International Journal of Production Economics, Elsevier, vol. 209(C), pages 121-133.
    11. Vilkkumaa, Eeva & Liesiö, Juuso & Salo, Ahti, 2014. "Optimal strategies for selecting project portfolios using uncertain value estimates," European Journal of Operational Research, Elsevier, vol. 233(3), pages 772-783.
    12. Berry, D. & Naim, M. M., 1996. "Quantifying the relative improvements of redesign strategies in a P.C. supply chain," International Journal of Production Economics, Elsevier, vol. 46(1), pages 181-196, December.
    13. Towill, Denis R. & Zhou, Li & Disney, Stephen M., 2007. "Reducing the bullwhip effect: Looking through the appropriate lens," International Journal of Production Economics, Elsevier, vol. 108(1-2), pages 444-453, July.
    14. Oliva, Rogelio, 2003. "Model calibration as a testing strategy for system dynamics models," European Journal of Operational Research, Elsevier, vol. 151(3), pages 552-568, December.
    15. Hazhir Rahmandad & Nelson Repenning, 2016. "Capability erosion dynamics," Strategic Management Journal, Wiley Blackwell, vol. 37(4), pages 649-672, April.
    16. Ma, Yungao & Wang, Nengmin & He, Zhengwen & Lu, Jizhou & Liang, Huigang, 2015. "Analysis of the bullwhip effect in two parallel supply chains with interacting price-sensitive demands," European Journal of Operational Research, Elsevier, vol. 243(3), pages 815-825.
    17. Thomas W. Keelin & Bradford W. Powley, 2011. "Quantile-Parameterized Distributions," Decision Analysis, INFORMS, vol. 8(3), pages 206-219, September.
    18. Rich, Karl M. & Ross, R. Brent & Baker, A. Derek & Negassa, Asfaw, 2011. "Quantifying value chain analysis in the context of livestock systems in developing countries," Food Policy, Elsevier, vol. 36(2), pages 214-222, April.
    19. Li Chen & Hau L. Lee, 2012. "Bullwhip Effect Measurement and Its Implications," Operations Research, INFORMS, vol. 60(4), pages 771-784, August.
    20. Anne Sofie Jore & James Mitchell & Shaun P. Vahey, 2010. "Combining forecast densities from VARs with uncertain instabilities," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 25(4), pages 621-634.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:oropre:v:68:y:2020:i:2:p:363-380. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.