IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v33y2017i1p267-279.html
   My bibliography  Save this article

I nvestigate D iscuss E stimate A ggregate for structured expert judgement

Author

Listed:
  • Hanea, A.M.
  • McBride, M.F.
  • Burgman, M.A.
  • Wintle, B.C.
  • Fidler, F.
  • Flander, L.
  • Twardy, C.R.
  • Manning, B.
  • Mascaro, S.

Abstract

This study presents the results of an approach to the prediction of the outcomes of geopolitical events, which we term the IDEA protocol. The participants investigate the background and causal factors behind a question, predict the outcome, and discuss their thinking with others. They then make a second, private and anonymous judgement of the probability of the event, which is subsequently aggregated mathematically. The method performed well relative to both an equally weighted linear pool and a prediction market, and is relatively simple to implement. The results indicate the value of discussion for removing arbitrary linguistic uncertainty and for sharing and debating knowledge, thereby improving the judgements. Weighting individual judgements based on prior performance using Cooke’s method improved group judgements. Even though some of the results are not statistically significant, the study may not have had sufficient power to detect some important effects. Nevertheless, the results help us to formulate conjectures, which can then be investigated further.

Suggested Citation

  • Hanea, A.M. & McBride, M.F. & Burgman, M.A. & Wintle, B.C. & Fidler, F. & Flander, L. & Twardy, C.R. & Manning, B. & Mascaro, S., 2017. "I nvestigate D iscuss E stimate A ggregate for structured expert judgement," International Journal of Forecasting, Elsevier, vol. 33(1), pages 267-279.
    • Handle: RePEc:eee:intfor:v:33:y:2017:i:1:p:267-279
    DOI: 10.1016/j.ijforecast.2016.02.008
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207016300450
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2016.02.008?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Graefe, Andreas & Armstrong, J. Scott, 2011. "Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task," International Journal of Forecasting, Elsevier, vol. 27(1), pages 183-195, January.
    2. Willy Aspinall, 2010. "A route to more tractable expert advice," Nature, Nature, vol. 463(7279), pages 294-295, January.
    3. Schultze, Thomas & Mojzisch, Andreas & Schulz-Hardt, Stefan, 2012. "Why groups perform better than individuals at quantitative judgment tasks: Group-to-individual transfer as an alternative to differential weighting," Organizational Behavior and Human Decision Processes, Elsevier, vol. 118(1), pages 24-36.
    4. Robin Hanson, 2003. "Combinatorial Information Market Design," Information Systems Frontiers, Springer, vol. 5(1), pages 107-119, January.
    5. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    6. Lyon, Aidan & Wintle, Bonnie C. & Burgman, Mark, 2015. "Collective wisdom: Methods of confidence interval aggregation," Journal of Business Research, Elsevier, vol. 68(8), pages 1759-1767.
    7. Simon French, 2012. "Expert Judgment, Meta-analysis, and Participatory Risk Analysis," Decision Analysis, INFORMS, vol. 9(2), pages 119-127, June.
    8. Weiss, David J & Shanteau, James, 2012. "Decloaking the privileged expert," Journal of Management & Organization, Cambridge University Press, vol. 18(3), pages 300-310, May.
    9. Jonathan Baron & Barbara A. Mellers & Philip E. Tetlock & Eric Stone & Lyle H. Ungar, 2014. "Two Reasons to Make Aggregated Probability Forecasts More Extreme," Decision Analysis, INFORMS, vol. 11(2), pages 133-145, June.
    10. Robin Hanson, 2007. "Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 3-15, February.
    11. Cooke, Roger M. & Goossens, Louis L.H.J., 2008. "TU Delft expert judgment data base," Reliability Engineering and System Safety, Elsevier, vol. 93(5), pages 657-674.
    12. Murphy, Allan H. & Winkler, Robert L., 1992. "Diagnostic verification of probability forecasts," International Journal of Forecasting, Elsevier, vol. 7(4), pages 435-455, March.
    13. Yates, J. Frank & Price, Paul C. & Lee, Ju-Whei & Ramirez, James, 1996. "Good probabilistic forecasters: The 'consumer's' perspective," International Journal of Forecasting, Elsevier, vol. 12(1), pages 41-56, March.
    14. Wei Sun & Robin Hanson & Kathryn Blackmond Laskey & Charles Twardy, 2012. "Probability and Asset Updating using Bayesian Networks for Combinatorial Prediction Markets," Papers 1210.4900, arXiv.org.
    15. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Bolger, Fergus & Wright, George, 2017. "Use of expert knowledge to anticipate the future: Issues, analysis and directions," International Journal of Forecasting, Elsevier, vol. 33(1), pages 230-243.
    2. Ling Cao & Benjamin S. Halpern & Max Troell & Rebecca Short & Cong Zeng & Ziyu Jiang & Yue Liu & Chengxuan Zou & Chunyu Liu & Shurong Liu & Xiangwei Liu & William W. L. Cheung & Richard S. Cottrell & , 2023. "Vulnerability of blue foods to human-induced environmental change," Nature Sustainability, Nature, vol. 6(10), pages 1186-1198, October.
    3. repec:cup:judgdm:v:15:y:2020:i:5:p:783-797 is not listed on IDEAS
    4. Di, Chen & Dimitrov, Stanko & He, Qi-Ming, 2019. "Incentive compatibility in prediction markets: Costly actions and external incentives," International Journal of Forecasting, Elsevier, vol. 35(1), pages 351-370.
    5. Michael C. Runge & Clark S. Rushing & James E. Lyons & Madeleine A. Rubenstein, 2023. "A Simplified Method for Value of Information Using Constructed Scales," Decision Analysis, INFORMS, vol. 20(3), pages 220-230, September.
    6. David R. Mandel & Robert N. Collins & Evan F. Risko & Jonathan A. Fugelsang, 2020. "Effect of confidence interval construction on judgment accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(5), pages 783-797, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mikuláš Gangur & Miroslav Plevný, 2014. "Tools for Consumer Rights Protection in the Prediction of Electronic Virtual Market and Technological Changes," The AMFITEATRU ECONOMIC journal, Academy of Economic Studies - Bucharest, Romania, vol. 16(36), pages 578-578, May.
    2. Galanis, S. & Ioannou, C. & Kotronis, S., 2019. "Information Aggregation Under Ambiguity: Theory and Experimental Evidence," Working Papers 20/05, Department of Economics, City University London.
    3. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.
    4. Anca M. Hanea & Marissa F. McBride & Mark A. Burgman & Bonnie C. Wintle, 2018. "The Value of Performance Weights and Discussion in Aggregated Expert Judgments," Risk Analysis, John Wiley & Sons, vol. 38(9), pages 1781-1794, September.
    5. Snowberg, Erik & Wolfers, Justin & Zitzewitz, Eric, 2013. "Prediction Markets for Economic Forecasting," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 657-687, Elsevier.
    6. Spyros Galanis & Sergei Mikhalishchev, 2024. "Information Aggregation with Costly Information Acquisition," Papers 2406.07186, arXiv.org, revised Nov 2024.
    7. Sung, Ming-Chien & McDonald, David C.J. & Johnson, Johnnie E.V. & Tai, Chung-Ching & Cheah, Eng-Tuck, 2019. "Improving prediction market forecasts by detecting and correcting possible over-reaction to price movements," European Journal of Operational Research, Elsevier, vol. 272(1), pages 389-405.
    8. von der Gracht, Heiko A. & Hommel, Ulrich & Prokesch, Tobias & Wohlenberg, Holger, 2016. "Testing weighting approaches for forecasting in a Group Wisdom Support System environment," Journal of Business Research, Elsevier, vol. 69(10), pages 4081-4094.
    9. Hajime Mizuyama & Seiyu Yamaguchi & Mizuho Sato, 2019. "A Prediction Market-Based Gamified Approach to Enhance Knowledge Sharing in Organizations," Simulation & Gaming, , vol. 50(5), pages 572-597, October.
    10. Lian Jian & Rahul Sami, 2012. "Aggregation and Manipulation in Prediction Markets: Effects of Trading Mechanism and Information Distribution," Management Science, INFORMS, vol. 58(1), pages 123-140, January.
    11. Pavel Atanasov & Phillip Rescober & Eric Stone & Samuel A. Swift & Emile Servan-Schreiber & Philip Tetlock & Lyle Ungar & Barbara Mellers, 2017. "Distilling the Wisdom of Crowds: Prediction Markets vs. Prediction Polls," Management Science, INFORMS, vol. 63(3), pages 691-706, March.
    12. Hanea, Anca & Wilkinson, David Peter & McBride, Marissa & Lyon, Aidan & van Ravenzwaaij, Don & Singleton Thorn, Felix & Gray, Charles T. & Mandel, David R. & Willcox, Aaron & Gould, Elliot, 2021. "Mathematically aggregating experts' predictions of possible futures," MetaArXiv rxmh7, Center for Open Science.
    13. Papakonstantinou, A. & Rogers, A & Gerding, E. H. & Jennings, N. R., 2010. "Mechanism Design for the truthful elicitation of costly probabilistic estimates in Distributed Information Systems," MPRA Paper 43324, University Library of Munich, Germany.
    14. Abraham Othman & Tuomas Sandholm, 2013. "The Gates Hillman prediction market," Review of Economic Design, Springer;Society for Economic Design, vol. 17(2), pages 95-128, June.
    15. Edoardo Gaffeo, 2013. "Using information markets in grantmaking. An assessment of the issues involved and an application to Italian banking foundations," DEM Discussion Papers 2013/08, Department of Economics and Management.
    16. Lambert, Nicolas S. & Langford, John & Wortman Vaughan, Jennifer & Chen, Yiling & Reeves, Daniel M. & Shoham, Yoav & Pennock, David M., 2015. "An axiomatic characterization of wagering mechanisms," Journal of Economic Theory, Elsevier, vol. 156(C), pages 389-416.
    17. Riekhof, Hans-Christian & Riekhof, Marie-Catherine & Brinkhoff, Stefan, 2012. "Predictive Markets: Ein vielversprechender Weg zur Verbesserung der Prognosequalität im Unternehmen?," PFH Forschungspapiere/Research Papers 2012/07, PFH Private University of Applied Sciences, Göttingen.
    18. Galanis Spyros & Kotronis Stelios, 2021. "Updating Awareness and Information Aggregation," The B.E. Journal of Theoretical Economics, De Gruyter, vol. 21(2), pages 613-635, June.
    19. Patrick Buckley & Fergal O’Brien, 0. "The effect of malicious manipulations on prediction market accuracy," Information Systems Frontiers, Springer, vol. 0, pages 1-13.
    20. Julia A. Minson & Jennifer S. Mueller & Richard P. Larrick, 2018. "The Contingent Wisdom of Dyads: When Discussion Enhances vs. Undermines the Accuracy of Collaborative Judgments," Management Science, INFORMS, vol. 64(9), pages 4177-4192, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:33:y:2017:i:1:p:267-279. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.