IDEAS home Printed from https://ideas.repec.org/p/usi/labsit/030.html
   My bibliography  Save this paper

Expert Elicitation Method Selection Process and Method Comparison

Author

Listed:
  • Angela Dalton
  • Alan Brothers
  • Stephen Walsh
  • Paul Whitney

Abstract

Research on integrative modeling has gained considerable attention in recent years and expert opinion has been increasingly recognized as an important data source and modeling contributor. However, little research has systematically compared and evaluated expert elicitation methods in terms of their ability to link with computational models that capture human behavior and social phenomena. In this paper, we describe a decision-making process we used for evaluating and selecting a task specific elicitation method within the framework of integrative computational social-behavioral modeling. From the existing literature, we identified the characteristics of problems that each candidate method is well suited to address. A small scale expert elicitation was also conducted to evaluate the comparative strength and weaknesses of the methods against a number of consensus-based decision criteria. By developing a set of explicit method evaluation criteria and a description characterizing decision problems for the candidate methods, we seek to gain a better understanding of the feasibility and costeffectiveness of integrating elicitation methods with computational modeling techniques. This serves an important first step toward expanding our research effort and trajectory toward greater interdisciplinary modeling research of human behavior.

Suggested Citation

  • Angela Dalton & Alan Brothers & Stephen Walsh & Paul Whitney, 2010. "Expert Elicitation Method Selection Process and Method Comparison," Labsi Experimental Economics Laboratory University of Siena 030, University of Siena.
  • Handle: RePEc:usi:labsit:030
    as

    Download full text from publisher

    File URL: http://www.labsi.org/wp/labsi30.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Kesten Green & J. Scott Armstrong & Andreas Graefe, 2007. "Methods to Elicit Forecasts from Groups: Delphi and Prediction Markets Compared," Foresight: The International Journal of Applied Forecasting, International Institute of Forecasters, issue 8, pages 17-20, Fall.
    2. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    3. Oded Netzer & Olivier Toubia & Eric Bradlow & Ely Dahan & Theodoros Evgeniou & Fred Feinberg & Eleanor Feit & Sam Hui & Joseph Johnson & John Liechty & James Orlin & Vithala Rao, 2008. "Beyond conjoint analysis: Advances in preference measurement," Marketing Letters, Springer, vol. 19(3), pages 337-354, December.
    4. Dewispelare, Aaron R. & Herren, L. Tandy & Clemen, Robert T., 1995. "The use of probability elicitation in the high-level nuclear waste regulation program," International Journal of Forecasting, Elsevier, vol. 11(1), pages 5-24, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robert Reig & Ramona Schoder, 2010. "Forecasting Accuracy: Comparing Prediction Markets And Surveys – An Experimental Study," Journal of Prediction Markets, University of Buckingham Press, vol. 4(3), pages 1-19.
    2. Kerr, Norbert L. & Tindale, R. Scott, 2011. "Group-based forecasting?: A social psychological analysis," International Journal of Forecasting, Elsevier, vol. 27(1), pages 14-40, January.
    3. Graefe, Andreas & Armstrong, J. Scott, 2011. "Comparing face-to-face meetings, nominal groups, Delphi and prediction markets on an estimation task," International Journal of Forecasting, Elsevier, vol. 27(1), pages 183-195, January.
    4. Goodwin, Paul & Meeran, Sheik & Dyussekeneva, Karima, 2014. "The challenges of pre-launch forecasting of adoption time series for new durable products," International Journal of Forecasting, Elsevier, vol. 30(4), pages 1082-1097.
    5. Kerr, Norbert L. & Tindale, R. Scott, 2011. "Group-based forecasting?: A social psychological analysis," International Journal of Forecasting, Elsevier, vol. 27(1), pages 14-40.
    6. Meirowitz, Adam, 2005. "Deliberative Democracy or Market Democracy: Designing Institutions to Aggregate Preferences and Information," Papers 03-28-2005, Princeton University, Research Program in Political Economy.
    7. Siemroth, Christoph, 2014. "Why prediction markets work : The role of information acquisition and endogenous weighting," Working Papers 14-02, University of Mannheim, Department of Economics.
    8. Coleman, S., 2010. "Russian Election Reform and the Effect of Social Conformity on Voting and the Party System: 2007 and 2008," Journal of the New Economic Association, New Economic Association, issue 5, pages 73-90.
    9. Kitsul, Yuriy & Wright, Jonathan H., 2013. "The economics of options-implied inflation probability density functions," Journal of Financial Economics, Elsevier, vol. 110(3), pages 696-711.
    10. Lawrence Choo & Todd R. Kaplan & Ro’i Zultan, 2019. "Information aggregation in Arrow–Debreu markets: an experiment," Experimental Economics, Springer;Economic Science Association, vol. 22(3), pages 625-652, September.
    11. Wang, Wei & Rothschild, David & Goel, Sharad & Gelman, Andrew, 2015. "Forecasting elections with non-representative polls," International Journal of Forecasting, Elsevier, vol. 31(3), pages 980-991.
    12. Fabio Milani, 2010. "Public option and private profits," Applied Health Economics and Health Policy, Springer, vol. 8(3), pages 155-165, May.
    13. Mikuláš Gangur & Miroslav Plevný, 2014. "Tools for Consumer Rights Protection in the Prediction of Electronic Virtual Market and Technological Changes," The AMFITEATRU ECONOMIC journal, Academy of Economic Studies - Bucharest, Romania, vol. 16(36), pages 578-578, May.
    14. Dai, Min & Jia, Yanwei & Kou, Steven, 2021. "The wisdom of the crowd and prediction markets," Journal of Econometrics, Elsevier, vol. 222(1), pages 561-578.
    15. David Card & Gordon B. Dahl, 2011. "Family Violence and Football: The Effect of Unexpected Emotional Cues on Violent Behavior," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(1), pages 103-143.
    16. Herm, Steffen & Callsen-Bracker, Hans-Markus & Kreis, Henning, 2014. "When the crowd evaluates soccer players’ market values: Accuracy and evaluation attributes of an online community," Sport Management Review, Elsevier, vol. 17(4), pages 484-492.
    17. Zhiyuan Wang & Zhiqiang (Eric) Zheng & Wei Jiang & Shaojie Tang, 2021. "Blockchain‐Enabled Data Sharing in Supply Chains: Model, Operationalization, and Tutorial," Production and Operations Management, Production and Operations Management Society, vol. 30(7), pages 1965-1985, July.
    18. Elberse, Anita & Anand, Bharat, 2007. "The effectiveness of pre-release advertising for motion pictures: An empirical investigation using a simulated market," Information Economics and Policy, Elsevier, vol. 19(3-4), pages 319-343, October.
    19. Stekler, H.O. & Sendor, David & Verlander, Richard, 2010. "Issues in sports forecasting," International Journal of Forecasting, Elsevier, vol. 26(3), pages 606-621, July.
      • Herman O. Stekler & David Sendor & Richard Verlander, 2009. "Issues in Sports Forecasting," Working Papers 2009-002, The George Washington University, Department of Economics, H. O. Stekler Research Program on Forecasting.
    20. Urmee Khan & Robert Lieli, 2010. "Information Processing in Prediction Markets: An Empirical Investigation," Working Papers 201426, University of California at Riverside, Department of Economics.

    More about this item

    Keywords

    expert elicitation method.;

    JEL classification:

    • D90 - Microeconomics - - Micro-Based Behavioral Economics - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:usi:labsit:030. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Alessandro Innocenti (email available below). General contact details of provider: https://edirc.repec.org/data/lasieit.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.