IDEAS home Printed from https://ideas.repec.org/a/eee/jeborg/v189y2021icp234-256.html
   My bibliography  Save this article

Experimental methods: Eliciting beliefs

Author

Listed:
  • Charness, Gary
  • Gneezy, Uri
  • Rasocha, Vlastimil

Abstract

Expectations are a critical factor in determining actions in a great variety of economic interactions. Hence, being able to measure beliefs is important in many economic environments. In this article, we review the approaches that have been used to measure beliefs and make comparisons of their effectiveness. We also discuss belief elicitation when the truth is not verifiable. We find that introspection (non-incentivized responses) seems to do as well as rather complex incentivized methods. We conjecture that simple and easy-to-comprehend incentivized methods are superior to introspection, in fact there are very few studies making such comparisons; this is an avenue for future research. We also discuss ways in which relatively complex methods could be made easier for usage in experimental work.

Suggested Citation

  • Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    • Handle: RePEc:eee:jeborg:v:189:y:2021:i:c:p:234-256
    DOI: 10.1016/j.jebo.2021.06.032
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167268121002717
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jebo.2021.06.032?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Terrance Hurley & Jason Shogren, 2005. "An Experimental Comparison of Induced and Elicited Beliefs," Journal of Risk and Uncertainty, Springer, vol. 30(2), pages 169-188, January.
    2. Rutström, E. Elisabet & Wilcox, Nathaniel T., 2009. "Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test," Games and Economic Behavior, Elsevier, vol. 67(2), pages 616-632, November.
    3. Steffen Andersen & John Fountain & Glenn Harrison & E. Rutström, 2014. "Estimating subjective probabilities," Journal of Risk and Uncertainty, Springer, vol. 48(3), pages 207-229, June.
    4. Reinhard Selten & Abdolkarim Sadrieh & Klaus Abbink, 1999. "Money Does Not Induce Risk Neutral Behavior, but Binary Lotteries Do even Worse," Theory and Decision, Springer, vol. 46(3), pages 213-252, June.
    5. Holt, Charles A. & Smith, Angela M., 2009. "An update on Bayesian updating," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 125-134, February.
    6. Frank Heinemann & Rosemarie Nagel & Peter Ockenfels, 2009. "Measuring Strategic Uncertainty in Coordination Games," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(1), pages 181-221.
    7. Miguel A. Costa-Gomes & Georg Weizsäcker, 2008. "Stated Beliefs and Play in Normal-Form Games," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 75(3), pages 729-762.
    8. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    9. Nolan Miller & Paul Resnick & Richard Zeckhauser, 2005. "Eliciting Informative Feedback: The Peer-Prediction Method," Management Science, INFORMS, vol. 51(9), pages 1359-1373, September.
    10. Morgan R Frank & Manuel Cebrian & Galen Pickard & Iyad Rahwan, 2017. "Validating Bayesian truth serum in large-scale online human experiments," PLOS ONE, Public Library of Science, vol. 12(5), pages 1-13, May.
    11. Edi Karni, 2009. "A Mechanism for Eliciting Probabilities," Econometrica, Econometric Society, vol. 77(2), pages 603-606, March.
    12. Tanjim Hossain & Ryo Okui, 2013. "The Binarized Scoring Rule," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 80(3), pages 984-1001.
    13. Simon Gächter & Elke Renner, 2010. "The effects of (incentivized) belief elicitation in public goods experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(3), pages 364-377, September.
    14. Shengwu Li, 2017. "Obviously Strategy-Proof Mechanisms," American Economic Review, American Economic Association, vol. 107(11), pages 3257-3287, November.
    15. Kyle Hyndman & Antoine Terracol & Jonathan Vaksmann, 2022. "Beliefs and (in)stability in normal-form games," Experimental Economics, Springer;Economic Science Association, vol. 25(4), pages 1146-1172, September.
    16. Fehr, Dietmar & Kübler, Dorothea & Danz, David, 2008. "Information and beliefs in a repeated normal-form game," SFB 649 Discussion Papers 2008-026, Humboldt University Berlin, Collaborative Research Center 649: Economic Risk.
    17. Ingrid Burfurd & Tom Wilkening, 2018. "Experimental guidance for eliciting beliefs with the Stochastic Becker–DeGroot–Marschak mechanism," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 4(1), pages 15-28, July.
    18. Palfrey, Thomas R. & Wang, Stephanie W., 2009. "On eliciting beliefs in strategic games," Journal of Economic Behavior & Organization, Elsevier, vol. 71(2), pages 98-109, August.
    19. Charles A. Holt & Susan K. Laury, 2002. "Risk Aversion and Incentive Effects," American Economic Review, American Economic Association, vol. 92(5), pages 1644-1655, December.
    20. Ernst Fehr & Klaus M. Schmidt, 1999. "A Theory of Fairness, Competition, and Cooperation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 114(3), pages 817-868.
    21. David Danz & Dietmar Fehr & Dorothea Kübler, 2012. "Information and beliefs in a repeated normal-form game," Experimental Economics, Springer;Economic Science Association, vol. 15(4), pages 622-640, December.
    22. Ryan Oprea, 2020. "What Makes a Rule Complex?," American Economic Review, American Economic Association, vol. 110(12), pages 3913-3951, December.
    23. Rustichini, Aldo & Villeval, Marie Claire, 2014. "Moral hypocrisy, power and social preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 107(PA), pages 10-24.
    24. Gary Charness & Dan Levin, 2005. "When Optimal Choices Feel Wrong: A Laboratory Study of Bayesian Updating, Complexity, and Affect," American Economic Review, American Economic Association, vol. 95(4), pages 1300-1309, September.
    25. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    26. Ernan Haruvy & Yaron Lahav & Charles N. Noussair, 2007. "Traders' Expectations in Asset Markets: Experimental Evidence," American Economic Review, American Economic Association, vol. 97(5), pages 1901-1920, December.
    27. Manski, Charles F. & Neri, Claudia, 2013. "First- and second-order subjective expectations in strategic decision-making: Experimental evidence," Games and Economic Behavior, Elsevier, vol. 81(C), pages 232-254.
    28. David Danz & Lise Vesterlund & Alistair J. Wilson, 2020. "Belief Elicitation: Limiting Truth Telling with Information on Incentives," NBER Working Papers 27327, National Bureau of Economic Research, Inc.
    29. Nathaniel T. Wilcox & Nick Feltovich, 2000. "Thinking Like a Game Theorist: Comment," Monash Economics Working Papers archive-30, Monash University, Department of Economics.
    30. Alejandro Martínez-Marquina & Muriel Niederle & Emanuel Vespa, 2019. "Failures in Contingent Reasoning: The Role of Uncertainty," American Economic Review, American Economic Association, vol. 109(10), pages 3437-3474, October.
    31. Erin L. Krupka & Roberto A. Weber, 2013. "Identifying Social Norms Using Coordination Games: Why Does Dictator Game Sharing Vary?," Journal of the European Economic Association, European Economic Association, vol. 11(3), pages 495-524, June.
    32. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd, 2014. "Eliciting subjective probabilities with binary lotteries," Journal of Economic Behavior & Organization, Elsevier, vol. 101(C), pages 128-140.
    33. Rey-Biel, Pedro, 2009. "Equilibrium play and best response to (stated) beliefs in normal form games," Games and Economic Behavior, Elsevier, vol. 65(2), pages 572-585, March.
    34. Matthew Rabin, 2000. "Risk Aversion and Expected-Utility Theory: A Calibration Theorem," Econometrica, Econometric Society, vol. 68(5), pages 1281-1292, September.
    35. Karl Schlag & James Tremewan, 2021. "Simple belief elicitation: An experimental evaluation," Journal of Risk and Uncertainty, Springer, vol. 62(2), pages 137-155, April.
    36. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd & Ulm, Eric R., 2017. "Scoring rules for subjective probability distributions," Journal of Economic Behavior & Organization, Elsevier, vol. 134(C), pages 430-448.
    37. Michael D. Hurd, 2009. "Subjective Probabilities in Household Surveys," Annual Review of Economics, Annual Reviews, vol. 1(1), pages 543-564, May.
    38. Xiao, Erte & Houser, Daniel, 2011. "Punish in public," Journal of Public Economics, Elsevier, vol. 95(7), pages 1006-1017.
    39. Emanuel Vespa & Alistair J. Wilson, 2016. "Communication with multiple senders: An experiment," Quantitative Economics, Econometric Society, vol. 7(1), pages 1-36, March.
    40. Charness, Gary & Gneezy, Uri & Imas, Alex, 2013. "Experimental methods: Eliciting risk preferences," Journal of Economic Behavior & Organization, Elsevier, vol. 87(C), pages 43-51.
    41. Charles A. Holt & Angela M. Smith, 2016. "Belief Elicitation with a Synchronized Lottery Choice Menu That Is Invariant to Risk Attitudes," American Economic Journal: Microeconomics, American Economic Association, vol. 8(1), pages 110-139, February.
    42. Armantier, Olivier & Treich, Nicolas, 2013. "Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging," European Economic Review, Elsevier, vol. 62(C), pages 17-40.
    43. Charness, Gary & Dufwenberg, Martin, 2003. "Promises & Partnership," Research Papers in Economics 2003:3, Stockholm University, Department of Economics.
    44. Smith, Vernon L & Walker, James M, 1993. "Monetary Rewards and Decision Cost in Experimental Economics," Economic Inquiry, Western Economic Association International, vol. 31(2), pages 245-261, April.
    45. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    46. Theo Offerman & Joep Sonnemans & Gijs Van De Kuilen & Peter P. Wakker, 2009. "A Truth Serum for Non-Bayesians: Correcting Proper Scoring Rules for Risk Attitudes ," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(4), pages 1461-1489.
    47. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    48. Ivanov, Asen, 2011. "Attitudes to ambiguity in one-shot normal-form games: An experimental study," Games and Economic Behavior, Elsevier, vol. 71(2), pages 366-394, March.
    49. Franklin Allen, 1987. "Notes--Discovering Personal Probabilities When Utility Functions are Unknown," Management Science, INFORMS, vol. 33(4), pages 542-544, April.
    50. Gary Charness & Martin Dufwenberg, 2006. "Promises and Partnership," Econometrica, Econometric Society, vol. 74(6), pages 1579-1601, November.
    51. Theo Offerman & Asa B. Palley, 2016. "Lossed in translation: an off-the-shelf method to recover probabilistic beliefs from loss-averse agents," Experimental Economics, Springer;Economic Science Association, vol. 19(1), pages 1-30, March.
    52. Wang, Stephanie W., 2011. "Incentive effects: The case of belief elicitation from individuals in groups," Economics Letters, Elsevier, vol. 111(1), pages 30-33, April.
    53. Aurélien Baillon & Han Bleichrodt, 2015. "Testing Ambiguity Models through the Measurement of Probabilities for Gains and Losses," American Economic Journal: Microeconomics, American Economic Association, vol. 7(2), pages 77-100, May.
    54. Daniel Zizzo, 2010. "Experimenter demand effects in economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(1), pages 75-98, March.
    55. Jakša Cvitanić & Dražen Prelec & Blake Riley & Benjamin Tereick, 2019. "Honesty via Choice-Matching," American Economic Review: Insights, American Economic Association, vol. 1(2), pages 179-192, September.
    56. Dufwenberg, Martin & Gneezy, Uri, 2000. "Measuring Beliefs in an Experimental Lost Wallet Game," Games and Economic Behavior, Elsevier, vol. 30(2), pages 163-182, February.
    57. Theo Offerman & Asa Palley, 2016. "Lossed in translation: an off-the-shelf method to recover probabilistic beliefs from loss-averse agents," Experimental Economics, Springer;Economic Science Association, vol. 19(1), pages 1-30, March.
    58. Barrage, Lint & Lee, Min Sok, 2010. "A penny for your thoughts: Inducing truth-telling in stated preference elicitation," Economics Letters, Elsevier, vol. 106(2), pages 140-142, February.
    59. Yaw Nyarko & Andrew Schotter, 2002. "An Experimental Study of Belief Learning Using Elicited Beliefs," Econometrica, Econometric Society, vol. 70(3), pages 971-1005, May.
    60. Yoram Halevy, 2015. "Time Consistency: Stationarity and Time Invariance," Econometrica, Econometric Society, vol. 83, pages 335-352, January.
    61. Croson, Rachel T. A., 2000. "Thinking like a game theorist: factors affecting the frequency of equilibrium play," Journal of Economic Behavior & Organization, Elsevier, vol. 41(3), pages 299-314, March.
    62. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    63. Li Hao & Daniel Houser, 2012. "Belief elicitation in the presence of naïve respondents: An experimental study," Journal of Risk and Uncertainty, Springer, vol. 44(2), pages 161-180, April.
    64. Erkal, Nisvan & Gangadharan, Lata & Koh, Boon Han, 2020. "Replication: Belief elicitation with quadratic and binarized scoring rules," Journal of Economic Psychology, Elsevier, vol. 81(C).
    65. Ronald Bosman & Frans van Winden, 2002. "Emotional Hazard in a Power-to-take Experiment," Economic Journal, Royal Economic Society, vol. 112(476), pages 147-169, January.
    66. Erte Xiao & Daniel Houser, 2005. "Emotion expression in human punishment behavior," Experimental 0504003, University Library of Munich, Germany, revised 18 May 2005.
    67. Ramsey, Frank P., 1926. "Truth and Probability," Histoy of Economic Thought Chapters, in: Braithwaite, R. B. (ed.),The Foundations of Mathematics and other Logical Essays, chapter 7, pages 156-198, McMaster University Archive for the History of Economic Thought.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    2. Eyting, Markus & Schmidt, Patrick, 2021. "Belief elicitation with multiple point predictions," European Economic Review, Elsevier, vol. 135(C).
    3. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    4. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    5. Armantier, Olivier & Treich, Nicolas, 2013. "Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging," European Economic Review, Elsevier, vol. 62(C), pages 17-40.
    6. Steffen Andersen & John Fountain & Glenn Harrison & E. Rutström, 2014. "Estimating subjective probabilities," Journal of Risk and Uncertainty, Springer, vol. 48(3), pages 207-229, June.
    7. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    8. Polonio, Luca & Coricelli, Giorgio, 2019. "Testing the level of consistency between choices and beliefs in games using eye-tracking," Games and Economic Behavior, Elsevier, vol. 113(C), pages 566-586.
    9. Bauer, Dominik & Wolff, Irenaeus, 2019. "Biases in Beliefs," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203601, Verein für Socialpolitik / German Economic Association.
    10. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    11. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    12. Markus Eyting & Patrick Schmidt, 2019. "Belief Elicitation with Multiple Point Predictions," Working Papers 1818, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz, revised 16 Nov 2020.
    13. Blanco, Mariana & Engelmann, Dirk & Koch, Alexander K. & Normann, Hans-Theo, 2014. "Preferences and beliefs in a sequential social dilemma: a within-subjects analysis," Games and Economic Behavior, Elsevier, vol. 87(C), pages 122-135.
    14. Ingrid Burfurd & Tom Wilkening, 2022. "Cognitive heterogeneity and complex belief elicitation," Experimental Economics, Springer;Economic Science Association, vol. 25(2), pages 557-592, April.
    15. Claudia Neri, 2015. "Eliciting beliefs in continuous-choice games: a double auction experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(4), pages 569-608, December.
    16. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    17. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Theory and Decision, Springer, vol. 80(3), pages 363-387, March.
    18. Tsakas, Elias, 2018. "Robust scoring rules," Research Memorandum 023, Maastricht University, Graduate School of Business and Economics (GSBE).
    19. Evdokimov, Piotr & Rustichini, Aldo, 2016. "Forward induction: Thinking and behavior," Journal of Economic Behavior & Organization, Elsevier, vol. 128(C), pages 195-208.
    20. Kai Barron, 2021. "Belief updating: does the ‘good-news, bad-news’ asymmetry extend to purely financial domains?," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 31-58, March.

    More about this item

    Keywords

    Experimental methodology; Belief elicitation; Comprehension; Scoring rules; Incentives;
    All these keywords.

    JEL classification:

    • B49 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Other
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • C92 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Group Behavior

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jeborg:v:189:y:2021:i:c:p:234-256. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jebo .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.