IDEAS home Printed from https://ideas.repec.org/a/eee/joepsy/v81y2020ics0167487020300763.html
   My bibliography  Save this article

Replication: Belief elicitation with quadratic and binarized scoring rules

Author

Listed:
  • Erkal, Nisvan
  • Gangadharan, Lata
  • Koh, Boon Han

Abstract

Researchers increasingly elicit beliefs to understand the underlying motivations of decision makers. Two commonly used methods are the quadratic scoring rule (QSR) and the binarized scoring rule (BSR). Hossain and Okui (2013) use a within-subject design to evaluate the performance of these two methods in an environment where subjects report probabilistic beliefs over binary outcomes with objective probabilities. In a near replication of their study, we show that their results continue to hold with a between-subject design. This is an important validation of the BSR given that researchers typically implement only one method to elicit beliefs. In favor of the BSR, reported beliefs are less accurate under the QSR than the BSR. Consistent with theoretical predictions, risk-averse subjects distort their reported beliefs under the QSR.

Suggested Citation

  • Erkal, Nisvan & Gangadharan, Lata & Koh, Boon Han, 2020. "Replication: Belief elicitation with quadratic and binarized scoring rules," Journal of Economic Psychology, Elsevier, vol. 81(C).
    • Handle: RePEc:eee:joepsy:v:81:y:2020:i:c:s0167487020300763
    DOI: 10.1016/j.joep.2020.102315
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167487020300763
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.joep.2020.102315?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    2. Charness, Gary & Gneezy, Uri & Kuhn, Michael A., 2012. "Experimental methods: Between-subject and within-subject design," Journal of Economic Behavior & Organization, Elsevier, vol. 81(1), pages 1-8.
    3. Tanjim Hossain & Ryo Okui, 2013. "The Binarized Scoring Rule," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 80(3), pages 984-1001.
    4. Mohammed Abdellaoui & Ahmed Driouchi & Olivier L’Haridon, 2011. "Risk aversion elicitation: reconciling tractability and bias minimization," Theory and Decision, Springer, vol. 71(1), pages 63-80, July.
    5. Brown, Paul M. & Stewart, Steven, 1999. "Avoiding severe environmental consequences: evidence on the role of loss avoidance and risk attitudes," Journal of Economic Behavior & Organization, Elsevier, vol. 38(2), pages 179-198, February.
    6. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd, 2014. "Eliciting subjective probabilities with binary lotteries," Journal of Economic Behavior & Organization, Elsevier, vol. 101(C), pages 128-140.
    7. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd & Ulm, Eric R., 2015. "Eliciting subjective probability distributions with binary lotteries," Economics Letters, Elsevier, vol. 127(C), pages 68-71.
    8. Marie-Pierre Dargnies & Rustamdjan Hakimov & Dorothea Kübler, 2019. "Self-Confidence and Unraveling in Matching Markets," Management Science, INFORMS, vol. 65(12), pages 5603-5618, December.
    9. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    10. Linda Babcock & Maria P. Recalde & Lise Vesterlund & Laurie Weingart, 2017. "Gender Differences in Accepting and Receiving Requests for Tasks with Low Promotability," American Economic Review, American Economic Association, vol. 107(3), pages 714-747, March.
    11. Charles A. Holt & Angela M. Smith, 2016. "Belief Elicitation with a Synchronized Lottery Choice Menu That Is Invariant to Risk Attitudes," American Economic Journal: Microeconomics, American Economic Association, vol. 8(1), pages 110-139, February.
    12. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    13. Eva Wölbert & Arno Riedl, 2013. "Measuring Time and Risk Preferences: Realiability, Stability, Domain Specificity," CESifo Working Paper Series 4339, CESifo.
    14. Paolo Crosetto & Antonio Filippin, 2016. "A theoretical and experimental appraisal of four risk elicitation methods," Experimental Economics, Springer;Economic Science Association, vol. 19(3), pages 613-641, September.
    15. Franklin Allen, 1987. "Notes--Discovering Personal Probabilities When Utility Functions are Unknown," Management Science, INFORMS, vol. 33(4), pages 542-544, April.
    16. Berg, Joyce E. & Rietz, Thomas A. & Dickhaut, John W., 2008. "On the Performance of the Lottery Procedure for Controlling Risk Preferences," Handbook of Experimental Economics Results, in: Charles R. Plott & Vernon L. Smith (ed.), Handbook of Experimental Economics Results, edition 1, volume 1, chapter 115, pages 1087-1097, Elsevier.
    17. McKelvey, Richard D & Page, Talbot, 1990. "Public and Private Information: An Experimental Study of Information Pooling," Econometrica, Econometric Society, vol. 58(6), pages 1321-1339, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    2. Nisvan Erkal & Lata Gangadharan & Boon Han Koh, 2022. "By chance or by choice? Biased attribution of others’ outcomes when social preferences matter," Experimental Economics, Springer;Economic Science Association, vol. 25(2), pages 413-443, April.
    3. Aksoy, Billur & Chadd, Ian & Koh, Boon Han, 2023. "Sexual identity, gender, and anticipated discrimination in prosocial behavior," European Economic Review, Elsevier, vol. 154(C).
    4. Nisvan Erkal & Lata Gangadharan & Boon Han Koh, 2023. "Discrimination in Evaluation Criteria: The Role of Beliefs versus Outcomes," Discussion Papers 2316, University of Exeter, Department of Economics.
    5. Erkal, Nisvan & Gangadharan, Lata & Koh, Boon Han, 2023. "Do women receive less blame than men? Attribution of outcomes in a prosocial setting," Journal of Economic Behavior & Organization, Elsevier, vol. 210(C), pages 441-452.
    6. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    7. Lata Gangadharan & Philip J. Grossman & Nina Xue, 2022. "Stepping Stone: Identifying self-image concerns from motivated beliefs: Does it matter how and whom you ask?," Monash Economics Working Papers 2022-05, Monash University, Department of Economics.
    8. Alexander Coutts & Boon Han Koh & Zahra Murad, 2024. "The signals we give: Performance feedback, gender, and competition," Working Papers in Economics & Finance 2024-02, University of Portsmouth, Portsmouth Business School, Economics and Finance Subject Group.
    9. Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    10. Sofianos, Andis, 2022. "Self-reported & revealed trust: Experimental evidence," Journal of Economic Psychology, Elsevier, vol. 88(C).
    11. Nisvan Erkal & Lata Gangadharan & Boon Han Koh, 2021. "Gender Biases in Performance Evaluation: The Role of Beliefs Versus Outcomes," University of East Anglia School of Economics Working Paper Series 2021-09, School of Economics, University of East Anglia, Norwich, UK..
    12. Charles A. Holt & Sean P. Sullivan, 2023. "Permutation tests for experimental data," Experimental Economics, Springer;Economic Science Association, vol. 26(4), pages 775-812, September.
    13. Bnaya Dreyfuss & Raphael Raux, 2024. "Human Learning about AI Performance," Papers 2406.05408, arXiv.org.
    14. Lata Gangadharan & Philip J. Grossman & Nina Xue, 2021. "Identifying self-image concerns from motivated beliefs: Does it matter how and whom you ask?," Monash Economics Working Papers 2021-17, Monash University, Department of Economics.
    15. Janas, Moritz & Jordan, Michelle, 2024. "Cheap signaling of altruism," Journal of Economic Psychology, Elsevier, vol. 102(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eyting, Markus & Schmidt, Patrick, 2021. "Belief elicitation with multiple point predictions," European Economic Review, Elsevier, vol. 135(C).
    2. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    3. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    4. Bauer, Dominik & Wolff, Irenaeus, 2019. "Biases in Beliefs," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203601, Verein für Socialpolitik / German Economic Association.
    5. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    6. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    7. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    8. David Danz & Lise Vesterlund & Alistair J. Wilson, 2020. "Belief Elicitation: Limiting Truth Telling with Information on Incentives," CESifo Working Paper Series 8048, CESifo.
    9. Ingrid Burfurd & Tom Wilkening, 2022. "Cognitive heterogeneity and complex belief elicitation," Experimental Economics, Springer;Economic Science Association, vol. 25(2), pages 557-592, April.
    10. Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    11. Lata Gangadharan & Philip J. Grossman & Nina Xue, 2021. "Identifying self-image concerns from motivated beliefs: Does it matter how and whom you ask?," Monash Economics Working Papers 2021-17, Monash University, Department of Economics.
    12. Markus Eyting & Patrick Schmidt, 2019. "Belief Elicitation with Multiple Point Predictions," Working Papers 1818, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz, revised 16 Nov 2020.
    13. Lata Gangadharan & Philip J. Grossman & Nina Xue, 2022. "Stepping Stone: Identifying self-image concerns from motivated beliefs: Does it matter how and whom you ask?," Monash Economics Working Papers 2022-05, Monash University, Department of Economics.
    14. Crosetto, Paolo & Filippin, Antonio & Katuščák, Peter & Smith, John, 2020. "Central tendency bias in belief elicitation," Journal of Economic Psychology, Elsevier, vol. 78(C).
    15. Corazzini, Luca & Galavotti, Stefano & Valbonesi, Paola, 2019. "An experimental study on sequential auctions with privately known capacities," Games and Economic Behavior, Elsevier, vol. 117(C), pages 289-315.
    16. Kirchkamp, Oliver & Oechssler, Joerg & Sofianos, Andis, 2021. "The Binary Lottery Procedure does not induce risk neutrality in the Holt & Laury and Eckel & Grossman tasks," Journal of Economic Behavior & Organization, Elsevier, vol. 185(C), pages 348-369.
    17. Jean-Pierre Benoît & Juan Dubra & Giorgia Romagnoli, 2022. "Belief Elicitation When More than Money Matters: Controlling for "Control"," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 837-888, August.
    18. Rafkin, Charlie & Shreekumar, Advik & Vautrey, Pierre-Luc, 2021. "When guidance changes: Government stances and public beliefs," Journal of Public Economics, Elsevier, vol. 196(C).
    19. Tsakas, Elias, 2018. "Robust scoring rules," Research Memorandum 023, Maastricht University, Graduate School of Business and Economics (GSBE).
    20. Tsakas, Elias, 2020. "Robust scoring rules," Theoretical Economics, Econometric Society, vol. 15(3), July.

    More about this item

    Keywords

    Belief elicitation; Risk preferences; Experimental methodology; Scoring rules; Prediction accuracy;
    All these keywords.

    JEL classification:

    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • D81 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Criteria for Decision-Making under Risk and Uncertainty
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:joepsy:v:81:y:2020:i:c:s0167487020300763. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/joep .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.