IDEAS home Printed from https://ideas.repec.org/p/rco/dpaper/146.html
   My bibliography  Save this paper

Incentives, Search Engines, and the Elicitation of Subjective Beliefs: Evidence From Representative Online Survey Experiments

Author

Listed:
  • Grewenig, Elisabeth

    (ifo Institute)

  • Lergetporer, Philipp

    (ifo Institute)

  • Werner, Katharina

    (ifo Institute)

  • Woessmann, Ludger

    (ifo Institute and LMU Munich)

Abstract

A large literature studies subjective beliefs about economic facts using unincentivized survey questions. We devise randomized experiments in a representative online survey to investigate whether incentivizing belief accuracy affects stated beliefs about average earnings by professional degree and average public school spending. Incentive provision does not impact earnings beliefs, but improves school-spending beliefs. Response patterns suggest that the latter effect likely reflects increased online-search activity. Consistently, an experiment that just encourages search-engine usage produces very similar results. Another experiment provides no evidence of experimenter-demand effects. Overall, results suggest that incentive provision does not reduce bias in our survey-based belief measures.

Suggested Citation

  • Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2019. "Incentives, Search Engines, and the Elicitation of Subjective Beliefs: Evidence From Representative Online Survey Experiments," Rationality and Competition Discussion Paper Series 146, CRC TRR 190 Rationality and Competition.
    • Handle: RePEc:rco:dpaper:146
    as

    Download full text from publisher

    File URL: https://rationality-and-competition.de/wp-content/uploads/discussion_paper/146.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Luis Armona & Andreas Fuster & Basit Zafar, 2019. "Home Price Expectations and Behaviour: Evidence from a Randomized Information Experiment," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 86(4), pages 1371-1410.
    2. Olivier Armantier & Scott Nelson & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2016. "The Price Is Right: Updating Inflation Expectations in a Randomized Price Information Experiment," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 503-523, July.
    3. Terrance Hurley & Jason Shogren, 2005. "An Experimental Comparison of Induced and Elicited Beliefs," Journal of Risk and Uncertainty, Springer, vol. 30(2), pages 169-188, January.
    4. Olivier Coibion & Yuriy Gorodnichenko & Saten Kumar, 2018. "How Do Firms Form Their Expectations? New Survey Evidence," American Economic Review, American Economic Association, vol. 108(9), pages 2671-2713, September.
    5. Ansolabehere, Stephen & Meredith, Marc & Snowberg, Erik, 2013. "Asking About Numbers: Why and How," Political Analysis, Cambridge University Press, vol. 21(1), pages 48-69, January.
    6. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    7. Antonio A. Arechar & Simon Gächter & Lucas Molleman, 2018. "Conducting interactive experiments online," Experimental Economics, Springer;Economic Science Association, vol. 21(1), pages 99-131, March.
    8. Bleemer, Zachary & Zafar, Basit, 2018. "Intended college attendance: Evidence from an experiment on college returns and costs," Journal of Public Economics, Elsevier, vol. 157(C), pages 184-211.
    9. Olivier Coibion & Yuriy Gorodnichenko, 2012. "What Can Survey Forecasts Tell Us about Information Rigidities?," Journal of Political Economy, University of Chicago Press, vol. 120(1), pages 116-159.
    10. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    11. Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & West, Martin R. & Woessmann, Ludger, 2018. "How information affects support for education spending: Evidence from survey experiments in Germany and the United States," Journal of Public Economics, Elsevier, vol. 167(C), pages 138-157.
    12. Mummolo, Jonathan & Peterson, Erik, 2019. "Demand Effects in Survey Experiments: An Empirical Assessment," American Political Science Review, Cambridge University Press, vol. 113(2), pages 517-529, May.
    13. John Horton & David Rand & Richard Zeckhauser, 2011. "The online laboratory: conducting experiments in a real labor market," Experimental Economics, Springer;Economic Science Association, vol. 14(3), pages 399-425, September.
    14. Charles F. Manski, 2018. "Survey Measurement of Probabilistic Macroeconomic Expectations: Progress and Promise," NBER Macroeconomics Annual, University of Chicago Press, vol. 32(1), pages 411-471.
    15. Arcidiacono, Peter & Hotz, V. Joseph & Kang, Songman, 2012. "Modeling college major choices using elicited measures of expectations and counterfactuals," Journal of Econometrics, Elsevier, vol. 166(1), pages 3-16.
    16. Grigorieff, Alexis & Roth, Christopher & Ubfal, Diego, 2016. "Does Information Change Attitudes Towards Immigrants? Representative Evidence from Survey Experiments," IZA Discussion Papers 10419, Institute of Labor Economics (IZA).
    17. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    18. Katja Maria Kaufmann, 2014. "Understanding the income gradient in college attendance in Mexico: The role of heterogeneity in expected returns," Quantitative Economics, Econometric Society, vol. 5(3), pages 583-630, November.
    19. Florian Zimmermann, 2020. "The Dynamics of Motivated Beliefs," American Economic Review, American Economic Association, vol. 110(2), pages 337-361, February.
    20. Mounir Karadja & Johanna Mollerstrom & David Seim, 2017. "Richer (and Holier) Than Thou? The Effect of Relative Income Improvements on Demand for Redistribution," The Review of Economics and Statistics, MIT Press, vol. 99(2), pages 201-212, May.
    21. Teodora Boneva & Christopher Rauh, 2017. "Socio-Economic Gaps in University Enrollment: The Role of Perceived Pecuniary and Non-Pecuniary Returns," Working Papers 2017-080, Human Capital and Economic Opportunity Working Group.
    22. N. Gregory Mankiw & Ricardo Reis & Justin Wolfers, 2004. "Disagreement about Inflation Expectations," NBER Chapters, in: NBER Macroeconomics Annual 2003, Volume 18, pages 209-270, National Bureau of Economic Research, Inc.
    23. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).
    24. Alberto Alesina & Stefanie Stantcheva & Edoardo Teso, 2018. "Intergenerational Mobility and Preferences for Redistribution," American Economic Review, American Economic Association, vol. 108(2), pages 521-554, February.
    25. Zachary Bleemer & Basit Zafar, 2014. "Information heterogeneity and intended college enrollment," Staff Reports 685, Federal Reserve Bank of New York.
    26. Simon Gächter & Elke Renner, 2010. "The effects of (incentivized) belief elicitation in public goods experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(3), pages 364-377, September.
    27. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    28. Alesina, Alberto & La Ferrara, Eliana, 2005. "Preferences for redistribution in the land of opportunities," Journal of Public Economics, Elsevier, vol. 89(5-6), pages 897-931, June.
    29. Christopher Roth & Johannes Wohlfart, 2020. "How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior?," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 731-748, October.
    30. Teodora Boneva & Christopher Rauh, 2018. "Parental Beliefs about Returns to Educational Investments—The Later the Better?," Journal of the European Economic Association, European Economic Association, vol. 16(6), pages 1669-1711.
    31. Biroli, Pietro & Boneva, Teodora & Raja, Akash & Rauh, Christopher, 2022. "Parental beliefs about returns to child health investments," Journal of Econometrics, Elsevier, vol. 231(1), pages 33-57.
    32. Martin G. Kocher & David Schindler & Stefan T. Trautmann & Yilong Xu, 2019. "Risk, time pressure, and selection effects," Experimental Economics, Springer;Economic Science Association, vol. 22(1), pages 216-246, March.
    33. Ilyana Kuziemko & Michael I. Norton & Emmanuel Saez & Stefanie Stantcheva, 2015. "How Elastic Are Preferences for Redistribution? Evidence from Randomized Survey Experiments," American Economic Review, American Economic Association, vol. 105(4), pages 1478-1508, April.
    34. Olivier Armantier & Wändi Bruine de Bruin & Simon Potter & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2013. "Measuring Inflation Expectations," Annual Review of Economics, Annual Reviews, vol. 5(1), pages 273-301, May.
    35. Arcidiacono, Peter, 2004. "Ability sorting and the returns to college major," Journal of Econometrics, Elsevier, vol. 121(1-2), pages 343-375.
    36. Basit Zafar, 2011. "How Do College Students Form Expectations?," Journal of Labor Economics, University of Chicago Press, vol. 29(2), pages 301-348.
    37. Wang, Stephanie W., 2011. "Incentive effects: The case of belief elicitation from individuals in groups," Economics Letters, Elsevier, vol. 111(1), pages 30-33, April.
    38. Andreas Fuster & Ricardo Perez-Truglia & Mirko Wiederholt & Basit Zafar, 2022. "Expectations with Endogenous Information Acquisition: An Experimental Investigation," The Review of Economics and Statistics, MIT Press, vol. 104(5), pages 1059-1078, December.
    39. Alberto Cavallo & Guillermo Cruces & Ricardo Perez-Truglia, 2017. "Inflation Expectations, Learning, and Supermarket Prices: Evidence from Survey Experiments," American Economic Journal: Macroeconomics, American Economic Association, vol. 9(3), pages 1-35, July.
    40. Matthew Wiswall & Basit Zafar, 2015. "Determinants of College Major Choice: Identification using an Information Experiment," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 82(2), pages 791-824.
    41. Daniel Zizzo, 2010. "Experimenter demand effects in economic experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(1), pages 75-98, March.
    42. Armin Falk & Anke Becker & Thomas Dohmen & David Huffman & Uwe Sunde, 2023. "The Preference Survey Module: A Validated Instrument for Measuring Risk, Time, and Social Preferences," Management Science, INFORMS, vol. 69(4), pages 1935-1950, April.
    43. Orazio Attanasio & Teodora Boneva & Christopher Rauh, 2022. "Parental Beliefs about Returns to Different Types of Investments in School Children," Journal of Human Resources, University of Wisconsin Press, vol. 57(6), pages 1789-1825.
    44. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," IZA Discussion Papers 11799, Institute of Labor Economics (IZA).
    45. Prior, Markus & Sood, Gaurav & Khanna, Kabir, 2015. "You Cannot be Serious: The Impact of Accuracy Incentives on Partisan Bias in Reports of Economic Perceptions," Quarterly Journal of Political Science, now publishers, vol. 10(4), pages 489-518, December.
    46. Adeline Delavande & Xavier Giné & David McKenzie, 2011. "Eliciting probabilistic expectations with visual aids in developing countries: how sensitive are answers to variations in elicitation design?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(3), pages 479-497, April.
    47. Grigorieff, Alexis & Roth, Christopher & Ubfal, Diego, 2016. "Does Information Change Attitudes Towards Immigrants? Representative Evidence from Survey Experiments," IZA Discussion Papers 10419, Institute of Labor Economics (IZA).
    48. Jonathan de Quidt & Johannes Haushofer & Christopher Roth, 2018. "Measuring and Bounding Experimenter Demand," American Economic Review, American Economic Association, vol. 108(11), pages 3266-3302, November.
    49. Ralph Stinebrickner & Todd R. Stinebrickner, 2014. "A Major in Science? Initial Beliefs and Final Outcomes for College Major and Dropout," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(1), pages 426-472.
    50. Bullock, John G. & Gerber, Alan S. & Hill, Seth J. & Huber, Gregory A., 2015. "Partisan Bias in Factual Beliefs about Politics," Quarterly Journal of Political Science, now publishers, vol. 10(4), pages 519-578, December.
    51. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    52. Hugo Benítez-Silva & Moshe Buchinsky & Hiu Man Chan & Sofia Cheidvasser & John Rust, 2004. "How large is the bias in self-reported disability?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 19(6), pages 649-670.
    53. Clifford, Scott & Jerit, Jennifer, 2014. "Is There a Cost to Convenience? An Experimental Comparison of Data Quality in Laboratory and Online Studies," Journal of Experimental Political Science, Cambridge University Press, vol. 1(2), pages 120-131, January.
    54. Chris Belfield & Teodora Boneva & Christopher Rauh & Jonathan Shaw, 2020. "What Drives Enrolment Gaps in Further Education? The Role of Beliefs in Sequential Schooling Decisions," Economica, London School of Economics and Political Science, vol. 87(346), pages 490-529, April.
    55. Adeline Delavande & Basit Zafar, 2019. "University Choice: The Role of Expected Earnings, Nonpecuniary Outcomes, and Financial Constraints," Journal of Political Economy, University of Chicago Press, vol. 127(5), pages 2343-2393.
    56. Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
    57. Emin Karagözoğlu & Martin G. Kocher, 2019. "Bargaining under time pressure from deadlines," Experimental Economics, Springer;Economic Science Association, vol. 22(2), pages 419-440, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Katharina Dowling & Daniel Guhl & Daniel Klapper & Martin Spann & Lucas Stich & Narine Yegoryan, 2020. "Behavioral biases in marketing," Journal of the Academy of Marketing Science, Springer, vol. 48(3), pages 449-477, May.
    2. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2023. "Designing Information Provision Experiments," Journal of Economic Literature, American Economic Association, vol. 61(1), pages 3-40, March.
    3. Starkov, Egor, 2023. "Only time will tell: Credible dynamic signaling," Journal of Mathematical Economics, Elsevier, vol. 109(C).
    4. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    5. Juerg Schweri, 2021. "Predicting polytomous career choices in healthcare using probabilistic expectations data," Health Economics, John Wiley & Sons, Ltd., vol. 30(3), pages 544-563, March.
    6. Angerer, Silvia & Glätzle-Rützler, Daniela & Lergetporer, Philipp & Rittmannsberger, Thomas, 2024. "Beliefs about social norms and gender-based polarization of COVID-19 vaccination readiness," European Economic Review, Elsevier, vol. 163(C).
    7. Brad R. Taylor, 2020. "The psychological foundations of rational ignorance: biased heuristics and decision costs," Constitutional Political Economy, Springer, vol. 31(1), pages 70-88, March.
    8. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    9. Dietmar Fehr & Johanna Mollerstrom & Ricardo Perez-Truglia, 2022. "Your Place in the World: Relative Income and Global Inequality," American Economic Journal: Economic Policy, American Economic Association, vol. 14(4), pages 232-268, November.
    10. Freundl Vera & Grewenig Elisabeth & Kugler Franziska & Lergetporer Philipp & Schüler Ruth & Wedel Katharina & Werner Katharina & Wirth Olivia & Woessmann Ludger, 2023. "The ifo Education Survey 2014–2021: A New Dataset on Public Preferences for Education Policy in Germany," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 243(6), pages 699-710, December.
    11. Patrick Bareinz & Fabian Koenings, 2021. "Framing of Economic News and Policy Support During a Pandemic: Evidence from an Information Experiment," Jena Economics Research Papers 2021-004, Friedrich-Schiller-University Jena.
    12. Lergetporer, Philipp & Woessmann, Ludger, 2023. "Earnings information and public preferences for university tuition: Evidence from representative experiments," Journal of Public Economics, Elsevier, vol. 226(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ingar Haaland & Christopher Roth & Johannes Wohlfart, 2023. "Designing Information Provision Experiments," Journal of Economic Literature, American Economic Association, vol. 61(1), pages 3-40, March.
    2. Roth, Christopher & Settele, Sonja & Wohlfart, Johannes, 2022. "Beliefs about public debt and the demand for government spending," Journal of Econometrics, Elsevier, vol. 231(1), pages 165-187.
    3. Christopher Roth & Johannes Wohlfart, 2020. "How Do Expectations about the Macroeconomy Affect Personal Expectations and Behavior?," The Review of Economics and Statistics, MIT Press, vol. 102(4), pages 731-748, October.
    4. Lergetporer, Philipp & Woessmann, Ludger, 2019. "The Political Economy of Higher Education Finance: How Information and Design Affect Public Preferences for Tuition," Rationality and Competition Discussion Paper Series 145, CRC TRR 190 Rationality and Competition.
    5. Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina, 2020. "Gender Norms and Labor-Supply Expectations: Experimental Evidence from Adolescents," Rationality and Competition Discussion Paper Series 259, CRC TRR 190 Rationality and Competition.
    6. Katharina Werner, 2019. "The Role of Information for Public Preferences on Education – Evidence from Representative Survey Experiments," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 82.
    7. Pamela Giustinelli, 2022. "Expectations in Education: Framework, Elicitation, and Evidence," Working Papers 2022-026, Human Capital and Economic Opportunity Working Group.
    8. Philipp Lergetporer & Katharina Werner & Ludger Woessmann, 2021. "Does Ignorance of Economic Returns and Costs Explain the Educational Aspiration Gap? Representative Evidence from Adults and Adolescents," Economica, London School of Economics and Political Science, vol. 88(351), pages 624-670, July.
    9. Lergetporer, Philipp & Piopiunik, Marc & Simon, Lisa, 2021. "Does the education level of refugees affect natives’ attitudes?," European Economic Review, Elsevier, vol. 134(C).
    10. Lergetporer, Philipp & Woessmann, Ludger, 2023. "Earnings information and public preferences for university tuition: Evidence from representative experiments," Journal of Public Economics, Elsevier, vol. 226(C).
    11. Starkov, Egor, 2023. "Only time will tell: Credible dynamic signaling," Journal of Mathematical Economics, Elsevier, vol. 109(C).
    12. Blesse, Sebastian & Heinemann, Friedrich & Krieger, Tommy, 2021. "Informationsdefizite als Hindernis rationaler Wirtschaftspolitik: Ausmass, Ursachen und Gegenstrategien. Eine Studie mit Unterstützung der Brigitte Strube Stiftung," ZEW Expertises, ZEW - Leibniz Centre for European Economic Research, number 241989.
    13. Lekfuangfu, Warn N. & Odermatt, Reto, 2022. "All I have to do is dream? The role of aspirations in intergenerational mobility and well-being," European Economic Review, Elsevier, vol. 148(C).
    14. Marcelo Gantier & Rafael Novella & Andrea Repetto, 2024. "Subjective expectations and schooling choices in Latin America and the Caribbean," Journal of International Development, John Wiley & Sons, Ltd., vol. 36(3), pages 1593-1621, April.
    15. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    16. Dylong, Patrick & Koenings, Fabian, 2023. "Framing of economic news and policy support during a pandemic: Evidence from a survey experiment," European Journal of Political Economy, Elsevier, vol. 76(C).
    17. Michele Giannola, 2024. "Parental Investments and Intra-household Inequality in Child Human Capital: Evidence from a Survey Experiment," The Economic Journal, Royal Economic Society, vol. 134(658), pages 671-727.
    18. Cattaneo, Maria & Lergetporer, Philipp & Schwerdt, Guido & Werner, Katharina & Woessmann, Ludger & Wolter, Stefan C., 2020. "Information provision and preferences for education spending: Evidence from representative survey experiments in three countries," European Journal of Political Economy, Elsevier, vol. 63(C).
    19. Ingar Haaland & Christopher Roth, 2023. "Beliefs about Racial Discrimination and Support for Pro-Black Policies," The Review of Economics and Statistics, MIT Press, vol. 105(1), pages 40-53, January.
    20. Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2020. "Educational inequality and public policy preferences: Evidence from representative survey experiments," Journal of Public Economics, Elsevier, vol. 188(C).

    More about this item

    Keywords

    beliefs; incentives; online search; survey experiment;
    All these keywords.

    JEL classification:

    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:rco:dpaper:146. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Viviana Lalli (email available below). General contact details of provider: https://rationality-and-competition.de .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.