IDEAS home Printed from https://ideas.repec.org/p/tor/tecipa/tecipa-578.html
   My bibliography  Save this paper

Evaluating Strategic Forecasters

Author

Listed:
  • Rahul Deb
  • Mallesh Pai
  • Maher Said

Abstract

Motivated by the question of how one should evaluate professional election forecasters, we study a novel dynamic mechanism design problem without transfers. A principal who wishes to hire only high quality forecasters is faced with an agent of unknown quality. The agent privately observes signals about a publicly observable future event, and may strategically misrepresent information to inflate the principal's perception of his quality. We show that the optimal deterministic mechanism is simple and easy to implement in practice: it evaluates a single, optimally timed prediction. We study the generality of this result and its robustness to randomization and noncommitment.

Suggested Citation

  • Rahul Deb & Mallesh Pai & Maher Said, 2017. "Evaluating Strategic Forecasters," Working Papers tecipa-578, University of Toronto, Department of Economics.
    • Handle: RePEc:tor:tecipa:tecipa-578
    as

    Download full text from publisher

    File URL: https://www.economics.utoronto.ca/public/workingPapers/tecipa-578.pdf
    File Function: Main Text
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Guo, Yingni & Hörner, Johannes, 2015. "Dynamic Mechanisms without Money," Economics Series 310, Institute for Advanced Studies.
    2. Marco Battaglini, 2005. "Long-Term Contracting with Markovian Consumers," American Economic Review, American Economic Association, vol. 95(3), pages 637-658, June.
    3. Eső, Péter & Szentes, Balázs, 2017. "Dynamic contracting: an irrelevance theorem," LSE Research Online Documents on Economics 69403, London School of Economics and Political Science, LSE Library.
    4. Strausz, Roland, 2003. "Deterministic mechanisms and the revelation principle," Economics Letters, Elsevier, vol. 79(3), pages 333-337, June.
    5. Olszewski, Wojciech, 2015. "Calibration and Expert Testing," Handbook of Game Theory with Economic Applications,, Elsevier.
    6. Ying Chen & Navin Kartik & Joel Sobel, 2008. "Selecting Cheap-Talk Equilibria," Econometrica, Econometric Society, vol. 76(1), pages 117-136, January.
    7. Philippe Aghion & Matthew O. Jackson, 2016. "Inducing Leaders to Take Risky Decisions: Dismissal, Tenure, and Term Limits," American Economic Journal: Microeconomics, American Economic Association, vol. 8(3), pages 1-38, August.
    8. Graham Elliott & Allan Timmermann, 2016. "Forecasting in Economics and Finance," Annual Review of Economics, Annual Reviews, vol. 8(1), pages 81-110, October.
    9. Trueman, Brett, 1994. "Analyst Forecasts and Herding Behavior," The Review of Financial Studies, Society for Financial Studies, vol. 7(1), pages 97-124.
    10. Wojciech Olszewski & Marcin Pęski, 2011. "The Principal-Agent Approach to Testing Experts," American Economic Journal: Microeconomics, American Economic Association, vol. 3(2), pages 89-113, May.
    11. Lahiri, Kajal & Sheng, Xuguang, 2008. "Evolution of forecast disagreement in a Bayesian learning model," Journal of Econometrics, Elsevier, vol. 144(2), pages 325-340, June.
    12. Daniel Krähmer & Roland Strausz, 2011. "Optimal Procurement Contracts with Pre-Project Planning," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 78(3), pages 1015-1041.
    13. Bengt Holmström, 1999. "Managerial Incentive Problems: A Dynamic Perspective," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 66(1), pages 169-182.
    14. G. Elliott & C. Granger & A. Timmermann (ed.), 2013. "Handbook of Economic Forecasting," Handbook of Economic Forecasting, Elsevier, edition 1, volume 2, number 2.
    15. Roger B. Myerson, 1981. "Optimal Auction Design," Mathematics of Operations Research, INFORMS, vol. 6(1), pages 58-73, February.
    16. Alessandro Pavan & Ilya Segal & Juuso Toikka, 2014. "Dynamic Mechanism Design: A Myersonian Approach," Econometrica, Econometric Society, vol. 82(2), pages 601-653, March.
    17. Crawford, Vincent P & Sobel, Joel, 1982. "Strategic Information Transmission," Econometrica, Econometric Society, vol. 50(6), pages 1431-1451, November.
    18. Raphael Boleslavsky & Maher Said, 2013. "Progressive Screening: Long-Term Contracting with a Privately Known Stochastic Process," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 80(1), pages 1-34.
    19. Cremer, Jacques & McLean, Richard P, 1988. "Full Extraction of the Surplus in Bayesian and Dominant Strategy Auctions," Econometrica, Econometric Society, vol. 56(6), pages 1247-1257, November.
    20. Krishna, Vijay & Morgan, John, 2004. "The art of conversation: eliciting information from experts through multi-stage communication," Journal of Economic Theory, Elsevier, vol. 117(2), pages 147-179, August.
    21. Bengt Holmstrom, 1999. "Managerial Incentive Problems: A Dynamic Perspective," NBER Working Papers 6875, National Bureau of Economic Research, Inc.
    22. Wei Li, 2007. "Changing One's Mind when the Facts Change: Incentives of Experts and the Design of Reporting Protocols," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 74(4), pages 1175-1194.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Cipullo, Davide & Reslow, André, 2019. "Biased Forecasts to Affect Voting Decisions? The Brexit Case," Working Paper Series 364, Sveriges Riksbank (Central Bank of Sweden).
    2. Aleksei Smirnov & Egor Starkov, 2019. "Timing of predictions in dynamic cheap talk: experts vs. quacks," ECON - Working Papers 334, Department of Economics - University of Zurich.
    3. Dirk Bergemann & Tan Gan & Yingkai Li, 2023. "Managing Persuasion Robustly: The Optimality of Quota Rules," Papers 2310.10024, arXiv.org.
    4. repec:cup:judgdm:v:15:y:2020:i:5:p:863-880 is not listed on IDEAS
    5. Yingkai Li & Jonathan Libgober, 2023. "Implementing Evidence Acquisition: Time Dependence in Contracts for Advice," Papers 2310.19147, arXiv.org, revised Sep 2024.
    6. Andrew Gelman & Jessica Hullman & Christopher Wlezien & George Elliott Morris, 2020. "Information, incentives, and goals in election forecasts," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 15(5), pages 863-880, September.
    7. Deb, Rahul & Mitchell, Matthew & Pai, Mallesh M., 2022. "(Bad) reputation in relational contracting," Theoretical Economics, Econometric Society, vol. 17(2), May.
    8. Lukyanov, Georgy, 2023. "Reputation for competence in a cheap-talk setting," Research in Economics, Elsevier, vol. 77(3), pages 285-294.
    9. Rajiv Vohra & Francisco Espinosa & Debraj Ray, 2021. "A Principal-Agent Relationship with No Advantage to Commitment," Working Papers 2021-003, Brown University, Department of Economics.
    10. Rahul Deb & Matthew Mitchell & Mallesh Pai, 2019. "Our distrust is very expensive," Working Papers tecipa-632, University of Toronto, Department of Economics.
    11. Dell’Era, Michele, 2020. "Talking to influence," Economics Letters, Elsevier, vol. 192(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Pavesi, Filippo & Scotti, Massimo, 2022. "Good lies," European Economic Review, Elsevier, vol. 141(C).
      • Filippo Pavesi & Massimo Scotti, 2019. "Good Lies," Working Paper Series 39, Economics Discipline Group, UTS Business School, University of Technology, Sydney.
    2. Arve, Malin & Zwart, Gijsbert, 2023. "Optimal procurement and investment in new technologies under uncertainty," Journal of Economic Dynamics and Control, Elsevier, vol. 147(C).
    3. Dirk Bergemann & Alessandro Pavan, 2015. "Introduction to JET Symposium Issue on "Dynamic Contracts and Mechanism Design"," Cowles Foundation Discussion Papers 2016, Cowles Foundation for Research in Economics, Yale University.
    4. Irene Valsecchi, 2013. "The expert problem: a survey," Economics of Governance, Springer, vol. 14(4), pages 303-331, November.
    5. Bergemann, Dirk & Pavan, Alessandro, 2015. "Introduction to Symposium on Dynamic Contracts and Mechanism Design," Journal of Economic Theory, Elsevier, vol. 159(PB), pages 679-701.
    6. Deb, Rahul & Said, Maher, 2015. "Dynamic screening with limited commitment," Journal of Economic Theory, Elsevier, vol. 159(PB), pages 891-928.
    7. Boleslavsky, Raphael & Lewis, Tracy R., 2016. "Evolving influence: Mitigating extreme conflicts of interest in advisory relationships," Games and Economic Behavior, Elsevier, vol. 98(C), pages 110-134.
    8. Boaz Zik, 2023. "Efficient sequential screening with informational externalities," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 75(2), pages 567-590, February.
    9. Aleksei Smirnov & Egor Starkov, 2019. "Timing of predictions in dynamic cheap talk: experts vs. quacks," ECON - Working Papers 334, Department of Economics - University of Zurich.
    10. Jean Tirole, 2016. "From Bottom of the Barrel to Cream of the Crop: Sequential Screening With Positive Selection," Econometrica, Econometric Society, vol. 84(4), pages 1291-1343, July.
    11. Terstiege, Stefan, 2016. "Gathering imperfect information before signing a contract," Games and Economic Behavior, Elsevier, vol. 97(C), pages 70-87.
    12. Dirk Bergemann & Juuso Välimäki, 2019. "Dynamic Mechanism Design: An Introduction," Journal of Economic Literature, American Economic Association, vol. 57(2), pages 235-274, June.
    13. Hao Li & Xianwen Shi, 2017. "Discriminatory Information Disclosure," American Economic Review, American Economic Association, vol. 107(11), pages 3363-3385, November.
    14. Liu, Yaozhou Franklin & Sanyal, Amal, 2012. "When second opinions hurt: A model of expert advice under career concerns," Journal of Economic Behavior & Organization, Elsevier, vol. 84(1), pages 1-16.
    15. Ottaviani, Marco & Sorensen, Peter Norman, 2006. "The strategy of professional forecasting," Journal of Financial Economics, Elsevier, vol. 81(2), pages 441-466, August.
    16. Peter Sorensen & Marco Ottaviani, 2000. "Herd Behavior and Investment: Comment," American Economic Review, American Economic Association, vol. 90(3), pages 695-704, June.
    17. Irene Valsecchi, 2013. "Non-uniqueness of equilibrium action profiles with equal size in one-shot cheap-talk games," Theory and Decision, Springer, vol. 74(1), pages 31-53, January.
    18. Krähmer, Daniel & Strausz, Roland, 2015. "Ex post information rents in sequential screening," Games and Economic Behavior, Elsevier, vol. 90(C), pages 257-273.
    19. Lu, Jingfeng & Wang, Zijia, 2021. "Optimal selling mechanisms with buyer price search," Journal of Economic Theory, Elsevier, vol. 196(C).
    20. Lim, Wooyoung, 2014. "Communication in bargaining over decision rights," Games and Economic Behavior, Elsevier, vol. 85(C), pages 159-179.

    More about this item

    Keywords

    dynamic mechanism design; mechanism design without transfers; forecasting; learning; election predictions.;
    All these keywords.

    JEL classification:

    • D82 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Asymmetric and Private Information; Mechanism Design
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • D86 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Economics of Contract Law

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tor:tecipa:tecipa-578. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: RePEc Maintainer (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.