IDEAS home Printed from https://ideas.repec.org/p/pra/mprapa/43324.html
   My bibliography  Save this paper

Mechanism Design for the truthful elicitation of costly probabilistic estimates in Distributed Information Systems

Author

Listed:
  • Papakonstantinou, A.
  • Rogers, A
  • Gerding, E. H.
  • Jennings, N. R.

Abstract

This paper reports on the design of a novel two-stage mechanism, based on strictly proper scoring rules, that allows a centre to acquire a costly forecast of a future event (such as a meteorological phenomenon) or a probabilistic estimate of a specific parameter (such as the quality of an expected service), with a specified minimum precision, from one or more agents. In the first stage, the centre elicits the agents' true costs and identifies the agent that can provide an estimate of the specified precision at the lowest cost. Then, in the second stage, the centre uses an appropriately scaled strictly proper scoring rule to incentivise this agent to generate the estimate with the required precision, and to truthfully report it. In particular, this is the first mechanism that can be applied to settings in which the centre has no knowledge about the actual costs involved in the generation an agents' estimates and also has no external means of evaluating the quality and accuracy of the estimates it receives. En route to this mechanism, we first consider a setting in which any single agent can provide an estimate of the required precision, and the centre can evaluate this estimate by comparing it with the outcome which is observed at a later stage. This mechanism is then extended, so that it can be applied in a setting where the agents' different capabilities are reflected in the maximum precision of the estimates that they can provide, potentially requiring the centre to select multiple agents and combine their individual results in order to obtain an estimate of the required precision. For all three mechanisms (the original and the two extensions), we prove their economic properties (i.e. incentive compatibility and individual rationality) and then perform a number of numerical simulations. For the single agent mechanism we compare the quadratic, spherical and logarithmic scoring rules with a parametric family of scoring rules. We show that although the logarithmic scoring rule minimises both the mean and variance of the centre's total payments, using this rule means that an agent may face an unbounded penalty if it provides an estimate of extremely poor quality. We show that this is not the case for the parametric family, and thus, we suggest that the parametric scoring rule is the best candidate in our setting. Furthermore, we show that the 'multiple agent' extension describes a family of possible approaches to select agents in the first stage of our mechanism, and we show empirically and prove analytically that there is one approach that dominates all others. Finally, we compare our mechanism to the peer prediction mechanism introduced by Miller et al. (2007) [29] and show that the centre's total expected payment is the same in both mechanisms (and is equal to total expected payment in the case that the estimates can be compared to the actual outcome), while the variance in these payments is significantly reduced within our mechanism.

Suggested Citation

  • Papakonstantinou, A. & Rogers, A & Gerding, E. H. & Jennings, N. R., 2010. "Mechanism Design for the truthful elicitation of costly probabilistic estimates in Distributed Information Systems," MPRA Paper 43324, University Library of Munich, Germany.
  • Handle: RePEc:pra:mprapa:43324
    as

    Download full text from publisher

    File URL: https://mpra.ub.uni-muenchen.de/43324/1/MPRA_paper_43324.pdf
    File Function: original version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    2. Krishna, Vijay, 2009. "Auction Theory," Elsevier Monographs, Elsevier, edition 2, number 9780123745071.
    3. Yokoo, Makoto & Sakurai, Yuko & Matsubara, Shigeo, 2004. "The effect of false-name bids in combinatorial auctions: new fraud in internet auctions," Games and Economic Behavior, Elsevier, vol. 46(1), pages 174-188, January.
    4. Justin Wolfers & Eric Zitzewitz, 2004. "Prediction Markets," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 107-126, Spring.
    5. Joyce E. Berg & Thomas A. Rietz, 2003. "Prediction Markets as Decision Support Systems," Information Systems Frontiers, Springer, vol. 5(1), pages 79-93, January.
    6. Edward Clarke, 1971. "Multipart pricing of public goods," Public Choice, Springer, vol. 11(1), pages 17-33, September.
    7. Chong Ju Choi & Carla C. J. M. Millar & Caroline Y. L. Wong, 2005. "Knowledge and the State," Palgrave Macmillan Books, in: Knowledge Entanglements, chapter 0, pages 19-38, Palgrave Macmillan.
    8. Jehiel, Philippe & Moldovanu, Benny, 2001. "Efficient Design with Interdependent Valuations," Econometrica, Econometric Society, vol. 69(5), pages 1237-1259, September.
    9. Robert Day & Paul Milgrom, 2008. "Core-selecting package auctions," International Journal of Game Theory, Springer;Game Theory Society, vol. 36(3), pages 393-407, March.
    10. William Vickrey, 1961. "Counterspeculation, Auctions, And Competitive Sealed Tenders," Journal of Finance, American Finance Association, vol. 16(1), pages 8-37, March.
    11. Grossman, Sanford J & Hart, Oliver D, 1983. "An Analysis of the Principal-Agent Problem," Econometrica, Econometric Society, vol. 51(1), pages 7-45, January.
    12. Claudio Mezzetti, 2004. "Mechanism Design with Interdependent Valuations: Efficiency," Econometrica, Econometric Society, vol. 72(5), pages 1617-1626, September.
    13. repec:reg:rpubli:259 is not listed on IDEAS
    14. Claudio Mezzetti, 2007. "Mechanism Design with Interdependent Valuations: Surplus Extraction," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 31(3), pages 473-488, June.
    15. Mas-Colell, Andreu & Whinston, Michael D. & Green, Jerry R., 1995. "Microeconomic Theory," OUP Catalogue, Oxford University Press, number 9780195102680.
    16. Robin Hanson, 2003. "Combinatorial Information Market Design," Information Systems Frontiers, Springer, vol. 5(1), pages 107-119, January.
    17. Robin Hanson, 2007. "Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 3-15, February.
    18. Papakonstantinou, A. & Rogers, A. & Gerding, E. H & Jennings, N. R., 2010. "Mechanism Design for eliciting probabilistic estimates from multiple suppliers with unknown costs and limited precision," MPRA Paper 43323, University Library of Munich, Germany.
    19. Papakonstantinou, A. & Rogers, A. & Gerding, E. H. & Jennings, N. R, 2008. "A Truthful Two-Stage Mechanism for Eliciting Probabilistic Estimates with Unknown Costs," MPRA Paper 43320, University Library of Munich, Germany.
    20. Reinhard Selten, 1998. "Axiomatic Characterization of the Quadratic Scoring Rule," Experimental Economics, Springer;Economic Science Association, vol. 1(1), pages 43-61, June.
    21. Groves, Theodore, 1973. "Incentives in Teams," Econometrica, Econometric Society, vol. 41(4), pages 617-631, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Papakonstantinou, A. & Bogetoft, P., 2013. "Crowd-sourcing with uncertain quality - an auction approach," MPRA Paper 46055, University Library of Munich, Germany.
    2. Papakonstantinou, Athanasios & Bogetoft, Peter, 2017. "Multi-dimensional procurement auction under uncertain and asymmetric information," European Journal of Operational Research, Elsevier, vol. 258(3), pages 1171-1180.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Miller, Nolan H. & Pratt, John W. & Zeckhauser, Richard J. & Johnson, Scott, 2007. "Mechanism design with multidimensional, continuous types and interdependent valuations," Journal of Economic Theory, Elsevier, vol. 136(1), pages 476-496, September.
    2. Liu, Heng, 2018. "Efficient dynamic mechanisms in environments with interdependent valuations: the role of contingent transfers," Theoretical Economics, Econometric Society, vol. 13(2), May.
    3. Swaprava Nath & Onno Zoeter & Y. Narahari & Christopher Dance, 2015. "Dynamic mechanism design with interdependent valuations," Review of Economic Design, Springer;Society for Economic Design, vol. 19(3), pages 211-228, September.
    4. Delacrétaz, David & Loertscher, Simon & Marx, Leslie M. & Wilkening, Tom, 2019. "Two-sided allocation problems, decomposability, and the impossibility of efficient trade," Journal of Economic Theory, Elsevier, vol. 179(C), pages 416-454.
    5. He, Wei & Li, Jiangtao, 2016. "Efficient dynamic mechanisms with interdependent valuations," Games and Economic Behavior, Elsevier, vol. 97(C), pages 166-173.
    6. Mikuláš Gangur & Miroslav Plevný, 2014. "Tools for Consumer Rights Protection in the Prediction of Electronic Virtual Market and Technological Changes," The AMFITEATRU ECONOMIC journal, Academy of Economic Studies - Bucharest, Romania, vol. 16(36), pages 578-578, May.
    7. Philippe Jehiel & Laurent Lamy, 2018. "A Mechanism Design Approach to the Tiebout Hypothesis," Journal of Political Economy, University of Chicago Press, vol. 126(2), pages 735-760.
    8. Ramchurn, Sarvapali D. & Dash, Rajdeep K. & Jennings, Nicholas R. & Giovannucci, Andrea & Rodriguez-Aguilar, Juan A. & Mezzetti, Claudio, 2008. "Trust-Based Mechanisms for Robust and Efficient Task Allocation in the Presence of Execution Uncertainty," Economic Research Papers 269891, University of Warwick - Department of Economics.
    9. Lorentziadis, Panos L., 2016. "Optimal bidding in auctions from a game theory perspective," European Journal of Operational Research, Elsevier, vol. 248(2), pages 347-371.
    10. Matsushima, Hitoshi & Noda, Shunya, 2023. "Mechanism design with general ex-ante investments," Journal of Mathematical Economics, Elsevier, vol. 106(C).
    11. Papakonstantinou, A. & Bogetoft, P., 2013. "Crowd-sourcing with uncertain quality - an auction approach," MPRA Paper 44236, University Library of Munich, Germany.
    12. Hitoshi Matsushima & Shunya Noda, 2019. "Mechanism Design with General Ex-Ante Investments (Revised version of F415 )," CARF F-Series CARF-F-464, Center for Advanced Research in Finance, Faculty of Economics, The University of Tokyo.
    13. Johnson, Scott & Miller, Nolan & Pratt, John W. & Zeckhauser, Richard, 2003. "Efficient Design with Multidimensional, Continuous Types, and Interdependent Valuations," Working Paper Series rwp03-020, Harvard University, John F. Kennedy School of Government.
    14. Eric Maskin, 2004. "The Unity of Auction Theory: Paul Milgrom's Masterclass," Economics Working Papers 0044, Institute for Advanced Study, School of Social Science.
    15. William H. Sandholm, 2005. "Negative Externalities and Evolutionary Implementation," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 72(3), pages 885-915.
    16. Soumyakanti Chakraborty & Anup K. Sen & Amitava Bagchi, 2015. "Addressing the valuation problem in multi-round combinatorial auctions," Information Systems Frontiers, Springer, vol. 17(5), pages 1145-1160, October.
    17. Hiroki Saitoh & Shigehiro Serizawa, 2008. "Vickrey allocation rule with income effect," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 35(2), pages 391-401, May.
    18. Takashi Kunimoto & Cuiling Zhang, 2021. "On incentive compatible, individually rational public good provision mechanisms," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 57(2), pages 431-468, August.
    19. Yoon, Kiho, 2008. "The participatory Vickrey-Clarke-Groves mechanism," Journal of Mathematical Economics, Elsevier, vol. 44(3-4), pages 324-336, February.
    20. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.

    More about this item

    Keywords

    Multiagent systems; Scoring rules; Auction theory; Mechanism design;
    All these keywords.

    JEL classification:

    • D81 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Criteria for Decision-Making under Risk and Uncertainty
    • D44 - Microeconomics - - Market Structure, Pricing, and Design - - - Auctions
    • D82 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Asymmetric and Private Information; Mechanism Design

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:pra:mprapa:43324. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joachim Winter (email available below). General contact details of provider: https://edirc.repec.org/data/vfmunde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.