IDEAS home Printed from https://ideas.repec.org/a/inm/ordeca/v18y2021i3p169-184.html
   My bibliography  Save this article

Scoring Probability Forecasts by a User’s Bets Against a Market Consensus

Author

Listed:
  • David Johnstone

    (Faculty of Business, University of Wollongong, Wollongong, New South Wales 2500, Australia)

  • Stewart Jones

    (University of Sydney Business School, University of Sydney, Sydney 2006, Australia)

  • Oliver Jones

    (Faculty of Mathematics, East Los Angeles College, Monterey Park, California 91754)

  • Steve Tulig

    (Faculty of Business, University of Wollongong, Wollongong, New South Wales 2500, Australia)

Abstract

The purpose of our paper is to describe a probability scoring rule that reflects the economic performance of a hypothetical investor who acts upon the probability forecasts emanating from a given model or human expert by trading against a market-clearing consensus of competing models and forecasts. The probability forecasts being compared are aggregated by an equilibrium condition into a market consensus reflecting the wisdom of the crowd. A good forecaster (model or human expert) is measured as one who allows the user to bet profitably against the market consensus. By asking forecasts to beat the market, forecasters are discouraged from herding and motivated to obtain better information than rival forecasters. We illustrate and prove that each trader’s personal incentive to hedge or fudge disappears when the number of forecasts in the market is sufficiently large. Our score exhibits the forecaster’s ability to assist economically profitable action and reveals how the user’s profits depend strongly on the accuracy of the forecasts and the decision rule (boldness) with which they are acted upon.

Suggested Citation

  • David Johnstone & Stewart Jones & Oliver Jones & Steve Tulig, 2021. "Scoring Probability Forecasts by a User’s Bets Against a Market Consensus," Decision Analysis, INFORMS, vol. 18(3), pages 169-184, September.
    • Handle: RePEc:inm:ordeca:v:18:y:2021:i:3:p:169-184
    DOI: 10.1287/deca.2020.0424
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/deca.2020.0424
    Download Restriction: no
    File URL: https://libkey.io/10.1287/deca.2020.0424?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Leonard Maclean & Edward Thorp & William Ziemba, 2010. "Long-term capital growth: the good and bad properties of the Kelly and fractional Kelly capital growth criteria," Quantitative Finance, Taylor & Francis Journals, vol. 10(7), pages 681-687.
    2. D. J. Johnstone, 2012. "Log-optimal economic evaluation of probability forecasts," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(3), pages 661-689, July.
    3. Victor Richmond R. Jose & Robert F. Nau & Robert L. Winkler, 2008. "Scoring Rules, Generalized Entropy, and Utility Maximization," Operations Research, INFORMS, vol. 56(5), pages 1146-1157, October.
    4. repec:cup:judgdm:v:10:y:2015:i:1:p:86-96 is not listed on IDEAS
    5. Marco Ottaviani & Peter Norman Sørensen, 2015. "Price Reaction to Information with Heterogeneous Beliefs and Wealth Effects: Underreaction, Momentum, and Reversal," American Economic Review, American Economic Association, vol. 105(1), pages 1-34, January.
    6. Andrew Grant & David Johnstone & Oh Kang Kwon, 2019. "A Probability Scoring Rule for Simultaneous Events," Decision Analysis, INFORMS, vol. 16(4), pages 301-313, December.
    7. Zachary J. Smith & J. Eric Bickel, 2020. "Additive Scoring Rules for Discrete Sample Spaces," Decision Analysis, INFORMS, vol. 17(2), pages 115-133, June.
    8. David J. Johnstone, 2007. "The Parimutuel Kelly Probability Scoring Rule," Decision Analysis, INFORMS, vol. 4(2), pages 66-75, June.
    9. Lawrence Blume & David Easley, 2006. "If You're so Smart, why Aren't You Rich? Belief Selection in Complete and Incomplete Markets," Econometrica, Econometric Society, vol. 74(4), pages 929-966, July.
    10. Alvaro Sandroni, 2000. "Do Markets Favor Agents Able to Make Accurate Predicitions?," Econometrica, Econometric Society, vol. 68(6), pages 1303-1342, November.
    11. Arthur Carvalho, 2016. "An Overview of Applications of Proper Scoring Rules," Decision Analysis, INFORMS, vol. 13(4), pages 223-242, December.
    12. Leonard Maclean & William Ziemba & Yuming Li, 2005. "Time to wealth goals in capital accumulation," Quantitative Finance, Taylor & Francis Journals, vol. 5(4), pages 343-355.
    13. Roopesh Ranjan & Tilmann Gneiting, 2010. "Combining probability forecasts," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(1), pages 71-91, January.
    14. Yuming Li, 1993. "Growth-Security Investment Strategy for Long and Short Runs," Management Science, INFORMS, vol. 39(8), pages 915-924, August.
    15. Robin Hanson, 2007. "Logarithmic Market Scoring Rules for Modular Combinatorial Information Aggregation," Journal of Prediction Markets, University of Buckingham Press, vol. 1(1), pages 3-15, February.
    16. MacLean, Leonard C. & Sanegre, Rafael & Zhao, Yonggan & Ziemba, William T., 2004. "Capital growth with security," Journal of Economic Dynamics and Control, Elsevier, vol. 28(5), pages 937-954, February.
    17. D. Marc Kilgour & Yigal Gerchak, 2004. "Elicitation of Probabilities Using Competitive Scoring Rules," Decision Analysis, INFORMS, vol. 1(2), pages 108-113, June.
    18. R. Winkler & Javier Muñoz & José Cervera & José Bernardo & Gail Blattenberger & Joseph Kadane & Dennis Lindley & Allan Murphy & Robert Oliver & David Ríos-Insua, 1996. "Scoring rules and the evaluation of probabilities," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 5(1), pages 1-60, June.
    19. Donald C. Aucamp, 1993. "On the Extensive Number of Plays to Achieve Superior Performance with the Geometric Mean Strategy," Management Science, INFORMS, vol. 39(9), pages 1163-1172, September.
    20. J. Eric Bickel, 2007. "Some Comparisons among Quadratic, Spherical, and Logarithmic Scoring Rules," Decision Analysis, INFORMS, vol. 4(2), pages 49-65, June.
    21. Stewart Jones & David Johnstone & Roy Wilson, 2017. "Predicting Corporate Bankruptcy: An Evaluation of Alternative Statistical Frameworks," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 44(1-2), pages 3-34, January.
    22. Sendhil Mullainathan & Marianne Bertrand, 2001. "Do People Mean What They Say? Implications for Subjective Survey Data," American Economic Review, American Economic Association, vol. 91(2), pages 67-72, May.
    23. Traczynski, Jeffrey, 2017. "Firm Default Prediction: A Bayesian Model-Averaging Approach," Journal of Financial and Quantitative Analysis, Cambridge University Press, vol. 52(3), pages 1211-1245, June.
    24. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    25. David J. Johnstone & Victor Richmond R. Jose & Robert L. Winkler, 2011. "Tailored Scoring Rules for Probabilities," Decision Analysis, INFORMS, vol. 8(4), pages 256-268, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    2. D. J. Johnstone, 2021. "Accounting information, disclosure, and expected utility: Do investors really abhor uncertainty?," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 48(1-2), pages 3-35, January.
    3. David J. Johnstone, 2007. "The Parimutuel Kelly Probability Scoring Rule," Decision Analysis, INFORMS, vol. 4(2), pages 66-75, June.
    4. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    5. D. J. Johnstone & S. Jones & V. R. R. Jose & M. Peat, 2013. "Measures of the economic value of probabilities of bankruptcy," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(3), pages 635-653, June.
    6. Andrew Grant & David Johnstone & Oh Kang Kwon, 2019. "A Probability Scoring Rule for Simultaneous Events," Decision Analysis, INFORMS, vol. 16(4), pages 301-313, December.
    7. Edgar C. Merkle & Mark Steyvers, 2013. "Choosing a Strictly Proper Scoring Rule," Decision Analysis, INFORMS, vol. 10(4), pages 292-304, December.
    8. David J Johnstone, 2023. "Capital budgeting and Kelly betting," Australian Journal of Management, Australian School of Business, vol. 48(3), pages 625-651, August.
    9. Zachary J. Smith & J. Eric Bickel, 2020. "Additive Scoring Rules for Discrete Sample Spaces," Decision Analysis, INFORMS, vol. 17(2), pages 115-133, June.
    10. L. Robin Keller & Ali Abbas & J. Eric Bickel & Vicki M. Bier & David V. Budescu & John C. Butler & Philippe Delquié & Kenneth C. Lichtendahl & Jason R. W. Merrick & Ahti Salo & George Wu, 2011. "From the Editors ---Probability Scoring Rules, Ambiguity, Multiattribute Terrorist Utility, and Sensitivity Analysis," Decision Analysis, INFORMS, vol. 8(4), pages 251-255, December.
    11. Aitazaz Ali Raja & Pierre Pinson & Jalal Kazempour & Sergio Grammatico, 2022. "A Market for Trading Forecasts: A Wagering Mechanism," Papers 2205.02668, arXiv.org, revised Oct 2022.
    12. Tino Werner, 2022. "Elicitability of Instance and Object Ranking," Decision Analysis, INFORMS, vol. 19(2), pages 123-140, June.
    13. Rose D. Baker & Ian G. McHale, 2013. "Optimal Betting Under Parameter Uncertainty: Improving the Kelly Criterion," Decision Analysis, INFORMS, vol. 10(3), pages 189-199, September.
    14. David J. Johnstone & Victor Richmond R. Jose & Robert L. Winkler, 2011. "Tailored Scoring Rules for Probabilities," Decision Analysis, INFORMS, vol. 8(4), pages 256-268, December.
    15. Lambert, Nicolas S. & Langford, John & Wortman Vaughan, Jennifer & Chen, Yiling & Reeves, Daniel M. & Shoham, Yoav & Pennock, David M., 2015. "An axiomatic characterization of wagering mechanisms," Journal of Economic Theory, Elsevier, vol. 156(C), pages 389-416.
    16. Raja, Aitazaz Ali & Pinson, Pierre & Kazempour, Jalal & Grammatico, Sergio, 2024. "A market for trading forecasts: A wagering mechanism," International Journal of Forecasting, Elsevier, vol. 40(1), pages 142-159.
    17. D.J. Johnstone, 2015. "Information and the Cost of Capital in a Mean-Variance Efficient Market," Journal of Business Finance & Accounting, Wiley Blackwell, vol. 42(1-2), pages 79-100, January.
    18. L. Robin Keller & Ali Abbas & Manel Baucells & Vicki M. Bier & David Budescu & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Ahti Salo & George Wu, 2010. "From the Editors..," Decision Analysis, INFORMS, vol. 7(4), pages 327-330, December.
      • L. Robin Keller & Manel Baucells & Kevin F. McCardle & Gregory S. Parnell & Ahti Salo, 2007. "From the Editors..," Decision Analysis, INFORMS, vol. 4(4), pages 173-175, December.
      • L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2008. "From the Editors..," Decision Analysis, INFORMS, vol. 5(4), pages 173-176, December.
      • L. Robin Keller & Manel Baucells & John C. Butler & Philippe Delquié & Jason R. W. Merrick & Gregory S. Parnell & Ahti Salo, 2009. "From the Editors ..," Decision Analysis, INFORMS, vol. 6(4), pages 199-201, December.
    19. Ali E. Abbas, 2009. "A Kullback-Leibler View of Linear and Log-Linear Pools," Decision Analysis, INFORMS, vol. 6(1), pages 25-37, March.
    20. David Johnstone, 2007. "Economic Darwinism: Who has the Best Probabilities?," Theory and Decision, Springer, vol. 62(1), pages 47-96, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ordeca:v:18:y:2021:i:3:p:169-184. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.