IDEAS home Printed from https://ideas.repec.org/a/inm/ordeca/v11y2014i2p133-145.html
   My bibliography  Save this article

Two Reasons to Make Aggregated Probability Forecasts More Extreme

Author

Listed:
  • Jonathan Baron

    (Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

  • Barbara A. Mellers

    (Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

  • Philip E. Tetlock

    (Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

  • Eric Stone

    (Department of Psychology, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

  • Lyle H. Ungar

    (Department of Computer and Information Science, University of Pennsylvania, Philadelphia, Pennsylvania 19104)

Abstract

When aggregating the probability estimates of many individuals to form a consensus probability estimate of an uncertain future event, it is common to combine them using a simple weighted average. Such aggregated probabilities correspond more closely to the real world if they are transformed by pushing them closer to 0 or 1. We explain the need for such transformations in terms of two distorting factors: The first factor is the compression of the probability scale at the two ends, so that random error tends to push the average probability toward 0.5. This effect does not occur for the median forecast, or, arguably, for the mean of the log odds of individual forecasts. The second factor---which affects mean, median, and mean of log odds---is the result of forecasters taking into account their individual ignorance of the total body of information available. Individual confidence in the direction of a probability judgment (high/low) thus fails to take into account the wisdom of crowds that results from combining different evidence available to different judges. We show that the same transformation function can approximately eliminate both distorting effects with different parameters for the mean and the median. And we show how, in principle, use of the median can help distinguish the two effects.

Suggested Citation

  • Jonathan Baron & Barbara A. Mellers & Philip E. Tetlock & Eric Stone & Lyle H. Ungar, 2014. "Two Reasons to Make Aggregated Probability Forecasts More Extreme," Decision Analysis, INFORMS, vol. 11(2), pages 133-145, June.
    • Handle: RePEc:inm:ordeca:v:11:y:2014:i:2:p:133-145
    DOI: 10.1287/deca.2014.0293
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/deca.2014.0293
    Download Restriction: no
    File URL: https://libkey.io/10.1287/deca.2014.0293?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. David J. Johnstone & Victor Richmond R. Jose & Robert L. Winkler, 2011. "Tailored Scoring Rules for Probabilities," Decision Analysis, INFORMS, vol. 8(4), pages 256-268, December.
    2. A. Drăgulescu & V.M. Yakovenko, 2001. "Evidence for the exponential distribution of income in the USA," The European Physical Journal B: Condensed Matter and Complex Systems, Springer;EDP Sciences, vol. 20(4), pages 585-589, April.
    3. Satopää, Ville A. & Baron, Jonathan & Foster, Dean P. & Mellers, Barbara A. & Tetlock, Philip E. & Ungar, Lyle H., 2014. "Combining multiple probability predictions using a simple logit model," International Journal of Forecasting, Elsevier, vol. 30(2), pages 344-356.
    4. Wallsten, Thomas S. & Diederich, Adele, 2001. "Understanding pooled subjective probability estimates," Mathematical Social Sciences, Elsevier, vol. 41(1), pages 1-18, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hernández-Ramírez, E. & del Castillo-Mussot, M. & Hernández-Casildo, J., 2021. "World per capita gross domestic product measured nominally and across countries with purchasing power parity: Stretched exponential or Boltzmann–Gibbs distribution?," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 568(C).
    2. Schulz, Jan & Mayerhoffer, Daniel M., 2021. "A network approach to consumption," BERG Working Paper Series 173, Bamberg University, Bamberg Economic Research Group.
    3. Ricardo Lopez-Ruiz, 2012. "Complex Systems with Trivial Dynamics," Papers 1210.6481, arXiv.org.
    4. Fabio Clementi & Mauro Gallegati, 2005. "Pareto's Law of Income Distribution: Evidence for Grermany, the United Kingdom, and the United States," Microeconomics 0505006, University Library of Munich, Germany.
    5. Tao, Yong, 2015. "Universal laws of human society’s income distribution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 435(C), pages 89-94.
    6. Kaldasch, Joachim, 2012. "Evolutionary model of the personal income distribution," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 391(22), pages 5628-5642.
    7. Kayal, Suchandan & Kumar, Somesh, 2013. "Estimation of the Shannon’s entropy of several shifted exponential populations," Statistics & Probability Letters, Elsevier, vol. 83(4), pages 1127-1135.
    8. Valérie Berenger & Cuauhtémoc Calderón Villarreal & Franck Celestini, 2009. "Modelling the Distribution of Multidimensional Poverty Scores: Evidence from Mexico," Estudios Económicos, El Colegio de México, Centro de Estudios Económicos, vol. 24(1), pages 3-34.
    9. Gregor Semieniuk & Victor M. Yakovenko, 2020. "Historical Evolution of Global Inequality in Carbon Emissions and Footprints versus Redistributive Scenarios," Papers 2004.00111, arXiv.org.
    10. repec:cup:judgdm:v:14:y:2019:i:2:p:135-147 is not listed on IDEAS
    11. Satopää, Ville A. & Baron, Jonathan & Foster, Dean P. & Mellers, Barbara A. & Tetlock, Philip E. & Ungar, Lyle H., 2014. "Combining multiple probability predictions using a simple logit model," International Journal of Forecasting, Elsevier, vol. 30(2), pages 344-356.
    12. Johannes Müller-Trede & Shoham Choshen-Hillel & Meir Barneron & Ilan Yaniv, 2018. "The Wisdom of Crowds in Matters of Taste," Management Science, INFORMS, vol. 64(4), pages 1779-1803, April.
    13. Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose & Robert L. Winkler, 2017. "Quantile Evaluation, Sensitivity to Bracketing, and Sharing Business Payoffs," Operations Research, INFORMS, vol. 65(3), pages 712-728, June.
    14. Chakrabarti, Anindya S., 2011. "An almost linear stochastic map related to the particle system models of social sciences," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 390(23), pages 4370-4378.
    15. Makowsky, Michael D., 2011. "Religion, clubs, and emergent social divides," Journal of Economic Behavior & Organization, Elsevier, vol. 80(1), pages 74-87.
    16. Istvan Gere & Szabolcs Kelemen & Geza Toth & Tamas Biro & Zoltan Neda, 2021. "Wealth distribution in modern societies: collected data and a master equation approach," Papers 2104.04134, arXiv.org.
    17. Saemi Park & David V. Budescu, 2015. "Aggregating multiple probability intervals to improve calibration," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 10(2), pages 130-143, March.
    18. Braun, Dieter, 2006. "Nonequilibrium thermodynamics of wealth condensation," Physica A: Statistical Mechanics and its Applications, Elsevier, vol. 369(2), pages 714-722.
    19. Willis, Geoff, 2011. "Why money trickles up – wealth & income distributions," MPRA Paper 30851, University Library of Munich, Germany.
    20. Li, Bin Grace & McAndrews, James & Wang, Zhu, 2020. "Two-sided market, R&D, and payments system evolution," Journal of Monetary Economics, Elsevier, vol. 115(C), pages 180-199.
    21. Markus P. A. Schneider, 2018. "Revisiting the thermal and superthermal two-class distribution of incomes: A critical perspective," Papers 1804.06341, arXiv.org.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ordeca:v:11:y:2014:i:2:p:133-145. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.