IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v30y2014i2p344-356.html
   My bibliography  Save this article

Combining multiple probability predictions using a simple logit model

Author

Listed:
  • Satopää, Ville A.
  • Baron, Jonathan
  • Foster, Dean P.
  • Mellers, Barbara A.
  • Tetlock, Philip E.
  • Ungar, Lyle H.

Abstract

This paper begins by presenting a simple model of the way in which experts estimate probabilities. The model is then used to construct a likelihood-based aggregation formula for combining multiple probability forecasts. The resulting aggregator has a simple analytical form that depends on a single, easily-interpretable parameter. This makes it computationally simple, attractive for further development, and robust against overfitting. Based on a large-scale dataset in which over 1300 experts tried to predict 69 geopolitical events, our aggregator is found to be superior to several widely-used aggregation algorithms.

Suggested Citation

  • Satopää, Ville A. & Baron, Jonathan & Foster, Dean P. & Mellers, Barbara A. & Tetlock, Philip E. & Ungar, Lyle H., 2014. "Combining multiple probability predictions using a simple logit model," International Journal of Forecasting, Elsevier, vol. 30(2), pages 344-356.
    • Handle: RePEc:eee:intfor:v:30:y:2014:i:2:p:344-356
    DOI: 10.1016/j.ijforecast.2013.09.009
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207013001635
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2013.09.009?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Roopesh Ranjan & Tilmann Gneiting, 2010. "Combining probability forecasts," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 72(1), pages 71-91, January.
    2. Clemen, Robert T., 1989. "Combining forecasts: A review and annotated bibliography," International Journal of Forecasting, Elsevier, vol. 5(4), pages 559-583.
    3. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    4. Robert F. Bordley, 1982. "A Multiplicative Formula for Aggregating Probability Assessments," Management Science, INFORMS, vol. 28(10), pages 1137-1148, October.
    5. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    6. Wallsten, Thomas S. & Diederich, Adele, 2001. "Understanding pooled subjective probability estimates," Mathematical Social Sciences, Elsevier, vol. 41(1), pages 1-18, January.
    7. Graefe, Andreas & Armstrong, J. Scott & Jones, Randall J. & Cuzán, Alfred G., 2014. "Combining forecasts: An application to elections," International Journal of Forecasting, Elsevier, vol. 30(1), pages 43-54.
    8. McKenzie, Craig R.M. & Liersch, Michael J. & Yaniv, Ilan, 2008. "Overconfidence in interval estimates: What does expertise buy you?," Organizational Behavior and Human Decision Processes, Elsevier, vol. 107(2), pages 179-191, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Tao Lin & Yiling Chen, 2022. "Sample Complexity of Forecast Aggregation," Papers 2207.13126, arXiv.org, revised Oct 2023.
    2. Yakov Babichenko & Dan Garber, 2021. "Learning Optimal Forecast Aggregation in Partial Evidence Environments," Mathematics of Operations Research, INFORMS, vol. 46(2), pages 628-641, May.
    3. repec:cup:judgdm:v:14:y:2019:i:4:p:395-411 is not listed on IDEAS
    4. Jonathan Baron & Barbara A. Mellers & Philip E. Tetlock & Eric Stone & Lyle H. Ungar, 2014. "Two Reasons to Make Aggregated Probability Forecasts More Extreme," Decision Analysis, INFORMS, vol. 11(2), pages 133-145, June.
    5. repec:cup:judgdm:v:14:y:2019:i:2:p:135-147 is not listed on IDEAS
    6. Hanea, Anca & Wilkinson, David Peter & McBride, Marissa & Lyon, Aidan & van Ravenzwaaij, Don & Singleton Thorn, Felix & Gray, Charles T. & Mandel, David R. & Willcox, Aaron & Gould, Elliot, 2021. "Mathematically aggregating experts' predictions of possible futures," MetaArXiv rxmh7, Center for Open Science.
    7. repec:cup:judgdm:v:13:y:2018:i:2:p:185-201 is not listed on IDEAS
    8. Jason Dana & Pavel Atanasov & Philip Tetlock & Barbara Mellers, 2019. "Are markets more accurate than polls? The surprising informational value of “just askingâ€," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(2), pages 135-147, March.
    9. Satopää, Ville A., 2021. "Improving the wisdom of crowds with analysis of variance of predictions of related outcomes," International Journal of Forecasting, Elsevier, vol. 37(4), pages 1728-1747.
    10. Jared A. Beekman & Ronald F. A. Woodaman & Dennis M. Buede, 2020. "A Review of Probabilistic Opinion Pooling Algorithms with Application to Insider Threat Detection," Decision Analysis, INFORMS, vol. 17(1), pages 39-55, March.
    11. Ying Han & David Budescu, 2019. "A universal method for evaluating the quality of aggregators," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(4), pages 395-411, July.
    12. Satopää, Ville A. & Salikhov, Marat & Tetlock, Philip E. & Mellers, Barbara, 2023. "Decomposing the effects of crowd-wisdom aggregators: The bias–information–noise (BIN) model," International Journal of Forecasting, Elsevier, vol. 39(1), pages 470-485.
    13. Zanin, Luca, 2020. "Combining multiple probability predictions in the presence of class imbalance to discriminate between potential bad and good borrowers in the peer-to-peer lending market," Journal of Behavioral and Experimental Finance, Elsevier, vol. 25(C).
    14. Karvetski, Christopher W. & Meinel, Carolyn & Maxwell, Daniel T. & Lu, Yunzi & Mellers, Barbara A. & Tetlock, Philip E., 2022. "What do forecasting rationales reveal about thinking patterns of top geopolitical forecasters?," International Journal of Forecasting, Elsevier, vol. 38(2), pages 688-704.
    15. Ville A. Satopää & Robin Pemantle & Lyle H. Ungar, 2016. "Modeling Probability Forecasts via Information Diversity," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(516), pages 1623-1633, October.
    16. Jens Witkowski & Rupert Freeman & Jennifer Wortman Vaughan & David M. Pennock & Andreas Krause, 2023. "Incentive-Compatible Forecasting Competitions," Management Science, INFORMS, vol. 69(3), pages 1354-1374, March.
    17. Niklas Valentin Lehmann, 2023. "Forecasting skill of a crowd-prediction platform: A comparison of exchange rate forecasts," Papers 2312.09081, arXiv.org.
    18. Ville A. Satopää & Marat Salikhov & Philip E. Tetlock & Barbara Mellers, 2021. "Bias, Information, Noise: The BIN Model of Forecasting," Management Science, INFORMS, vol. 67(12), pages 7599-7618, December.
    19. Edgar C. Merkle & Robert Hartman, 2018. "Weighted Brier score decompositions for topically heterogenous forecasting tournaments," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(2), pages 185-201, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Petropoulos, Fotios & Apiletti, Daniele & Assimakopoulos, Vassilios & Babai, Mohamed Zied & Barrow, Devon K. & Ben Taieb, Souhaib & Bergmeir, Christoph & Bessa, Ricardo J. & Bijak, Jakub & Boylan, Joh, 2022. "Forecasting: theory and practice," International Journal of Forecasting, Elsevier, vol. 38(3), pages 705-871.
      • Fotios Petropoulos & Daniele Apiletti & Vassilios Assimakopoulos & Mohamed Zied Babai & Devon K. Barrow & Souhaib Ben Taieb & Christoph Bergmeir & Ricardo J. Bessa & Jakub Bijak & John E. Boylan & Jet, 2020. "Forecasting: theory and practice," Papers 2012.03854, arXiv.org, revised Jan 2022.
    2. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    3. Wang, Xiaoqian & Hyndman, Rob J. & Li, Feng & Kang, Yanfei, 2023. "Forecast combinations: An over 50-year review," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1518-1547.
    4. Martin, Gael M. & Loaiza-Maya, Rubén & Maneesoonthorn, Worapree & Frazier, David T. & Ramírez-Hassan, Andrés, 2022. "Optimal probabilistic forecasts: When do they work?," International Journal of Forecasting, Elsevier, vol. 38(1), pages 384-406.
    5. Tommaso Proietti & Martyna Marczak & Gianluigi Mazzi, 2017. "Euromind‐ D : A Density Estimate of Monthly Gross Domestic Product for the Euro Area," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(3), pages 683-703, April.
    6. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    7. Yang, Dazhi & van der Meer, Dennis, 2021. "Post-processing in solar forecasting: Ten overarching thinking tools," Renewable and Sustainable Energy Reviews, Elsevier, vol. 140(C).
    8. Ruben Loaiza‐Maya & Gael M. Martin & David T. Frazier, 2021. "Focused Bayesian prediction," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 36(5), pages 517-543, August.
    9. Enrique Moral-Benito, 2015. "Model Averaging In Economics: An Overview," Journal of Economic Surveys, Wiley Blackwell, vol. 29(1), pages 46-75, February.
    10. Bjørnland, Hilde C. & Gerdrup, Karsten & Jore, Anne Sofie & Smith, Christie & Thorsrud, Leif Anders, 2011. "Weights and pools for a Norwegian density combination," The North American Journal of Economics and Finance, Elsevier, vol. 22(1), pages 61-76, January.
    11. Tony Chernis & Taylor Webley, 2022. "Nowcasting Canadian GDP with Density Combinations," Discussion Papers 2022-12, Bank of Canada.
    12. Stephen Hora & Erim Kardeş, 2015. "Calibration, sharpness and the weighting of experts in a linear opinion pool," Annals of Operations Research, Springer, vol. 229(1), pages 429-450, June.
    13. Ray, Evan L. & Brooks, Logan C. & Bien, Jacob & Biggerstaff, Matthew & Bosse, Nikos I. & Bracher, Johannes & Cramer, Estee Y. & Funk, Sebastian & Gerding, Aaron & Johansson, Michael A. & Rumack, Aaron, 2023. "Comparing trained and untrained probabilistic ensemble forecasts of COVID-19 cases and deaths in the United States," International Journal of Forecasting, Elsevier, vol. 39(3), pages 1366-1383.
    14. Baggio, Roberta & Muzy, Jean-François, 2024. "Improving probabilistic wind speed forecasting using M-Rice distribution and spatial data integration," Applied Energy, Elsevier, vol. 360(C).
    15. Jared A. Beekman & Ronald F. A. Woodaman & Dennis M. Buede, 2020. "A Review of Probabilistic Opinion Pooling Algorithms with Application to Insider Threat Detection," Decision Analysis, INFORMS, vol. 17(1), pages 39-55, March.
    16. Taylor, James W. & Taylor, Kathryn S., 2023. "Combining probabilistic forecasts of COVID-19 mortality in the United States," European Journal of Operational Research, Elsevier, vol. 304(1), pages 25-41.
    17. von der Gracht, Heiko A. & Hommel, Ulrich & Prokesch, Tobias & Wohlenberg, Holger, 2016. "Testing weighting approaches for forecasting in a Group Wisdom Support System environment," Journal of Business Research, Elsevier, vol. 69(10), pages 4081-4094.
    18. Reason Lesego Machete, 2011. "Early Warning with Calibrated and Sharper Probabilistic Forecasts," Papers 1112.6390, arXiv.org, revised Jan 2012.
    19. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    20. Gael M. Martin & David T. Frazier & Ruben Loaiza-Maya & Florian Huber & Gary Koop & John Maheu & Didier Nibbering & Anastasios Panagiotelis, 2023. "Bayesian Forecasting in the 21st Century: A Modern Review," Monash Econometrics and Business Statistics Working Papers 1/23, Monash University, Department of Econometrics and Business Statistics.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:30:y:2014:i:2:p:344-356. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.