IDEAS home Printed from https://ideas.repec.org/a/wly/riskan/v34y2014i3p435-452.html
   My bibliography  Save this article

Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models

Author

Listed:
  • Mark Steyvers
  • Thomas S. Wallsten
  • Edgar C. Merkle
  • Brandon M. Turner

Abstract

We propose the use of signal detection theory (SDT) to evaluate the performance of both probabilistic forecasting systems and individual forecasters. The main advantage of SDT is that it provides a principled way to distinguish the response from system diagnosticity, which is defined as the ability to distinguish events that occur from those that do not. There are two challenges in applying SDT to probabilistic forecasts. First, the SDT model must handle judged probabilities rather than the conventional binary decisions. Second, the model must be able to operate in the presence of sparse data generated within the context of human forecasting systems. Our approach is to specify a model of how individual forecasts are generated from underlying representations and use Bayesian inference to estimate the underlying latent parameters. Given our estimate of the underlying representations, features of the classic SDT model, such as the receiver operating characteristic (ROC) curve and the area under the ROC curve (AUC), follow immediately. We show how our approach allows ROC curves and AUCs to be applied to individuals within a group of forecasters, estimated as a function of time, and extended to measure differences in forecastability across different domains. Among the advantages of this method is that it depends only on the ordinal properties of the probabilistic forecasts. We conclude with a brief discussion of how this approach might facilitate decision making.

Suggested Citation

  • Mark Steyvers & Thomas S. Wallsten & Edgar C. Merkle & Brandon M. Turner, 2014. "Evaluating Probabilistic Forecasts with Bayesian Signal Detection Models," Risk Analysis, John Wiley & Sons, vol. 34(3), pages 435-452, March.
  • Handle: RePEc:wly:riskan:v:34:y:2014:i:3:p:435-452
    DOI: 10.1111/risa.12127
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/risa.12127
    Download Restriction: no

    File URL: https://libkey.io/10.1111/risa.12127?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Levi, Keith, 1985. "A signal detection framework for the evaluation of probabilistic forecasts," Organizational Behavior and Human Decision Processes, Elsevier, vol. 36(2), pages 143-166, October.
    2. Jeffrey Rouder & Jun Lu & Dongchu Sun & Paul Speckman & Richard Morey & Moshe Naveh-Benjamin, 2007. "Signal Detection Models with Random Participant and Item Effects," Psychometrika, Springer;The Psychometric Society, vol. 72(4), pages 621-642, December.
    3. Clemon, Robert T & Winkler, Robert L, 1986. "Combining Economic Forecasts," Journal of Business & Economic Statistics, American Statistical Association, vol. 4(1), pages 39-46, January.
    4. Robert T. Clemen, 1986. "Calibration and the Aggregation of Probabilities," Management Science, INFORMS, vol. 32(3), pages 312-314, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:cup:judgdm:v:16:y:2021:i:4:p:1072-1096 is not listed on IDEAS
    2. Ying Han & David V. Budescu, 2022. "Recalibrating probabilistic forecasts to improve their accuracy," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 17(1), pages 91-123, January.
    3. David A. Comerford, 2021. "Apparent age and gender differences in survival optimism: To what extent are they a bias in the translation of beliefs onto a percentage scale?," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 16(4), pages 1072-1096, July.
    4. repec:cup:judgdm:v:17:y:2022:i:1:p:91-123 is not listed on IDEAS
    5. J. Richard Eiser & Amy Donovan & R. Stephen J. Sparks, 2015. "Risk Perceptions and Trust Following the 2010 and 2011 Icelandic Volcanic Ash Crises," Risk Analysis, John Wiley & Sons, vol. 35(2), pages 332-343, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Weyant John, 2014. "Integrated assessment of climate change: state of the literature," Journal of Benefit-Cost Analysis, De Gruyter, vol. 5(3), pages 377-409, December.
    2. Dan Zhu & Qingwei Wang & John Goddard, 2022. "A new hedging hypothesis regarding prediction interval formation in stock price forecasting," Journal of Forecasting, John Wiley & Sons, Ltd., vol. 41(4), pages 697-717, July.
    3. Lahiri, Kajal & Yang, Liu, 2013. "Forecasting Binary Outcomes," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1025-1106, Elsevier.
    4. Lahiri, Kajal & Teigland, Christie, 1987. "On the normality of probability distributions of inflation and GNP forecasts," International Journal of Forecasting, Elsevier, vol. 3(2), pages 269-279.
    5. Rossi, Barbara, 2013. "Advances in Forecasting under Instability," Handbook of Economic Forecasting, in: G. Elliott & C. Granger & A. Timmermann (ed.), Handbook of Economic Forecasting, edition 1, volume 2, chapter 0, pages 1203-1324, Elsevier.
    6. repec:lan:wpaper:470 is not listed on IDEAS
    7. Ladha, Krishna K., 1995. "Information pooling through majority-rule voting: Condorcet's jury theorem with correlated votes," Journal of Economic Behavior & Organization, Elsevier, vol. 26(3), pages 353-372, May.
    8. Jaspersen, Johannes G., 2022. "Convex combinations in judgment aggregation," European Journal of Operational Research, Elsevier, vol. 299(2), pages 780-794.
    9. Patrick Afflerbach & Christopher Dun & Henner Gimpel & Dominik Parak & Johannes Seyfried, 2021. "A Simulation-Based Approach to Understanding the Wisdom of Crowds Phenomenon in Aggregating Expert Judgment," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 63(4), pages 329-348, August.
    10. Graham, John R. & Harvey, Campbell R., 1996. "Market timing ability and volatility implied in investment newsletters' asset allocation recommendations," Journal of Financial Economics, Elsevier, vol. 42(3), pages 397-421, November.
    11. Marcello Basili & Alain Chateauneuf, 2013. "Aggregation of coherent experts opinion: a tractable extreme-outcomes consistent rule," Department of Economics University of Siena 676, Department of Economics, University of Siena.
    12. Samuels, Jon D. & Sekkel, Rodrigo M., 2017. "Model Confidence Sets and forecast combination," International Journal of Forecasting, Elsevier, vol. 33(1), pages 48-60.
    13. Robert L. Winkler & Robert T. Clemen, 2004. "Multiple Experts vs. Multiple Methods: Combining Correlation Assessments," Decision Analysis, INFORMS, vol. 1(3), pages 167-176, September.
    14. Bernstein, Daniel M. & Rudd, Michael E. & Erdfelder, Edgar & Godfrey, Ryan & Loftus, Elizabeth F., 2008. "The revelation effect for autobiographical memory : a mixture-model analysis," Papers 08-25, Sonderforschungsbreich 504.
    15. Bessler, David A & Chamberlain, Peter J, 1987. "On Bayesian composite forecasting," Omega, Elsevier, vol. 15(1), pages 43-48.
    16. Sebastian M. Blanc & Thomas Setzer, 2020. "Bias–Variance Trade-Off and Shrinkage of Weights in Forecast Combination," Management Science, INFORMS, vol. 66(12), pages 5720-5737, December.
    17. Richard P. Larrick & Jack B. Soll, 2006. "Intuitions About Combining Opinions: Misappreciation of the Averaging Principle," Management Science, INFORMS, vol. 52(1), pages 111-127, January.
    18. Pennings, Clint L.P. & van Dalen, Jan & Rook, Laurens, 2019. "Coordinating judgmental forecasting: Coping with intentional biases," Omega, Elsevier, vol. 87(C), pages 46-56.
    19. Skold, Karl Durwood, 1989. "The integration of alternative information systems: an application to the Hogs and Pigs report," ISU General Staff Papers 1989010108000010239, Iowa State University, Department of Economics.
    20. Jose, Victor Richmond R. & Winkler, Robert L., 2008. "Simple robust averages of forecasts: Some empirical results," International Journal of Forecasting, Elsevier, vol. 24(1), pages 163-169.
    21. Paola Monari & Patrizia Agati, 2001. "Fiducial inference in combining expert judgements," Statistical Methods & Applications, Springer;Società Italiana di Statistica, vol. 10(1), pages 81-97, January.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:riskan:v:34:y:2014:i:3:p:435-452. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://doi.org/10.1111/(ISSN)1539-6924 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.