IDEAS home Printed from https://ideas.repec.org/a/eee/intfor/v33y2017i4p817-832.html
   My bibliography  Save this article

A neglected dimension of good forecasting judgment: The questions we choose also matter

Author

Listed:
  • Merkle, Edgar C.
  • Steyvers, Mark
  • Mellers, Barbara
  • Tetlock, Philip E.

Abstract

Forecasters are typically evaluated via proper scoring rules such as the Brier score. These scoring rules use only the reported forecasts for assessment, neglecting related variables such as the specific questions that a person chose to forecast. In this paper, we study whether information related to question selection influences our estimates of forecaster ability. In other words, do good and bad forecasters tend to select questions in different ways? If so, can we capitalize on these selections when estimating forecaster ability? We address these questions by extending a recently-developed psychometric model of forecasts to include question selection data. We compare the extended psychometric model to a simpler model, studying its unidimensionality assumption and highlighting the unique information that it can provide. We find that the model can make use of the fact that good forecasters tend to select more questions than bad forecasters, and we conclude that question selection data can be beneficial above and beyond reported forecasts. As a side benefit, the resulting model can potentially provide unique incentives for forecaster participation.

Suggested Citation

  • Merkle, Edgar C. & Steyvers, Mark & Mellers, Barbara & Tetlock, Philip E., 2017. "A neglected dimension of good forecasting judgment: The questions we choose also matter," International Journal of Forecasting, Elsevier, vol. 33(4), pages 817-832.
    • Handle: RePEc:eee:intfor:v:33:y:2017:i:4:p:817-832
    DOI: 10.1016/j.ijforecast.2017.04.002
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0169207017300377
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.ijforecast.2017.04.002?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Yu-Wei Chang & Rung-Ching Tsai & Nan-Jung Hsu, 2014. "A Speeded Item Response Model: Leave the Harder till Later," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 255-274, April.
    2. Jinming Zhang & William Stout, 1999. "The theoretical detect index of dimensionality and its application to approximate simple structure," Psychometrika, Springer;The Psychometric Society, vol. 64(2), pages 213-249, June.
    3. Fumiko Samejima, 1973. "Homogeneous case of the continuous response model," Psychometrika, Springer;The Psychometric Society, vol. 38(2), pages 203-219, June.
    4. Rosseel, Yves, 2012. "lavaan: An R Package for Structural Equation Modeling," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i02).
    5. C. O'Muircheartaigh & I. Moustaki, 1999. "Symmetric pattern models: a latent variable approach to item non‐response in attitude scales," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 162(2), pages 177-194.
    6. Li Cai, 2010. "High-dimensional Exploratory Item Factor Analysis by A Metropolis–Hastings Robbins–Monro Algorithm," Psychometrika, Springer;The Psychometric Society, vol. 75(1), pages 33-57, March.
    7. Arthur Carvalho, 2016. "An Overview of Applications of Proper Scoring Rules," Decision Analysis, INFORMS, vol. 13(4), pages 223-242, December.
    8. Jinming Zhang, 2007. "Conditional Covariance Theory and Detect for Polytomous Items," Psychometrika, Springer;The Psychometric Society, vol. 72(1), pages 69-91, March.
    9. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    10. Denwood, Matthew J., 2016. "runjags: An R Package Providing Interface Utilities, Model Templates, Parallel Computing Methods and Additional Distributions for MCMC Models in JAGS," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 71(i09).
    11. David Budescu & Yuanchao Bo, 2015. "Analyzing Test-Taking Behavior: Decision Theory Meets Psychometric Theory," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 1105-1122, December.
    12. Carel Peeters, 2012. "Rotational Uniqueness Conditions Under Oblique Factor Correlation Metric," Psychometrika, Springer;The Psychometric Society, vol. 77(2), pages 288-292, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:cup:judgdm:v:13:y:2018:i:2:p:185-201 is not listed on IDEAS
    2. Edgar C. Merkle & Robert Hartman, 2018. "Weighted Brier score decompositions for topically heterogenous forecasting tournaments," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(2), pages 185-201, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Norde, Henk & Voorneveld, Mark, 2019. "Feasible best-response correspondences and quadratic scoring rules," SSE Working Paper Series in Economics 2019:2, Stockholm School of Economics.
    2. César Merino-Soto & Arturo Juárez-García & Guillermo Salinas Escudero & Filiberto Toledano-Toledano, 2022. "Parametric and Nonparametric Analysis of the Internal Structure of the Psychosocial Work Processes Questionnaire (PROPSIT) as Applied to Workers," IJERPH, MDPI, vol. 19(13), pages 1-23, June.
    3. Andrew Grant & David Johnstone & Oh Kang Kwon, 2019. "A Probability Scoring Rule for Simultaneous Events," Decision Analysis, INFORMS, vol. 16(4), pages 301-313, December.
    4. J. Straat & L. Ark & Klaas Sijtsma, 2013. "Comparing Optimization Algorithms for Item Selection in Mokken Scale Analysis," Journal of Classification, Springer;The Classification Society, vol. 30(1), pages 75-99, April.
    5. Edgar C. Merkle & Daniel Furr & Sophia Rabe-Hesketh, 2019. "Bayesian Comparison of Latent Variable Models: Conditional Versus Marginal Likelihoods," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 802-829, September.
    6. Carroll, Gabriel, 2019. "Robust incentives for information acquisition," Journal of Economic Theory, Elsevier, vol. 181(C), pages 382-420.
    7. Wojciech Łukasz Dragan & Andrzej Śliwerski & Monika Folkierska-Żukowska, 2022. "New data on the validity of the Fazio Laterality Inventory," PLOS ONE, Public Library of Science, vol. 17(1), pages 1-15, January.
    8. Jules Ellis, 2014. "An Inequality for Correlations in Unidimensional Monotone Latent Variable Models for Binary Variables," Psychometrika, Springer;The Psychometric Society, vol. 79(2), pages 303-316, April.
    9. repec:cup:judgdm:v:13:y:2018:i:2:p:185-201 is not listed on IDEAS
    10. Tino Werner, 2022. "Elicitability of Instance and Object Ranking," Decision Analysis, INFORMS, vol. 19(2), pages 123-140, June.
    11. Jules L. Ellis & Klaas Sijtsma, 2023. "A Test to Distinguish Monotone Homogeneity from Monotone Multifactor Models," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 387-412, June.
    12. Robert L. Winkler & Yael Grushka-Cockayne & Kenneth C. Lichtendahl Jr. & Victor Richmond R. Jose, 2019. "Probability Forecasts and Their Combination: A Research Perspective," Decision Analysis, INFORMS, vol. 16(4), pages 239-260, December.
    13. Edgar C. Merkle & Robert Hartman, 2018. "Weighted Brier score decompositions for topically heterogenous forecasting tournaments," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 13(2), pages 185-201, March.
    14. Zachary J. Smith & J. Eric Bickel, 2020. "Additive Scoring Rules for Discrete Sample Spaces," Decision Analysis, INFORMS, vol. 17(2), pages 115-133, June.
    15. Nicholas J. Rockwood, 2021. "Efficient Likelihood Estimation of Generalized Structural Equation Models with a Mix of Normal and Nonnormal Responses," Psychometrika, Springer;The Psychometric Society, vol. 86(2), pages 642-667, June.
    16. Jinming Zhang, 2013. "A Procedure for Dimensionality Analyses of Response Data from Various Test Designs," Psychometrika, Springer;The Psychometric Society, vol. 78(1), pages 37-58, January.
    17. Sonia Nawrocka & Hans De Witte & Margherita Pasini & Margherita Brondino, 2023. "A Person-Centered Approach to Job Insecurity: Is There a Reciprocal Relationship between the Quantitative and Qualitative Dimensions of Job Insecurity?," IJERPH, MDPI, vol. 20(7), pages 1-27, March.
    18. Md. Mominur Rahman & Bilkis Akhter, 2021. "The impact of investment in human capital on bank performance: evidence from Bangladesh," Future Business Journal, Springer, vol. 7(1), pages 1-13, December.
    19. Azar, Pablo D. & Micali, Silvio, 2018. "Computational principal agent problems," Theoretical Economics, Econometric Society, vol. 13(2), May.
    20. Masashi Soga & Kevin J. Gaston & Yuichi Yamaura & Kiyo Kurisu & Keisuke Hanaki, 2016. "Both Direct and Vicarious Experiences of Nature Affect Children’s Willingness to Conserve Biodiversity," IJERPH, MDPI, vol. 13(6), pages 1-12, May.
    21. César Merino-Soto & Gina Chávez-Ventura & Verónica López-Fernández & Guillermo M. Chans & Filiberto Toledano-Toledano, 2022. "Learning Self-Regulation Questionnaire (SRQ-L): Psychometric and Measurement Invariance Evidence in Peruvian Undergraduate Students," Sustainability, MDPI, vol. 14(18), pages 1-17, September.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:intfor:v:33:y:2017:i:4:p:817-832. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/ijforecast .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.