IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v89y2024i4d10.1007_s11336-024-09988-z.html
   My bibliography  Save this article

Practical Implications of Sum Scores Being Psychometrics’ Greatest Accomplishment

Author

Listed:
  • Daniel McNeish

    (Arizona State University)

Abstract

This paper reflects on some practical implications of the excellent treatment of sum scoring and classical test theory (CTT) by Sijtsma et al. (Psychometrika 89(1):84–117, 2024). I have no major disagreements about the content they present and found it to be an informative clarification of the properties and possible extensions of CTT. In this paper, I focus on whether sum scores—despite their mathematical justification—are positioned to improve psychometric practice in empirical studies in psychology, education, and adjacent areas. First, I summarize recent reviews of psychometric practice in empirical studies, subsequent calls for greater psychometric transparency and validity, and how sum scores may or may not be positioned to adhere to such calls. Second, I consider limitations of sum scores for prediction, especially in the presence of common features like ordinal or Likert response scales, multidimensional constructs, and moderated or heterogeneous associations. Third, I review previous research outlining potential limitations of using sum scores as outcomes in subsequent analyses where rank ordering is not always sufficient to successfully characterize group differences or change over time. Fourth, I cover potential challenges for providing validity evidence for whether sum scores represent a single construct, particularly if one wishes to maintain minimal CTT assumptions. I conclude with thoughts about whether sum scores—even if mathematically justified—are positioned to improve psychometric practice in empirical studies.

Suggested Citation

  • Daniel McNeish, 2024. "Practical Implications of Sum Scores Being Psychometrics’ Greatest Accomplishment," Psychometrika, Springer;The Psychometric Society, vol. 89(4), pages 1148-1169, December.
    • Handle: RePEc:spr:psycho:v:89:y:2024:i:4:d:10.1007_s11336-024-09988-z
    DOI: 10.1007/s11336-024-09988-z
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-024-09988-z
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-024-09988-z?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Robert Tsutakawa & Jane Johnson, 1990. "The effect of uncertainty of item parameter estimation on ability estimates," Psychometrika, Springer;The Psychometric Society, vol. 55(2), pages 371-390, June.
    2. L. Ark, 2005. "Stochastic Ordering Of the Latent Trait by the Sum Score Under Various Polytomous IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 70(2), pages 283-304, June.
    3. Klaas Sijtsma, 2012. "Future of Psychometrics: Ask What Psychometrics Can Do for Psychology," Psychometrika, Springer;The Psychometric Society, vol. 77(1), pages 4-20, January.
    4. André Beauducel, 2007. "In Spite of Indeterminacy Many Common Factor Score Estimates Yield an Identical Reproduced Covariance Matrix," Psychometrika, Springer;The Psychometric Society, vol. 72(3), pages 437-441, September.
    5. Joshua D. Angrist, 2004. "American Education Research Changes Tack," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 20(2), pages 198-212, Summer.
    6. Stephen Sireci, 1998. "The Construct of Content Validity," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 45(1), pages 83-117, November.
    7. Robert Zwitser & Gunter Maris, 2016. "Ordering Individuals with Sum Scores: The Introduction of the Nonparametric Rasch Model," Psychometrika, Springer;The Psychometric Society, vol. 81(1), pages 39-59, March.
    8. Robert J. Zwitser & Gunter Maris, 2016. "Ordering Individuals with Sum Scores: The Introduction of the Nonparametric Rasch Model," Psychometrika, Springer;The Psychometric Society, vol. 81(1), pages 39-59, March.
    9. Bas Hemker & Klaas Sijtsma & Ivo Molenaar & Brian Junker, 1996. "Polytomous IRT models and monotone likelihood ratio of the total score," Psychometrika, Springer;The Psychometric Society, vol. 61(4), pages 679-693, December.
    10. Sacha Epskamp & Mijke Rhemtulla & Denny Borsboom, 2017. "Generalized Network Psychometrics: Combining Network and Latent Variable Models," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 904-927, December.
    11. Klaas Sijtsma, 2009. "On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha," Psychometrika, Springer;The Psychometric Society, vol. 74(1), pages 107-120, March.
    12. Wicherts, Jelte M. & Veldkamp, Coosje Lisabet Sterre & Augusteijn, Hilde & Bakker, Marjan & van Aert, Robbie Cornelis Maria & van Assen, Marcel A. L. M., 2016. "Degrees of freedom in planning, running, analyzing, and reporting psychological studies A checklist to avoid p-hacking," OSF Preprints umq8d, Center for Open Science.
    13. Denny Borsboom, 2006. "The attack of the psychometricians," Psychometrika, Springer;The Psychometric Society, vol. 71(3), pages 425-440, September.
    14. Klaas Sijtsma & Jules L. Ellis & Denny Borsboom, 2024. "Recognize the Value of the Sum Score, Psychometrics’ Greatest Accomplishment," Psychometrika, Springer;The Psychometric Society, vol. 89(1), pages 84-117, March.
    15. Bas Hemker & Klaas Sijtsma & Ivo Molenaar & Brian Junker, 1997. "Stochastic ordering using the latent trait and the sum score in polytomous IRT models," Psychometrika, Springer;The Psychometric Society, vol. 62(3), pages 331-347, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Klaas Sijtsma & Jules L. Ellis & Denny Borsboom, 2024. "Rejoinder to McNeish and Mislevy: What Does Psychological Measurement Require?," Psychometrika, Springer;The Psychometric Society, vol. 89(4), pages 1175-1185, December.
    2. Robert J. Mislevy, 2024. "Are Sum Scores a Great Accomplishment of Psychometrics or Intuitive Test Theory?," Psychometrika, Springer;The Psychometric Society, vol. 89(4), pages 1170-1174, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Klaas Sijtsma & Jules L. Ellis & Denny Borsboom, 2024. "Rejoinder to McNeish and Mislevy: What Does Psychological Measurement Require?," Psychometrika, Springer;The Psychometric Society, vol. 89(4), pages 1175-1185, December.
    2. Klaas Sijtsma & Jules L. Ellis & Denny Borsboom, 2024. "Recognize the Value of the Sum Score, Psychometrics’ Greatest Accomplishment," Psychometrika, Springer;The Psychometric Society, vol. 89(1), pages 84-117, March.
    3. Jesper Tijmstra & Maria Bolsinova, 2019. "Bayes Factors for Evaluating Latent Monotonicity in Polytomous Item Response Theory Models," Psychometrika, Springer;The Psychometric Society, vol. 84(3), pages 846-869, September.
    4. Daphna Harel & Russell J. Steele, 2018. "An Information Matrix Test for the Collapsing of Categories Under the Partial Credit Model," Journal of Educational and Behavioral Statistics, , vol. 43(6), pages 721-750, December.
    5. Wim Linden, 1998. "Stochastic order in dichotomous item response models for fixed, adaptive, and multidimensional tests," Psychometrika, Springer;The Psychometric Society, vol. 63(3), pages 211-226, September.
    6. L. Ark & Wicher Bergsma, 2010. "A Note on Stochastic Ordering of the Latent Trait Using the Sum of Polytomous Item Scores," Psychometrika, Springer;The Psychometric Society, vol. 75(2), pages 272-279, June.
    7. Rudy Ligtvoet, 2012. "An Isotonic Partial Credit Model for Ordering Subjects on the Basis of Their Sum Scores," Psychometrika, Springer;The Psychometric Society, vol. 77(3), pages 479-494, July.
    8. Rudy Ligtvoet, 2022. "Incomplete Tests of Conditional Association for the Assessment of Model Assumptions," Psychometrika, Springer;The Psychometric Society, vol. 87(4), pages 1214-1237, December.
    9. Jesper Tijmstra & Herbert Hoijtink & Klaas Sijtsma, 2015. "Evaluating Manifest Monotonicity Using Bayes Factors," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 880-896, December.
    10. van der Ark, L. Andries, 2012. "New Developments in Mokken Scale Analysis in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i05).
    11. Ligtvoet, R., 2015. "A test for using the sum score to obtain a stochastic ordering of subjects," Journal of Multivariate Analysis, Elsevier, vol. 133(C), pages 136-139.
    12. Klaas Sijtsma & Julius M. Pfadt, 2021. "Rejoinder: The Future of Reliability," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 887-892, December.
    13. J. Straat & L. Ark & Klaas Sijtsma, 2013. "Comparing Optimization Algorithms for Item Selection in Mokken Scale Analysis," Journal of Classification, Springer;The Classification Society, vol. 30(1), pages 75-99, April.
    14. Klaas Sijtsma, 2016. "Playing with Data—Or How to Discourage Questionable Research Practices and Stimulate Researchers to Do Things Right," Psychometrika, Springer;The Psychometric Society, vol. 81(1), pages 1-15, March.
    15. Bas Hemker & L. Andries van der Ark & Klaas Sijtsma, 2001. "On measurement properties of continuation ratio models," Psychometrika, Springer;The Psychometric Society, vol. 66(4), pages 487-506, December.
    16. Tendeiro, Jorge N. & Meijer, Rob R. & Niessen, A. Susan M., 2016. "PerFit: An R Package for Person-Fit Analysis in IRT," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 74(i05).
    17. van der Ark, L.A., 1999. "A reference card for the relationships between IRT models for ordered polytomous items and some relevant properties," WORC Paper 99.10.02, Tilburg University, Work and Organization Research Centre.
    18. repec:jss:jstsof:20:i11 is not listed on IDEAS
    19. Pascal Jordan & Meike C Shedden-Mora & Bernd Löwe, 2017. "Psychometric analysis of the Generalized Anxiety Disorder scale (GAD-7) in primary care using modern item response theory," PLOS ONE, Public Library of Science, vol. 12(8), pages 1-14, August.
    20. Ulrich Schroeders & Oliver Wilhelm & Gabriel Olaru, 2016. "Meta-Heuristics in Short Scale Construction: Ant Colony Optimization and Genetic Algorithm," PLOS ONE, Public Library of Science, vol. 11(11), pages 1-19, November.
    21. Martin Schlotter & Guido Schwerdt & Ludger Woessmann, 2011. "Econometric methods for causal evaluation of education policies and practices: a non-technical guide," Education Economics, Taylor & Francis Journals, vol. 19(2), pages 109-137.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:89:y:2024:i:4:d:10.1007_s11336-024-09988-z. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.