IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v89y2024i3d10.1007_s11336-024-09956-7.html
   My bibliography  Save this article

Proof of Reliability Convergence to 1 at Rate of Spearman–Brown Formula for Random Test Forms and Irrespective of Item Pool Dimensionality

Author

Listed:
  • Jules L. Ellis

    (Open University of The Netherlands
    Radboud University Nijmegen)

  • Klaas Sijtsma

    (Tilburg University)

Abstract

It is shown that the psychometric test reliability, based on any true-score model with randomly sampled items and uncorrelated errors, converges to 1 as the test length goes to infinity, with probability 1, assuming some general regularity conditions. The asymptotic rate of convergence is given by the Spearman–Brown formula, and for this it is not needed that the items are parallel, or latent unidimensional, or even finite dimensional. Simulations with the 2-parameter logistic item response theory model reveal that the reliability of short multidimensional tests can be positively biased, meaning that applying the Spearman–Brown formula in these cases would lead to overprediction of the reliability that results from lengthening a test. However, test constructors of short tests generally aim for short tests that measure just one attribute, so that the bias problem may have little practical relevance. For short unidimensional tests under the 2-parameter logistic model reliability is almost unbiased, meaning that application of the Spearman–Brown formula in these cases of greater practical utility leads to predictions that are approximately unbiased.

Suggested Citation

  • Jules L. Ellis & Klaas Sijtsma, 2024. "Proof of Reliability Convergence to 1 at Rate of Spearman–Brown Formula for Random Test Forms and Irrespective of Item Pool Dimensionality," Psychometrika, Springer;The Psychometric Society, vol. 89(3), pages 774-795, September.
    • Handle: RePEc:spr:psycho:v:89:y:2024:i:3:d:10.1007_s11336-024-09956-7
    DOI: 10.1007/s11336-024-09956-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-024-09956-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-024-09956-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Louis Guttman, 1945. "A basis for analyzing test-retest reliability," Psychometrika, Springer;The Psychometric Society, vol. 10(4), pages 255-282, December.
    2. Klaas Sijtsma & Julius M. Pfadt, 2021. "Part II: On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha: Discussing Lower Bounds and Correlated Errors," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 843-860, December.
    3. J. Woodward & George Joe, 1973. "Maximizing the coefficient of generalizability in multi-facet decision studies," Psychometrika, Springer;The Psychometric Society, vol. 38(2), pages 173-181, June.
    4. Goldine Gleser & Lee Cronbach & Nageswari Rajaratnam, 1965. "Generalizability of scores influenced by multiple sources of variance," Psychometrika, Springer;The Psychometric Society, vol. 30(4), pages 395-418, December.
    5. Paul Holland & Machteld Hoskens, 2003. "Classical Test Theory as a first-order Item Response Theory: Application to true-score prediction from a possibly nonparallel test," Psychometrika, Springer;The Psychometric Society, vol. 68(1), pages 123-149, March.
    6. P. Sanders & T. Theunissen & S. Baas, 1991. "Maximizing the coefficient of generalizability under the constraint of limited resources," Psychometrika, Springer;The Psychometric Society, vol. 56(1), pages 87-96, March.
    7. Klaas Sijtsma, 2009. "On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha," Psychometrika, Springer;The Psychometric Society, vol. 74(1), pages 107-120, March.
    8. Matthijs Warrens, 2015. "Some Relationships Between Cronbach’s Alpha and the Spearman-Brown Formula," Journal of Classification, Springer;The Classification Society, vol. 32(1), pages 127-137, April.
    9. Paul Jackson & Christian Agunwamba, 1977. "Lower bounds for the reliability of the total score on a test composed of non-homogeneous items: I: Algebraic lower bounds," Psychometrika, Springer;The Psychometric Society, vol. 42(4), pages 567-578, December.
    10. P. Sanders & T. Theunissen & S. Baas, 1989. "Minimizing the number of observations: A generalization of the spearman-brown formula," Psychometrika, Springer;The Psychometric Society, vol. 54(4), pages 587-598, September.
    11. John Hunter, 1968. "Probabilistic foundations for coefficients of generalizability," Psychometrika, Springer;The Psychometric Society, vol. 33(1), pages 1-18, March.
    12. Piet Sanders, 1992. "Alternative solutions for optimization problems in generalizability theory," Psychometrika, Springer;The Psychometric Society, vol. 57(3), pages 351-356, September.
    13. Jules L. Ellis, 2021. "A Test Can Have Multiple Reliabilities," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 869-876, December.
    14. William Revelle & Richard Zinbarg, 2009. "Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma," Psychometrika, Springer;The Psychometric Society, vol. 74(1), pages 145-154, March.
    15. Walk, Harro, 2008. "A universal strong law of large numbers for conditional expectations via nearest neighbors," Journal of Multivariate Analysis, Elsevier, vol. 99(6), pages 1035-1050, July.
    16. Jules Ellis & Brian Junker, 1997. "Tail-measurability in monotone latent variable models," Psychometrika, Springer;The Psychometric Society, vol. 62(4), pages 495-523, December.
    17. J. Berge & Frits Zegers, 1978. "A series of lower bounds to the reliability of a test," Psychometrika, Springer;The Psychometric Society, vol. 43(4), pages 575-579, December.
    18. Nageswari Rajaratnam & Lee Cronbach & Goldine Gleser, 1965. "Generalizability of stratified-parallel tests," Psychometrika, Springer;The Psychometric Society, vol. 30(1), pages 39-56, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eunseong Cho, 2021. "Neither Cronbach’s Alpha nor McDonald’s Omega: A Commentary on Sijtsma and Pfadt," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 877-886, December.
    2. David J. Hessen, 2017. "Lower Bounds to the Reliabilities of Factor Score Estimators," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 648-659, September.
    3. Tyler Hunt & Peter Bentler, 2015. "Quantile Lower Bounds to Reliability Based on Locally Optimal Splits," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 182-195, March.
    4. Klaas Sijtsma & Julius M. Pfadt, 2021. "Rejoinder: The Future of Reliability," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 887-892, December.
    5. Klaas Sijtsma & Julius M. Pfadt, 2021. "Part II: On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha: Discussing Lower Bounds and Correlated Errors," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 843-860, December.
    6. Markus Pauly & Maria Umlauft & Ali Ünlü, 2018. "Resampling-Based Inference Methods for Comparing Two Coefficients Alpha," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 203-222, March.
    7. Zhengguo Gu & Wilco H. M. Emons & Klaas Sijtsma, 2021. "Estimating Difference-Score Reliability in Pretest–Posttest Settings," Journal of Educational and Behavioral Statistics, , vol. 46(5), pages 592-610, October.
    8. Anne-Catherine Guio & David Gordon & Eric Marlier & Hector Najera & Marco Pomati, 2018. "Towards an EU measure of child deprivation," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 11(3), pages 835-860, June.
    9. Jules L. Ellis, 2021. "A Test Can Have Multiple Reliabilities," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 869-876, December.
    10. William Revelle & Richard Zinbarg, 2009. "Coefficients Alpha, Beta, Omega, and the glb: Comments on Sijtsma," Psychometrika, Springer;The Psychometric Society, vol. 74(1), pages 145-154, March.
    11. Carmen León-Mantero & José Carlos Casas-Rosal & Alexander Maz-Machado & Miguel E Villarraga Rico, 2020. "Analysis of attitudinal components towards statistics among students from different academic degrees," PLOS ONE, Public Library of Science, vol. 15(1), pages 1-13, January.
    12. Brian K Miller & Kay M Nicols & Silvia Clark & Alison Daniels & Whitney Grant, 2018. "Meta-analysis of coefficient alpha for scores on the Narcissistic Personality Inventory," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-16, December.
    13. Klaas Sijtsma & Ivo Molenaar, 1987. "Reliability of test scores in nonparametric item response theory," Psychometrika, Springer;The Psychometric Society, vol. 52(1), pages 79-97, March.
    14. Érika Martins Silva Ramos & Cecilia Jakobsson Bergstad, 2021. "The Psychology of Sharing: Multigroup Analysis among Users and Non-Users of Carsharing," Sustainability, MDPI, vol. 13(12), pages 1-17, June.
    15. Dorothy Watson & Bertrand Maitre, 2015. "Is Fuel Poverty in Ireland a Distinct Type of Deprivation?," The Economic and Social Review, Economic and Social Studies, vol. 46(2), pages 267-291.
    16. Peter M. Bentler, 2021. "Alpha, FACTT, and Beyond," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 861-868, December.
    17. Mary F. Zhang & Julie Selwyn, 2020. "The Subjective Well-Being of Children and Young People in out of Home Care: Psychometric Analyses of the “Your Life, your Care” Survey," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 13(5), pages 1549-1572, October.
    18. Wang, Selena & De Boeck, Paul, 2020. "When high reliability does not signal reliable detection of experimental effects," OSF Preprints gz8pw, Center for Open Science.
    19. P. Sanders & T. Theunissen & S. Baas, 1991. "Maximizing the coefficient of generalizability under the constraint of limited resources," Psychometrika, Springer;The Psychometric Society, vol. 56(1), pages 87-96, March.
    20. Cristina Wildermuth & Carlos A. Mello e Souza & Timothy Kozitza, 2017. "Circles of Ethics: The Impact of Proximity on Moral Reasoning," Journal of Business Ethics, Springer, vol. 140(1), pages 17-42, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:89:y:2024:i:3:d:10.1007_s11336-024-09956-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.