IDEAS home Printed from https://ideas.repec.org/a/gam/jmathe/v12y2024i8p1164-d1374797.html
   My bibliography  Save this article

Multivariate Structural Equation Modeling Techniques for Estimating Reliability, Measurement Error, and Subscale Viability When Using Both Composite and Subscale Scores in Practice

Author

Listed:
  • Walter Peter Vispoel

    (Department of Psychological and Quantitative Foundations, University of Iowa, Iowa City, IA 52242, USA)

  • Hyeryung Lee

    (Department of Psychological and Quantitative Foundations, University of Iowa, Iowa City, IA 52242, USA)

  • Tingting Chen

    (Department of Psychological and Quantitative Foundations, University of Iowa, Iowa City, IA 52242, USA)

Abstract

We illustrate how structural equation models (SEMs) can be used to assess the reliability and generalizability of composite and subscale scores, proportions of multiple sources of measurement error, and subscale added value within multivariate designs using data from a popular inventory measuring hierarchically structured personality traits. We compare these techniques between standard SEMs representing congeneric relations between indicators and underlying factors versus SEM-based generalizability theory (GT) designs with simplified essential tau-equivalent constraints. Results strongly emphasized the importance of accounting for multiple sources of measurement error in both contexts and revealed that, in most but not all instances, congeneric designs yielded higher score accuracy, lower proportions of measurement error, greater average subscale score viability, stronger model fits, and differing magnitudes of disattenuated subscale intercorrelations. Extending the congeneric analyses to the item level further highlighted consistent weaknesses in the psychometric properties of negatively versus positively keyed items. Collectively, these findings demonstrate the practical value and advantages of applying GT-based principles to congeneric SEMs that are much more commonly encountered in the research literature and more directly linked to the specific measures being analyzed. We also provide prophecy formulas to estimate reliability and generalizability coefficients, proportions of individual sources of measurement error, and subscale added-value indices for changes made to measurement procedures and offer guidelines and examples for running all illustrated analyses using the lavaan (Version 0.6-17) and semTools (Version 0.5-6) packages in R. The methods described for the analyzed designs are applicable to any objectively or subjectively scored assessments for which both composite and subcomponent scores are reported.

Suggested Citation

  • Walter Peter Vispoel & Hyeryung Lee & Tingting Chen, 2024. "Multivariate Structural Equation Modeling Techniques for Estimating Reliability, Measurement Error, and Subscale Viability When Using Both Composite and Subscale Scores in Practice," Mathematics, MDPI, vol. 12(8), pages 1-25, April.
    • Handle: RePEc:gam:jmathe:v:12:y:2024:i:8:p:1164-:d:1374797
    as

    Download full text from publisher

    File URL: https://www.mdpi.com/2227-7390/12/8/1164/pdf
    Download Restriction: no
    File URL: https://www.mdpi.com/2227-7390/12/8/1164/
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Lee Cronbach, 1951. "Coefficient alpha and the internal structure of tests," Psychometrika, Springer;The Psychometric Society, vol. 16(3), pages 297-334, September.
    2. Goldine Gleser & Lee Cronbach & Nageswari Rajaratnam, 1965. "Generalizability of scores influenced by multiple sources of variance," Psychometrika, Springer;The Psychometric Society, vol. 30(4), pages 395-418, December.
    3. Shelby Haberman & Sandip Sinharay, 2010. "Reporting of Subscores Using Multidimensional Item Response Theory," Psychometrika, Springer;The Psychometric Society, vol. 75(2), pages 209-227, June.
    4. Rosseel, Yves, 2012. "lavaan: An R Package for Structural Equation Modeling," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i02).
    5. P. Bentler, 1968. "Alpha-maximized factor analysis (alphamax): Its relation to alpha and canonical factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 33(3), pages 335-345, September.
    6. Nageswari Rajaratnam & Lee Cronbach & Goldine Gleser, 1965. "Generalizability of stratified-parallel tests," Psychometrika, Springer;The Psychometric Society, vol. 30(1), pages 39-56, March.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Md. Mominur Rahman & Bilkis Akhter, 2021. "The impact of investment in human capital on bank performance: evidence from Bangladesh," Future Business Journal, Springer, vol. 7(1), pages 1-13, December.
    2. Andreea-Ionela Puiu & Anca Monica Ardeleanu & Camelia Cojocaru & Anca Bratu, 2021. "Exploring the Effect of Status Quo, Innovativeness, and Involvement Tendencies on Luxury Fashion Innovations: The Mediation Role of Status Consumption," Mathematics, MDPI, vol. 9(9), pages 1-18, May.
    3. Allen, Jaime & Muñoz, Juan Carlos & Ortúzar, Juan de Dios, 2019. "On evasion behaviour in public transport: Dissatisfaction or contagion?," Transportation Research Part A: Policy and Practice, Elsevier, vol. 130(C), pages 626-651.
    4. Sai-fu Fung & Esther Oi-wah Chow & Chau-kiu Cheung, 2020. "Development and Evaluation of the Psychometric Properties of a Brief Wisdom Development Scale," IJERPH, MDPI, vol. 17(8), pages 1-14, April.
    5. Leonie Kuen & Fiona Schürmann & Daniel Westmattelmann & Sophie Hartwig & Shay Tzafrir & Gerhard Schewe, 2023. "Trust transfer effects and associated risks in telemedicine adoption," Electronic Markets, Springer;IIM University of St. Gallen, vol. 33(1), pages 1-22, December.
    6. Vitor Ciampolini & Fernando Santos & Ricardo Teixeira Quinaud & Martin Camiré & Maurício de Oliveira Migliano & Juarez Vieira do Nascimento & Michel Milistetd, 2021. "Cross-Cultural Adaptation and Psychometric Properties of the Portuguese Coaching Life Skills in Sport Questionnaire," SAGE Open, , vol. 11(2), pages 21582440211, June.
    7. Fred van Raaij, W. & Riitsalu, Leonore & Põder, Kaire, 2023. "Direct and indirect effects of self-control and future time perspective on financial well-being," Journal of Economic Psychology, Elsevier, vol. 99(C).
    8. John E. Hunter & Gerald M. Gillmore, 1974. "A Memory Search Model of Reliability," Sociological Methods & Research, , vol. 2(3), pages 281-311, February.
    9. Silvia Mariela Méndez-Prado & Vanessa Rodriguez & Kevin Peralta-Rizzo & Patricia Everaert & Martin Valcke, 2023. "An Assessment Tool to Identify the Financial Literacy Level of Financial Education Programs Participants’ Executed by Ecuadorian Financial Institutions," Sustainability, MDPI, vol. 15(2), pages 1-24, January.
    10. Sebastian Kurten & David Winant & Kathleen Beullens, 2021. "Mothers Matter: Using Regression Tree Algorithms to Predict Adolescents’ Sharing of Drunk References on Social Media," IJERPH, MDPI, vol. 18(21), pages 1-16, October.
    11. Ke-Hai Yuan & Zhiyong Zhang & Lijuan Wang, 2024. "Signal-to-Noise Ratio in Estimating and Testing the Mediation Effect: Structural Equation Modeling versus Path Analysis with Weighted Composites," Psychometrika, Springer;The Psychometric Society, vol. 89(3), pages 974-1006, September.
    12. P. Bentler, 1986. "Structural modeling and psychometrika: An historical perspective on growth and achievements," Psychometrika, Springer;The Psychometric Society, vol. 51(1), pages 35-51, March.
    13. Érika Martins Silva Ramos & Cecilia Jakobsson Bergstad, 2021. "The Psychology of Sharing: Multigroup Analysis among Users and Non-Users of Carsharing," Sustainability, MDPI, vol. 13(12), pages 1-17, June.
    14. Serwaah, Priscilla & Shneor, Rotem & Nyarko, Samuel Anokye & Nielsen, Kristian Roed, 2024. "Explaining gender differences in crowdfunding contribution intentions," Technology in Society, Elsevier, vol. 76(C).
    15. Daniel Schulze & Kathrin Heinitz & Timo Lorenz, 2018. "Comparative organizational research starts with sound measurement: Validity and invariance of Turker’s corporate social responsibility scale in five cross-cultural samples," PLOS ONE, Public Library of Science, vol. 13(11), pages 1-15, November.
    16. Shenghai Dai & Dubravka Svetina & Xiaolin Wang, 2017. "Reporting Subscores Using R: A Software Review," Journal of Educational and Behavioral Statistics, , vol. 42(5), pages 617-638, October.
    17. Essie Sutton & Kimberly A. Schonert-Reichl & Amery D. Wu & Molly Stewart Lawlor, 2018. "Evaluating the Reliability and Validity of the Self-Compassion Scale Short Form Adapted for Children Ages 8–12," Child Indicators Research, Springer;The International Society of Child Indicators (ISCI), vol. 11(4), pages 1217-1236, August.
    18. Lindsey W. Vilca & Evelyn L. Chambi-Mamani & Emely D. Quispe-Kana & Mónica Hernández-López & Tomás Caycho-Rodríguez, 2022. "Functioning of the EROS-R Scale in a Clinical Sample of Psychiatric Patients: New Psychometric Evidence from the Classical Test Theory and the Item Response Theory," IJERPH, MDPI, vol. 19(16), pages 1-14, August.
    19. Nambury Raju, 1979. "Note on two generalizations of coefficient alpha," Psychometrika, Springer;The Psychometric Society, vol. 44(3), pages 347-349, September.
    20. Peter M. Bentler, 2021. "Alpha, FACTT, and Beyond," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 861-868, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:gam:jmathe:v:12:y:2024:i:8:p:1164-:d:1374797. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: MDPI Indexing Manager (email available below). General contact details of provider: https://www.mdpi.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.