IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v58y2024i1d10.1007_s11135-023-01653-4.html
   My bibliography  Save this article

Scale reliability of alternative cultural theory survey measures

Author

Listed:
  • Branden B. Johnson

    (Decision Research)

  • Brendon Swedlow

    (Northern Illinois University)

Abstract

Grid-Group Cultural Theory (CT) has attracted significant attention across the social sciences and is increasingly being used in survey research. But the reliability of CT survey scales has mostly only been assessed using Cronbach’s alpha. Unfortunately, researchers frequently misinterpret this measure and often use the wrong or incomplete statistics to assess scale reliability. We compare the scale reliability of CT and its “cultural cognition theory” (CCT) variant as operationalized in surveys. Reviewing 73 survey studies using CT and CCT measures since 1990, we find limited application of appropriate method. CT measures, which are heterogeneous across studies, are far less reliable overall. But scales of roughly equal numbers of items are about equally reliable across CT and CCT studies. We apply recommended statistical measures to our data on both CT and CCT responses from an online national U.S. panel to illustrate a better approach to assessing scale reliability. We find that relative reliability is mixed (one CCT scale is more reliable, and one less, than the specific CT scales we used). But CT measures exhibit much better model fit in confirmatory factor analysis, a prerequisite for conducting valid reliability analyses. Thus, our literature review and survey analysis show that assessing the relative reliability of these scales is more complex than simply identifying the higher Cronbach’s alpha. Better tests of scale reliability could enhance CT survey research, and social science generally.

Suggested Citation

  • Branden B. Johnson & Brendon Swedlow, 2024. "Scale reliability of alternative cultural theory survey measures," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(1), pages 527-557, February.
    • Handle: RePEc:spr:qualqt:v:58:y:2024:i:1:d:10.1007_s11135-023-01653-4
    DOI: 10.1007/s11135-023-01653-4
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11135-023-01653-4
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11135-023-01653-4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Creed Tumlison & Geoboo Song, 2019. "Cultural Values, Trust, and Benefit‐Risk Perceptions of Hydraulic Fracturing: A Comparative Analysis of Policy Elites and the General Public," Risk Analysis, John Wiley & Sons, vol. 39(3), pages 511-534, March.
    2. Dan M. Kahan & Kathleen Hall Jamieson & Asheley Landrum & Kenneth Winneg, 2017. "Culturally antagonistic memes and the Zika virus: an experimental test," Journal of Risk Research, Taylor & Francis Journals, vol. 20(1), pages 1-40, January.
    3. Aysel Tikir & Bernard Lehmann, 2011. "Climate change, theory of planned behavior and values: a structural equation model with mediation analysis," Climatic Change, Springer, vol. 104(2), pages 389-402, January.
    4. Louis Guttman, 1945. "A basis for analyzing test-retest reliability," Psychometrika, Springer;The Psychometric Society, vol. 10(4), pages 255-282, December.
    5. Dan M. Kahan & Donald Braman & John Gastil & Paul Slovic & C. K. Mertz, 2007. "Culture and Identity‐Protective Cognition: Explaining the White‐Male Effect in Risk Perception," Journal of Empirical Legal Studies, John Wiley & Sons, vol. 4(3), pages 465-505, November.
    6. Chad M. Zanocco & Michael D. Jones, 2018. "Cultural Worldviews and Political Process Preferences," Social Science Quarterly, Southwestern Social Science Association, vol. 99(4), pages 1377-1389, December.
    7. Jean Brenot & Sylviane Bonnefous & Claire Marris, 1998. "Testing the Cultural Theory of Risk in France," Risk Analysis, John Wiley & Sons, vol. 18(6), pages 729-739, December.
    8. Joseph T. Ripberger & Geoboo Song & Matthew C. Nowlin & Michael D. Jones & Hank C. Jenkins-Smith, 2012. "Reconsidering the Relationship Between Cultural Theory, Political Ideology, and Political Knowledge," Social Science Quarterly, Southwestern Social Science Association, vol. 93(3), pages 713-731, September.
    9. Brendon Swedlow & Joseph T. Ripberger & Li‐Yin Liu & Carol L. Silva & Hank Jenkins‐Smith & Branden B. Johnson, 2020. "Construct Validity of Cultural Theory Survey Measures," Social Science Quarterly, Southwestern Social Science Association, vol. 101(6), pages 2332-2383, October.
    10. Claire Marris & Ian H. Langford & Timothy O'Riordan, 1998. "A Quantitative Test of the Cultural Theory of Risk Perceptions: Comparison with the Psychometric Paradigm," Risk Analysis, John Wiley & Sons, vol. 18(5), pages 635-647, October.
    11. Ian H. Langford & Stavros Georgiou & Ian J. Bateman & Rosemary J. Day & R. Kerry Turner, 2000. "Public Perceptions of Health Risks from Polluted Coastal Bathing Waters: A Mixed Methodological Analysis Using Cultural Theory," Risk Analysis, John Wiley & Sons, vol. 20(5), pages 691-704, October.
    12. Lee Cronbach, 1951. "Coefficient alpha and the internal structure of tests," Psychometrika, Springer;The Psychometric Society, vol. 16(3), pages 297-334, September.
    13. Branden B. Johnson & Brendon Swedlow, 2020. "Comparing cultural theory and cultural cognition theory survey measures to each other and as explanations for judged risk," Journal of Risk Research, Taylor & Francis Journals, vol. 23(10), pages 1278-1300, October.
    14. Peterson, Robert A, 1994. "A Meta-analysis of Cronbach's Coefficient Alpha," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 21(2), pages 381-391, September.
    15. Muriel Bouyer & Sophie Bagdassarian & Sveti Chaabanne & Etienne Mullet, 2001. "Personality Correlates of Risk Perception," Risk Analysis, John Wiley & Sons, vol. 21(3), pages 457-466, June.
    16. Rizopoulos, Dimitris, 2006. "ltm: An R Package for Latent Variable Modeling and Item Response Analysis," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 17(i05).
    17. Gail Markle, 2019. "Understanding Pro-Environmental Behavior in the US: Insights from Grid-Group Cultural Theory and Cognitive Sociology," Sustainability, MDPI, vol. 11(2), pages 1-14, January.
    18. Gunnar Grendstad & Per Selle, 2000. "Cultural Myths of Human and Physical Nature: Integrated or Separated?," Risk Analysis, John Wiley & Sons, vol. 20(1), pages 27-40, February.
    19. Ronald L. Wasserstein & Allen L. Schirm & Nicole A. Lazar, 2019. "Moving to a World Beyond “p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 1-19, March.
    20. Klaas Sijtsma, 2009. "On the Use, the Misuse, and the Very Limited Usefulness of Cronbach’s Alpha," Psychometrika, Springer;The Psychometric Society, vol. 74(1), pages 107-120, March.
    21. Geoboo Song, 2014. "Understanding Public Perceptions of Benefits and Risks of Childhood Vaccinations in the United States," Risk Analysis, John Wiley & Sons, vol. 34(3), pages 541-555, March.
    22. Susanne Rippl, 2002. "Cultural theory and risk perception: a proposal for a better measurement," Journal of Risk Research, Taylor & Francis Journals, vol. 5(2), pages 147-165, April.
    23. Branden B. Johnson & Brendon Swedlow, 2021. "Cultural Theory's Contributions to Risk Analysis: A Thematic Review with Directions and Resources for Further Research," Risk Analysis, John Wiley & Sons, vol. 41(3), pages 429-455, March.
    24. Branden B. Johnson & Brendon Swedlow & Marcus W. Mayorga, 2020. "Cultural theory and cultural cognition theory survey measures: confirmatory factoring and predictive validity of factor scores for judged risk," Journal of Risk Research, Taylor & Francis Journals, vol. 23(11), pages 1467-1490, November.
    25. Rachel A. Hirsch & Jamie Baxter, 2011. "Context, Cultural Bias, and Health Risk Perception: The “Everyday” Nature of Pesticide Policy Preferences in London, Calgary, and Halifax," Risk Analysis, John Wiley & Sons, vol. 31(5), pages 847-865, May.
    26. Ellis, Richard J. & Thompson, Fred, 1997. "Culture and the Environment in the Pacific Northwest," American Political Science Review, Cambridge University Press, vol. 91(4), pages 885-897, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Branden B. Johnson & Brendon Swedlow, 2021. "Cultural Theory's Contributions to Risk Analysis: A Thematic Review with Directions and Resources for Further Research," Risk Analysis, John Wiley & Sons, vol. 41(3), pages 429-455, March.
    2. Jingjing Zeng & Meiquan Jiang & Meng Yuan, 2020. "Environmental Risk Perception, Risk Culture, and Pro-Environmental Behavior," IJERPH, MDPI, vol. 17(5), pages 1-18, March.
    3. Qi Guo & Palizhati Muhetaer & Ping Hu, 2023. "Cultural worldviews and support for governmental management of COVID-19," Palgrave Communications, Palgrave Macmillan, vol. 10(1), pages 1-16, December.
    4. Matthew C. Nowlin, 2022. "Who should “do more” about climate change? Cultural theory, polycentricity, and public support for climate change actions across actors and governments," Review of Policy Research, Policy Studies Organization, vol. 39(4), pages 468-485, July.
    5. Brendon Swedlow & Joseph T. Ripberger & Li‐Yin Liu & Carol L. Silva & Hank Jenkins‐Smith & Branden B. Johnson, 2020. "Construct Validity of Cultural Theory Survey Measures," Social Science Quarterly, Southwestern Social Science Association, vol. 101(6), pages 2332-2383, October.
    6. Boschetti, Fabio & Richert, Claire & Walker, Iain & Price, Jennifer & Dutra, Leo, 2012. "Assessing attitudes and cognitive styles of stakeholders in environmental projects involving computer modelling," Ecological Modelling, Elsevier, vol. 247(C), pages 98-111.
    7. Hoogstra-Klein, Marjanke A. & Permadi, Dwiko B. & Yasmi, Yurdi, 2012. "The value of cultural theory for participatory processes in natural resource management," Forest Policy and Economics, Elsevier, vol. 20(C), pages 99-106.
    8. Markus Pauly & Maria Umlauft & Ali Ünlü, 2018. "Resampling-Based Inference Methods for Comparing Two Coefficients Alpha," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 203-222, March.
    9. Salim Moussa, 2016. "A Comment on the Estimation of the Reliability of Multidimensional Marketing Constructs: A Store Personality Scale Application," Global Business Review, International Management Institute, vol. 17(5), pages 1125-1144, October.
    10. Eunseong Cho, 2021. "Neither Cronbach’s Alpha nor McDonald’s Omega: A Commentary on Sijtsma and Pfadt," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 877-886, December.
    11. Zhengyan Li & David M. Konisky, 2023. "Personal attributes and (mis)perceptions of local environmental risk," Review of Policy Research, Policy Studies Organization, vol. 40(1), pages 119-152, January.
    12. David J. Hessen, 2017. "Lower Bounds to the Reliabilities of Factor Score Estimators," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 648-659, September.
    13. Érika Martins Silva Ramos & Cecilia Jakobsson Bergstad, 2021. "The Psychology of Sharing: Multigroup Analysis among Users and Non-Users of Carsharing," Sustainability, MDPI, vol. 13(12), pages 1-17, June.
    14. Creed Tumlison & Rachael M. Moyer & Geoboo Song, 2017. "The Origin and Role of Trust in Local Policy Elites’ Perceptions of High‐Voltage Power Line Installations in the State of Arkansas," Risk Analysis, John Wiley & Sons, vol. 37(5), pages 1018-1036, May.
    15. Jules L. Ellis, 2021. "A Test Can Have Multiple Reliabilities," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 869-876, December.
    16. van de Graaff, Shashi, 2016. "Understanding the nuclear controversy: An application of cultural theory," Energy Policy, Elsevier, vol. 97(C), pages 50-59.
    17. M. Aenne Schoop & Marco Verweij & Ulrich Kühnen & Shenghua Luan, 2020. "Political disagreement in the classroom: testing cultural theory through structured observation," Quality & Quantity: International Journal of Methodology, Springer, vol. 54(2), pages 623-643, April.
    18. Sunhee Kim & Seoyong Kim, 2018. "Exploring the Determinants of Perceived Risk of Middle East Respiratory Syndrome (MERS) in Korea," IJERPH, MDPI, vol. 15(6), pages 1-18, June.
    19. Michael Siegrist & Joseph Árvai, 2020. "Risk Perception: Reflections on 40 Years of Research," Risk Analysis, John Wiley & Sons, vol. 40(S1), pages 2191-2206, November.
    20. Wang, Selena & De Boeck, Paul, 2020. "When high reliability does not signal reliable detection of experimental effects," OSF Preprints gz8pw, Center for Open Science.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:58:y:2024:i:1:d:10.1007_s11135-023-01653-4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.