IDEAS home Printed from https://ideas.repec.org/a/sae/somere/v49y2020i1p79-107.html
   My bibliography  Save this article

Rating-Scale Labeling in Online Surveys: An Experimental Comparison of Verbal and Numeric Rating Scales with Respect to Measurement Quality and Respondents’ Cognitive Processes

Author

Listed:
  • Natalja Menold

Abstract

Unlike other data collection modes, the effect of labeling rating scales on reliability and validity, as relevant aspects of measurement quality, has seldom been addressed in online surveys. In this study, verbal and numeric rating scales were compared in split-ballot online survey experiments. In the first experiment, respondents’ cognitive processes were observed by means of eye tracking, that is, determining the respondent’s fixations in different areas of the screen. In the remaining experiments, data for reliability and validity analysis were collected from a German adult sample. The results show that respondents needed more fixations and more time to endorse a category when a rating scale had numeric labels. Cross-sectional reliability was lower and some hypotheses with respect to the criterion validity could not be supported when numeric rating scales were used. In conclusion, theoretical considerations and the empirical results contradict the current broad usage of numeric scales in online surveys.

Suggested Citation

  • Natalja Menold, 2020. "Rating-Scale Labeling in Online Surveys: An Experimental Comparison of Verbal and Numeric Rating Scales with Respect to Measurement Quality and Respondents’ Cognitive Processes," Sociological Methods & Research, , vol. 49(1), pages 79-107, February.
    • Handle: RePEc:sae:somere:v:49:y:2020:i:1:p:79-107
    DOI: 10.1177/0049124117729694
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0049124117729694
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0049124117729694?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kristel Wouters & Jeroen Maesschalck & Carel Peeters & Marijke Roosen, 2014. "Methodological Issues in the Design of Online Surveys for Measuring Unethical Work Behavior: Recommendations on the Basis of a Split-Ballot Experiment," Journal of Business Ethics, Springer, vol. 120(2), pages 275-289, March.
    2. Natalja Menold & Anja Tausch, 2016. "Measurement of Latent Variables With Different Rating Scales," Sociological Methods & Research, , vol. 45(4), pages 678-699, November.
    3. RatSWD Quality Standards Working Group, 2015. "Quality Standards for the Development, Application, and Evaluation of Measurement Instruments in Social Science Survey Research," RatSWD Working Papers 245, German Data Forum (RatSWD).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joseph McManus, 2018. "Hubris and Unethical Decision Making: The Tragedy of the Uncommon," Journal of Business Ethics, Springer, vol. 149(1), pages 169-185, April.
    2. Kai Chi Yam & Scott J. Reynolds & Pengcheng Zhang & Runkun Su, 2022. "The Unintended Consequences of Empowering Leadership: Increased Deviance for Some Followers," Journal of Business Ethics, Springer, vol. 181(3), pages 683-700, December.
    3. Shike Li & Kriti Jain & Konstantina Tzini, 2022. "When Supervisor Support Backfires: The Link Between Perceived Supervisor Support and Unethical Pro-supervisor Behavior," Journal of Business Ethics, Springer, vol. 179(1), pages 133-151, August.
    4. Mazzucchelli, Alice & Gurioli, Martina & Graziano, Domenico & Quacquarelli, Barbara & Aouina-Mejri, Chiraz, 2021. "How to fight against food waste in the digital era: Key factors for a successful food sharing platform," Journal of Business Research, Elsevier, vol. 124(C), pages 47-58.
    5. Stefan Liebig & Sebastian Hülle & Meike May, 2016. "Principles of the Just Distribution of Benefits and Burdens: The "Basic Social Justice Orientations" Scale for Measuring Order-Related Social Justice Attitudes," SOEPpapers on Multidisciplinary Panel Data Research 831, DIW Berlin, The German Socio-Economic Panel (SOEP).
    6. Micael-Lee Johnstone & Lay Tan, 2015. "Exploring the Gap Between Consumers’ Green Rhetoric and Purchasing Behaviour," Journal of Business Ethics, Springer, vol. 132(2), pages 311-328, December.
    7. Sebastian Hülle & Stefan Liebig & Meike Janina May, 2018. "Measuring Attitudes Toward Distributive Justice: The Basic Social Justice Orientations Scale," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 136(2), pages 663-692, April.
    8. So Young Song & Youn-Kyung Kim, 2018. "Theory of Virtue Ethics: Do Consumers’ Good Traits Predict Their Socially Responsible Consumption?," Journal of Business Ethics, Springer, vol. 152(4), pages 1159-1175, November.
    9. Abramovic, Gordana & Traavik, Laura E. Mercer, 2017. "Support for diversity practices in Norway: Depends on who you are and whom you have met," European Management Journal, Elsevier, vol. 35(4), pages 454-463.
    10. Sougata Ray & Bikramjit Ray Chaudhuri, 2018. "Business Group Affiliation and Corporate Sustainability Strategies of Firms: An Investigation of Firms in India," Journal of Business Ethics, Springer, vol. 153(4), pages 955-976, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:somere:v:49:y:2020:i:1:p:79-107. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.