IDEAS home Printed from https://ideas.repec.org/a/sae/sagope/v8y2018i1p2158244018764803.html
   My bibliography  Save this article

Respondent Robotics: Simulating Responses to Likert-Scale Survey Items

Author

Listed:
  • Jan Ketil Arnulf
  • Kai R. Larsen
  • Øyvind L. Martinsen

Abstract

The semantic theory of survey responses (STSR) proposes that the prime source of statistical covariance in survey data is the degree of semantic similarity (overlap of meaning) among the items of the survey. Because semantic structures are possible to estimate using digital text algorithms, it is possible to predict the response structures of Likert-type scales a priori. The present study applies STSR in an experimental way by computing real survey responses using such semantic information. A sample of 153 randomly chosen respondents to the Multifactor Leadership Questionnaire (MLQ) was used as target. We developed an algorithm based on unfolding theory, where data from digital text analysis of the survey items served as input. Upon deleting progressive numbers (from 20%-95%) of the real responses, we let the algorithm replace these with simulated ones, and then compared the simulated datasets with the real ones. The simulated scores displayed sum score levels, alphas, and factor structures highly resembling their real origins even if up to 86% were simulated. In contrast, this was not the case when the same algorithm was operating without access to semantic information. The procedure was briefly repeated on a different measurement instrument and a different sample. This not only yielded similar results but also pointed to need for further theoretical and practical developments. Our study opens for experimental research on the effect of semantics on survey responses using computational procedures.

Suggested Citation

  • Jan Ketil Arnulf & Kai R. Larsen & Øyvind L. Martinsen, 2018. "Respondent Robotics: Simulating Responses to Likert-Scale Survey Items," SAGE Open, , vol. 8(1), pages 21582440187, March.
  • Handle: RePEc:sae:sagope:v:8:y:2018:i:1:p:2158244018764803
    DOI: 10.1177/2158244018764803
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/2158244018764803
    Download Restriction: no

    File URL: https://libkey.io/10.1177/2158244018764803?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Singh, Jagdip, 2004. "Tackling measurement problems with Item Response Theory: Principles, characteristics, and assessment, with an illustrative example," Journal of Business Research, Elsevier, vol. 57(2), pages 184-208, February.
    2. Jan Ketil Arnulf & Kai Rune Larsen & Øyvind Lund Martinsen & Chih How Bong, 2014. "Predicting Survey Responses: How and Why Semantics Shape Survey Statistics on Organizational Behaviour," PLOS ONE, Public Library of Science, vol. 9(9), pages 1-13, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Salzberger, Thomas & Koller, Monika, 2013. "Towards a new paradigm of measurement in marketing," Journal of Business Research, Elsevier, vol. 66(9), pages 1307-1317.
    2. Dirk Temme & Lutz Hildebrandt, 2009. "Gruppenvergleiche bei hypothetischen Konstrukten — Die Prüfung der Übereinstimmung von Messmodellen mit der Strukturgleichungsmethodik," Schmalenbach Journal of Business Research, Springer, vol. 61(2), pages 138-185, March.
    3. Knoppen, Desirée & Saris, Willem & Moncagatta, Paolo, 2022. "Absorptive capacity dimensions and the measurement of cumulativeness," Journal of Business Research, Elsevier, vol. 139(C), pages 312-324.
    4. Maud Dampérat & Ping Lei & Florence Jeannot, 2019. "IRT Approach for rating scales: applications for normal and non-normal distributions," Post-Print hal-04325043, HAL.
    5. A. M. Valkengoed & G. Perlaviciute & L. Steg, 2022. "Relationships between climate change perceptions and climate adaptation actions: policy support, information seeking, and behaviour," Climatic Change, Springer, vol. 171(1), pages 1-20, March.
    6. Jean-Charles Pillet & Claudio Vitari & Jackie London & Kevin D Matthews, 2022. "Early-Stage Construct Development Practices in IS Research: A 2000-2020 Review," Post-Print hal-03876784, HAL.
    7. Hergesell, Anja, 2022. "Using Rasch analysis for scale development and refinement in tourism: Theory and illustration," Journal of Business Research, Elsevier, vol. 142(C), pages 551-561.
    8. Liu, Jing & Lin, Hua & Hu, Bing & Zhou, Zixiong & Agyeiwaah, Elizabeth & Xu, Ye, 2022. "Advancing the understanding of the resident pro-tourism behavior scale: An integration of item response theory and classical test theory," Journal of Business Research, Elsevier, vol. 141(C), pages 113-125.
    9. Michael Hennessy & Amy Bleakley & Morgan E. Ellithorpe, 2023. "Evaluating and tracking qualitative content coder performance using item response theory," Quality & Quantity: International Journal of Methodology, Springer, vol. 57(2), pages 1231-1245, April.
    10. Ganglmair-Wooliscroft, Alexandra & Wooliscroft, Ben, 2016. "Diffusion of innovation: The case of ethical tourism behavior," Journal of Business Research, Elsevier, vol. 69(8), pages 2711-2720.
    11. Sarstedt, Marko & Diamantopoulos, Adamantios & Salzberger, Thomas & Baumgartner, Petra, 2016. "Selecting single items to measure doubly concrete constructs: A cautionary tale," Journal of Business Research, Elsevier, vol. 69(8), pages 3159-3167.
    12. Ofir Turel & Yufei Yuan, 2007. "User Acceptance of Web-Based Negotiation Support Systems: The Role of Perceived Intention of the Negotiating Partner to Negotiate Online," Group Decision and Negotiation, Springer, vol. 16(5), pages 451-468, September.
    13. Salzberger, Thomas & Newton, Fiona J. & Ewing, Michael T., 2014. "Detecting gender item bias and differential manifest response behavior: A Rasch-based solution," Journal of Business Research, Elsevier, vol. 67(4), pages 598-607.
    14. Leonard Paas & Klaas Sijtsma, 2008. "Nonparametric item response theory for investigating dimensionality of marketing scales: A SERVQUAL application," Marketing Letters, Springer, vol. 19(2), pages 157-170, June.
    15. de Jong, M.G., 2006. "Response bias in international marketing research," Other publications TiSEM 5d4031be-97b5-4db3-962b-2, Tilburg University, School of Economics and Management.
    16. Salim Moussa, 2016. "A two-step item response theory procedure for a better measurement of marketing constructs," Journal of Marketing Analytics, Palgrave Macmillan, vol. 4(1), pages 28-50, March.
    17. Vanderleia Martins Lohn & Rafael Tezza & Graziela Dias Alperstedt & Lucila M. S. Campos, 2017. "Future Professionals: A Study of Sustainable Behavior," Sustainability, MDPI, vol. 9(3), pages 1-15, March.
    18. Silvana Bortolotti & Rafael Tezza & Dalton Andrade & Antonio Bornia & Afonso Sousa Júnior, 2013. "Relevance and advantages of using the item response theory," Quality & Quantity: International Journal of Methodology, Springer, vol. 47(4), pages 2341-2360, June.
    19. Ganglmair-Wooliscroft, Alexandra & Wooliscroft, Ben, 2022. "An investigation of sustainable consumption behavior systems – Exploring personal and socio-structural characteristics in different national contexts," Journal of Business Research, Elsevier, vol. 148(C), pages 161-173.
    20. Jan Ketil Arnulf & Kai Rune Larsen & Øyvind Lund Martinsen, 2018. "Semantic algorithms can detect how media language shapes survey responses in organizational behaviour," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-26, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:sagope:v:8:y:2018:i:1:p:2158244018764803. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.