IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0226408.html
   My bibliography  Save this article

The promise of open survey questions—The validation of text-based job satisfaction measures

Author

Listed:
  • Indy Wijngaards
  • Martijn Burger
  • Job van Exel

Abstract

Recent advances in computer-aided text analysis (CATA) have allowed organizational scientists to construct reliable and convenient measures from open texts. As yet, there is a lack of research into using CATA to analyze responses to open survey questions and constructing text-based measures of psychological constructs. In our study, we demonstrated the potential of CATA methods for the construction of text-based job satisfaction measures based on responses to a completely open and semi-open question. To do this, we employed three sentiment analysis techniques: Linguistic Inquiry and Word Count 2015, SentimentR and SentiStrength, and quantified the forms of measurement error they introduced: specific factor error, algorithm error and transient error. We conducted an initial test of the text-based measures’ validity, assessing their convergence with closed-question job satisfaction measures. We adopted a time-lagged survey design (Nwave 1 = 996; Nwave 2 = 116) to test our hypotheses. In line with our hypotheses, we found that specific factor error is higher in the open question text-based measure than in the semi-open question text-based measure. As expected, algorithm error was substantial for both the open and semi-open question text-based measures. Transient error in the text-based measures was higher than expected, as it generally exceeded the transient error in the human-coded and the closed job satisfaction question measures. Our initial test of convergent and discriminant validity indicated that the semi-open question text-based measure is especially suitable for measuring job satisfaction. Our article ends with a discussion of limitations and an agenda for future research.

Suggested Citation

  • Indy Wijngaards & Martijn Burger & Job van Exel, 2019. "The promise of open survey questions—The validation of text-based job satisfaction measures," PLOS ONE, Public Library of Science, vol. 14(12), pages 1-22, December.
    • Handle: RePEc:plo:pone00:0226408
    DOI: 10.1371/journal.pone.0226408
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0226408
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0226408&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0226408?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Mike Thelwall & Kevan Buckley & Georgios Paltoglou & Di Cai & Arvid Kappas, 2010. "Sentiment strength detection in short informal text," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 61(12), pages 2544-2558, December.
    2. Mikhaylov, Slava & Laver, Michael & Benoit, Kenneth R., 2012. "Coder Reliability and Misclassification in the Human Coding of Party Manifestos," Political Analysis, Cambridge University Press, vol. 20(1), pages 78-91, January.
    3. Christian Bjørnskov, 2010. "How Comparable are the Gallup World Poll Life Satisfaction Data?," Journal of Happiness Studies, Springer, vol. 11(1), pages 41-60, March.
    4. D. Shin & D. Johnson, 1978. "Avowed happiness as an overall assessment of the quality of life," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 5(1), pages 475-492, March.
    5. Ingrid Gilles & Mauro Mayer & Nelly Courvoisier & Isabelle Peytremann-Bridevaux, 2017. "Joint analyses of open comments and quantitative data: Added value in a job satisfaction survey of hospital professionals," PLOS ONE, Public Library of Science, vol. 12(3), pages 1-14, March.
    6. Glerum, Aurélie & Atasoy, Bilge & Bierlaire, Michel, 2014. "Using semi-open questions to integrate perceptions in choice models," Journal of choice modelling, Elsevier, vol. 10(C), pages 11-33.
    7. Ed Diener & Derrick Wirtz & William Tov & Chu Kim-Prieto & Dong-won Choi & Shigehiro Oishi & Robert Biswas-Diener, 2010. "New Well-being Measures: Short Scales to Assess Flourishing and Positive and Negative Feelings," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 97(2), pages 143-156, June.
    8. Gordon Wang & Rick D. Hackett, 2016. "Conceptualization and Measurement of Virtuous Leadership: Doing Well by Doing Good," Journal of Business Ethics, Springer, vol. 137(2), pages 321-345, August.
    9. Mike Thelwall & Kevan Buckley & Georgios Paltoglou & Di Cai & Arvid Kappas, 2010. "Sentiment strength detection in short informal text," Journal of the American Society for Information Science and Technology, Association for Information Science & Technology, vol. 61(12), pages 2544-2558, December.
    10. Mitchell Neubert & Dawn Carlson & K. Kacmar & James Roberts & Lawrence Chonko, 2009. "The Virtuous Influence of Ethical Leadership Behavior: Evidence from the Field," Journal of Business Ethics, Springer, vol. 90(2), pages 157-170, December.
    11. Ingwer Borg & Cornelia Zuell, 2012. "Write‐in comments in employee surveys," International Journal of Manpower, Emerald Group Publishing Limited, vol. 33(2), pages 206-220, May.
    12. N. Wang & M. Kosinski & D. Stillwell & J. Rust, 2014. "Can Well-Being be Measured Using Facebook Status Updates? Validation of Facebook’s Gross National Happiness Index," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 115(1), pages 483-491, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Indy Wijngaards & Owen C. King & Martijn J. Burger & Job Exel, 2022. "Worker Well-Being: What it Is, and how it Should Be Measured," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 17(2), pages 795-832, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Indy Wijngaards & Owen C. King & Martijn J. Burger & Job Exel, 2022. "Worker Well-Being: What it Is, and how it Should Be Measured," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 17(2), pages 795-832, April.
    2. Sergey Smetanin, 2022. "Pulse of the Nation: Observable Subjective Well-Being in Russia Inferred from Social Network Odnoklassniki," Mathematics, MDPI, vol. 10(16), pages 1-38, August.
    3. Ma, Jie & Tse, Ying Kei & Wang, Xiaojun & Zhang, Minhao, 2019. "Examining customer perception and behaviour through social media research – An empirical study of the United Airlines overbooking crisis," Transportation Research Part E: Logistics and Transportation Review, Elsevier, vol. 127(C), pages 192-205.
    4. Müller-Hansen, Finn & Lee, Yuan Ting & Callaghan, Max & Jankin, Slava & Minx, Jan C., 2022. "The German coal debate on Twitter: Reactions to a corporate policy process," Energy Policy, Elsevier, vol. 169(C).
    5. Lipizzi, Carlo & Iandoli, Luca & Ramirez Marquez, José Emmanuel, 2015. "Extracting and evaluating conversational patterns in social media: A socio-semantic analysis of customers’ reactions to the launch of new products using Twitter streams," International Journal of Information Management, Elsevier, vol. 35(4), pages 490-503.
    6. Martin Haselmayer & Marcelo Jenny, 2017. "Sentiment analysis of political communication: combining a dictionary approach with crowdcoding," Quality & Quantity: International Journal of Methodology, Springer, vol. 51(6), pages 2623-2646, November.
    7. Daesik Kim & Chung Joo Chung & Kihong Eom, 2022. "Measuring Online Public Opinion for Decision Making: Application of Deep Learning on Political Context," Sustainability, MDPI, vol. 14(7), pages 1-16, March.
    8. David M. Goldberg & Nohel Zaman & Arin Brahma & Mariano Aloiso, 2022. "Are mortgage loan closing delay risks predictable? A predictive analysis using text mining on discussion threads," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 73(3), pages 419-437, March.
    9. Gabriele Ranco & Ilaria Bordino & Giacomo Bormetti & Guido Caldarelli & Fabrizio Lillo & Michele Treccani, 2014. "Coupling news sentiment with web browsing data improves prediction of intra-day price dynamics," Papers 1412.3948, arXiv.org, revised Dec 2015.
    10. Tadić, Bosiljka & Mitrović Dankulov, Marija & Melnik, Roderick, 2023. "Evolving cycles and self-organised criticality in social dynamics," Chaos, Solitons & Fractals, Elsevier, vol. 171(C).
    11. Ping-Yu Hsu & Hong-Tsuen Lei & Shih-Hsiang Huang & Teng Hao Liao & Yao-Chung Lo & Chin-Chun Lo, 2019. "Effects of sentiment on recommendations in social network," Electronic Markets, Springer;IIM University of St. Gallen, vol. 29(2), pages 253-262, June.
    12. John J. Sosik & Jae Uk Chun & Ziya Ete & Fil J. Arenas & Joel A. Scherer, 2019. "Self-control Puts Character into Action: Examining How Leader Character Strengths and Ethical Leadership Relate to Leader Outcomes," Journal of Business Ethics, Springer, vol. 160(3), pages 765-781, December.
    13. Cohen, Scott & Stienmetz, Jason & Hanna, Paul & Humbracht, Michael & Hopkins, Debbie, 2020. "Shadowcasting tourism knowledge through media: Self-driving sex cars?," Annals of Tourism Research, Elsevier, vol. 85(C).
    14. Zhang, Xuetong & Zhang, Weiguo, 2023. "Information asymmetry, sentiment interactions, and asset price," The North American Journal of Economics and Finance, Elsevier, vol. 67(C).
    15. Takahiro Yabe & P. Suresh C. Rao & Satish V. Ukkusuri, 2021. "Modeling the Influence of Online Social Media Information on Post-Disaster Mobility Decisions," Sustainability, MDPI, vol. 13(9), pages 1-13, May.
    16. Patricia P. Iglesias-Sánchez & Gustavo Fabián Vaccaro Witt & Francisco E. Cabrera & Carmen Jambrino-Maldonado, 2020. "The Contagion of Sentiments during the COVID-19 Pandemic Crisis: The Case of Isolation in Spain," IJERPH, MDPI, vol. 17(16), pages 1-10, August.
    17. Junegak Joung & Ki-Hun Kim & Kwangsoo Kim, 2021. "Data-Driven Approach to Dual Service Failure Monitoring From Negative Online Reviews: Managerial Perspective," SAGE Open, , vol. 11(1), pages 21582440209, January.
    18. Wang, Fang & Du, Zhao & Wang, Shan, 2023. "Information multidimensionality in online customer reviews," Journal of Business Research, Elsevier, vol. 159(C).
    19. Frantisek Darena & Jonas Petrovsky & Jan Zizka & Jan Prichystal, 2016. "Analyzing the correlation between online texts and stock price movements at micro-level using machine learning," MENDELU Working Papers in Business and Economics 2016-67, Mendel University in Brno, Faculty of Business and Economics.
    20. Massimo Aria & Michelangelo Misuraca & Maria Spano, 2020. "Mapping the Evolution of Social Research and Data Science on 30 Years of Social Indicators Research," Social Indicators Research: An International and Interdisciplinary Journal for Quality-of-Life Measurement, Springer, vol. 149(3), pages 803-831, June.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0226408. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.