IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v56y2022i4d10.1007_s11135-021-01252-1.html
   My bibliography  Save this article

Ensuring survey research data integrity in the era of internet bots

Author

Listed:
  • Marybec Griffin

    (Rutgers University
    Rutgers University)

  • Richard J. Martino

    (Rutgers University)

  • Caleb LoSchiavo

    (Rutgers University
    Rutgers University)

  • Camilla Comer-Carruthers

    (Rutgers University
    Rutgers University)

  • Kristen D. Krause

    (Rutgers University
    Rutgers University)

  • Christopher B. Stults

    (Rutgers University
    City University of New York)

  • Perry N. Halkitis

    (Rutgers University
    Rutgers University
    Rutgers University
    Rutgers University)

Abstract

We used an internet-based survey platform to conduct a cross-sectional survey regarding the impact of COVID-19 on the LGBTQ + population in the United States. While this method of data collection was quick and inexpensive, the data collected required extensive cleaning due to the infiltration of bots. Based on this experience, we provide recommendations for ensuring data integrity. Recruitment conducted between May 7 and 8, 2020 resulted in an initial sample of 1251 responses. The Qualtrics survey was disseminated via social media and professional association listservs. After noticing data discrepancies, research staff developed a rigorous data cleaning protocol. A second wave of recruitment was conducted on June 11–12, 2020 using the original recruitment methods. The five-step data cleaning protocol led to the removal of 773 (61.8%) surveys from the initial dataset, resulting in a sample of 478 participants in the first wave of data collection. The protocol led to the removal of 46 (31.9%) surveys from the second two-day wave of data collection, resulting in a sample of 98 participants in the second wave of data collection. After verifying the two-day pilot process was effective at screening for bots, the survey was reopened for a third wave of data collection resulting in a total of 709 responses, which were identified as an additional 514 (72.5%) valid participants and led to the removal of an additional 194 (27.4%) possible bots. The final analytic sample consists of 1090 participants. Although a useful and efficient research tool, especially among hard-to-reach populations, internet-based research is vulnerable to bots and mischievous responders, despite survey platforms’ built-in protections. Beyond the depletion of research funds, bot infiltration threatens data integrity and may disproportionately harm research with marginalized populations. Based on our experience, we recommend the use of strategies such as qualitative questions, duplicate demographic questions, and incentive raffles to reduce likelihood of mischievous respondents. These protections can be undertaken to ensure data integrity and facilitate research on vulnerable populations.

Suggested Citation

  • Marybec Griffin & Richard J. Martino & Caleb LoSchiavo & Camilla Comer-Carruthers & Kristen D. Krause & Christopher B. Stults & Perry N. Halkitis, 2022. "Ensuring survey research data integrity in the era of internet bots," Quality & Quantity: International Journal of Methodology, Springer, vol. 56(4), pages 2841-2852, August.
    • Handle: RePEc:spr:qualqt:v:56:y:2022:i:4:d:10.1007_s11135-021-01252-1
    DOI: 10.1007/s11135-021-01252-1
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11135-021-01252-1
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11135-021-01252-1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Martine Selm & Nicholas Jankowski, 2006. "Conducting Online Surveys," Quality & Quantity: International Journal of Methodology, Springer, vol. 40(3), pages 435-456, June.
    2. Jeffrey M. Perkel, 2020. "Mischief-making bots attacked my scientific survey," Nature, Nature, vol. 579(7799), pages 461-461, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Oguzhan Essiz & Aysu Senyuz, 2024. "Predicting the value‐based determinants of sustainable luxury consumption: A multi‐analytical approach and pathway to sustainable development in the luxury industry," Business Strategy and the Environment, Wiley Blackwell, vol. 33(3), pages 1721-1758, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Morenike Oluwatoyin Folayan & Roberto Ariel Abeldaño Zuniga & Oliver C. Ezechi & Brandon Brown & Annie L. Nguyen & Nourhan M. Aly & Passent Ellakany & Ifeoma E. Idigbe & Abeedha Tu-Allah Khan & Folake, 2022. "Associations between Emotional Distress, Sleep Changes, Decreased Tooth Brushing Frequency, Self-Reported Oral Ulcers and SARS-Cov-2 Infection during the First Wave of the COVID-19 Pandemic: A Global ," IJERPH, MDPI, vol. 19(18), pages 1-11, September.
    2. Shannon Davis & Andrey Shevchuk & Denis Strebkov, 2014. "Pathways to Satisfaction with Work-Life Balance: The Case of Russian-Language Internet Freelancers," Journal of Family and Economic Issues, Springer, vol. 35(4), pages 542-556, December.
    3. Vicente Gea-Caballero & José Ramón Martínez-Riera & Pedro García-Martínez & Jorge Casaña-Mohedo & Isabel Antón-Solanas & María Virtudes Verdeguer-Gómez & Iván Santolaya-Arnedo & Raúl Juárez-Vela, 2021. "Study of the Strengths and Weaknesses of Nursing Work Environments in Primary Care in Spain," IJERPH, MDPI, vol. 18(2), pages 1-12, January.
    4. Vendrell-Herrero, Ferran & Bustinza, Oscar F. & Opazo-Basaez, Marco, 2021. "Information technologies and product-service innovation: The moderating role of service R&D team structure," Journal of Business Research, Elsevier, vol. 128(C), pages 673-687.
    5. Andrea Serge & Johana Quiroz Montoya & Francisco Alonso & Luis Montoro, 2021. "Socioeconomic Status, Health and Lifestyle Settings as Psychosocial Risk Factors for Road Crashes in Young People: Assessing the Colombian Case," IJERPH, MDPI, vol. 18(3), pages 1-22, January.
    6. Ahmed Hassan Abdou & Majed Abdulaziz Al Abdulathim & Nadia Rebhi Hussni Hasan & Maha Hassan Ahmed Salah & Howayda Said Ahmed Mohamed Ali & Nancy J. Kamel, 2023. "From Green Inclusive Leadership to Green Organizational Citizenship: Exploring the Mediating Role of Green Work Engagement and Green Organizational Identification in the Hotel Industry Context," Sustainability, MDPI, vol. 15(20), pages 1-22, October.
    7. Konstantinos Nikolaos Vasileiadis & Konstantinos Alexandros Tsioumis & Argyris Kyridis, 2013. "The Effects of Dominant Ideology on Teachers¡¯ Perceptions and Practices towards the ¡°Other¡±," International Journal of Learning and Development, Macrothink Institute, vol. 3(1), pages 33-48, December.
    8. Lara Fontanella & Paola Villano & Marika Di Donato, 2016. "Attitudes towards Roma people and migrants: a comparison through a Bayesian multidimensional IRT model," Quality & Quantity: International Journal of Methodology, Springer, vol. 50(2), pages 471-490, March.
    9. Vendrell-Herrero, Ferran & Gomes, Emanuel & Bustinza, Oscar F. & Mellahi, Kamel, 2018. "Uncovering the role of cross-border strategic alliances and expertise decision centralization in enhancing product-service innovation in MMNEs," International Business Review, Elsevier, vol. 27(4), pages 814-825.
    10. Ioan-Sebastian Brumă & Cristina Cautisanu & Lucian Tanasă & Simona-Roxana Ulman & Meda Gâlea & Alexandra Raluca Jelea, 2024. "Does the payment method matter in online shopping behaviour? Study on the Romanian market of vegetables during the pandemic crisis," Agricultural Economics, Czech Academy of Agricultural Sciences, vol. 70(1), pages 34-47.
    11. Evelyn Schapansky & Joke Depraetere & Ines Keygnaert & Christophe Vandeviver, 2021. "Prevalence and Associated Factors of Sexual Victimization: Findings from a National Representative Sample of Belgian Adults Aged 16–69," IJERPH, MDPI, vol. 18(14), pages 1-22, July.
    12. Martin Loidl & Christian Werner & Laura Heym & Patrick Kofler & Günther Innerebner, 2019. "Lifestyles and Cycling Behavior—Data from a Cross-Sectional Study," Data, MDPI, vol. 4(4), pages 1-19, October.
    13. Lara Fontanella & Mara Maretti & Annalina Sarra, 2014. "Gender fluidity across the world: a Multilevel Item Response Theory approach," Quality & Quantity: International Journal of Methodology, Springer, vol. 48(5), pages 2553-2568, September.
    14. Magdy Sayed Ahmed Abolnasser & Ahmed Hassan Abdou & Thowayeb H. Hassan & Amany E. Salem, 2023. "Transformational Leadership, Employee Engagement, Job Satisfaction, and Psychological Well-Being among Hotel Employees after the Height of the COVID-19 Pandemic: A Serial Mediation Model," IJERPH, MDPI, vol. 20(4), pages 1-27, February.
    15. Martin P. Fritze & Andreas B. Eisingerich & Martin Benkenstein, 2019. "Digital transformation and possession attachment: examining the endowment effect for consumers’ relationships with hedonic and utilitarian digital service technologies," Electronic Commerce Research, Springer, vol. 19(2), pages 311-337, June.
    16. Alyavina, Elena & Nikitas, Alexandros & Njoya, Eric Tchouamou, 2024. "Mobility-as-a-Service and unsustainable travel behaviour: Exploring the car ownership and public transport trip replacement side-effects of the MaaS paradigm," Transport Policy, Elsevier, vol. 150(C), pages 53-70.
    17. Prentice, Catherine & Handsjuk, Nikolai, 2016. "Insights into Vodka consumer attitude and purchasing behaviors," Journal of Retailing and Consumer Services, Elsevier, vol. 32(C), pages 7-14.
    18. Leigh N. Wood & Jim Psaros & Erica French & Jennifer W.M. Lai, 2015. "Learning experiences for the transition to professional work," Cogent Business & Management, Taylor & Francis Journals, vol. 2(1), pages 1042099-104, December.
    19. Edgar Treischl & Tobias Wolbring, 2017. "The Causal Effect of Survey Mode on Students’ Evaluations of Teaching: Empirical Evidence from Three Field Experiments," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(8), pages 904-921, December.
    20. Leenshya Gunnoo & Eric Bindah & Sudursun Thakoor, 2024. "Exploring Technology's Impact on Digitalization, Leadership, and Employee Performance: A Case Study of Local Authorities in Mauritius," Economics and Applied Informatics, "Dunarea de Jos" University of Galati, Faculty of Economics and Business Administration, issue 3, pages 55-66.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:56:y:2022:i:4:d:10.1007_s11135-021-01252-1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.