IDEAS home Printed from https://ideas.repec.org/a/spr/drugsa/v44y2021i3d10.1007_s40264-020-01028-w.html
   My bibliography  Save this article

Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation

Author

Listed:
  • Alex Gartland

    (University of Central Florida)

  • Andrew Bate

    (GlaxoSmithKline)

  • Jeffery L. Painter

    (JiveCast)

  • Tim A. Casperson

    (GlaxoSmithKline)

  • Gregory Eugene Powell

    (GlaxoSmithKline)

Abstract

Introduction Machine learning offers an alluring solution to developing automated approaches to the increasing individual case safety report burden being placed upon pharmacovigilance. Leveraging crowdsourcing to annotate unstructured data may provide accurate, efficient, and contemporaneous training data sets in support of machine learning. Objective The objective of this study was to evaluate whether crowdsourcing can be used to accurately and efficiently develop training data sets in support of pharmacovigilance automation. Materials and Methods Pharmacovigilance experts created a reference dataset by reviewing 15,490 de-identified social media posts of narratives pertaining to 15 drugs and 22 medically relevant topics. A random sampling of posts from the reference dataset was published on Amazon Turk and its users (Turkers) were asked a series of questions about those same medical concepts. Accuracy, price elasticity, and time efficiency were evaluated. Results Accuracy of crowdsourced curation exceeded 90% when compared to the reference dataset and was completed in about 5% of the time. There was an increase in time efficiency with higher pay, but there was no significant difference in accuracy. Additionally, having a social media post reviewed by more than one Turker (using a voting system) did not offer significant improvements in terms of accuracy. Conclusions Crowdsourcing is an accurate and efficient method that can be used to develop training data sets in support of pharmacovigilance automation. More research is needed to better understand the breadth and depth of possible uses as well as strengths, limitations, and generalizability of results.

Suggested Citation

  • Alex Gartland & Andrew Bate & Jeffery L. Painter & Tim A. Casperson & Gregory Eugene Powell, 2021. "Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation," Drug Safety, Springer, vol. 44(3), pages 373-382, March.
  • Handle: RePEc:spr:drugsa:v:44:y:2021:i:3:d:10.1007_s40264-020-01028-w
    DOI: 10.1007/s40264-020-01028-w
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s40264-020-01028-w
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s40264-020-01028-w?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Landers, Richard N. & Behrend, Tara S., 2015. "An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other Convenience Samples," Industrial and Organizational Psychology, Cambridge University Press, vol. 8(2), pages 142-164, June.
    2. Shaun Comfort & Sujan Perera & Zoe Hudson & Darren Dorrell & Shawman Meireis & Meenakshi Nagarajan & Cartic Ramakrishnan & Jennifer Fine, 2018. "Sorting Through the Safety Data Haystack: Using Machine Learning to Identify Individual Case Safety Reports in Social-Digital Media," Drug Safety, Springer, vol. 41(6), pages 579-590, June.
    3. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Krukowski, Kipp A. & Pollack, Jeffrey M. & Rutherford, Matthew W., 2023. "Winning the opportunity to pitch: Piquing startup investors’ interest by sending the right signals in executive summaries," Business Horizons, Elsevier, vol. 66(1), pages 75-86.
    2. Clarke, Samuel L. & Rhodes, Eric S., 2020. "Entrepreneurial apologies: The mediating role of forgiveness on future cooperation," Journal of Business Venturing Insights, Elsevier, vol. 13(C).
    3. Deena A. Isom & Hunter M. Boehme & Toniqua C. Mikell & Stephen Chicoine & Marion Renner, 2021. "Status Threat, Social Concerns, and Conservative Media: A Look at White America and the Alt-Right," Societies, MDPI, vol. 11(3), pages 1-20, July.
    4. Allison, Thomas H. & Davis, Blakley C. & Webb, Justin W. & Short, Jeremy C., 2017. "Persuasion in crowdfunding: An elaboration likelihood model of crowdfunding performance," Journal of Business Venturing, Elsevier, vol. 32(6), pages 707-725.
    5. Yuhyung Shin & Won-Moo Hur, 2019. "When Do Service Employees Suffer More from Job Insecurity? The Moderating Role of Coworker and Customer Incivility," IJERPH, MDPI, vol. 16(7), pages 1-17, April.
    6. Andrea Gragnano & Silvia Simbula & Massimo Miglioretti, 2020. "Work–Life Balance: Weighing the Importance of Work–Family and Work–Health Balance," IJERPH, MDPI, vol. 17(3), pages 1-20, February.
    7. Nitika Sharma & Arminda Paço & Raysa Geaquinto Rocha & Maria Palazzo & Alfonso Siano, 2024. "Examining a theoretical model of eco‐anxiety on consumers' intentions towards green products," Corporate Social Responsibility and Environmental Management, John Wiley & Sons, vol. 31(3), pages 1868-1885, May.
    8. Indy Wijngaards & Owen C. King & Martijn J. Burger & Job Exel, 2022. "Worker Well-Being: What it Is, and how it Should Be Measured," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 17(2), pages 795-832, April.
    9. Shanmugavel, Nagarajan & Balakrishnan, Janarthanan, 2023. "Influence of pro-environmental behaviour towards behavioural intention of electric vehicles," Technological Forecasting and Social Change, Elsevier, vol. 187(C).
    10. Paul Benjamin Lowry & Jun Zhang & Chuang Wang & Mikko Siponen, 2016. "Why Do Adults Engage in Cyberbullying on Social Media? An Integration of Online Disinhibition and Deindividuation Effects with the Social Structure and Social Learning Model," Information Systems Research, INFORMS, vol. 27(4), pages 962-986, December.
    11. Christopher M. Castille & John E. Buckner & Christian N. Thoroughgood, 2018. "Prosocial Citizens Without a Moral Compass? Examining the Relationship Between Machiavellianism and Unethical Pro-Organizational Behavior," Journal of Business Ethics, Springer, vol. 149(4), pages 919-930, June.
    12. Kristin A. Horan & Mindy K. Shoss & Cynthia Mejia & Katherine Ciarlante, 2021. "Industry Context as an Essential Tool for the Future of Healthy and Safe Work: Illustrative Examples for Occupational Health Psychology from the Hospitality Industry," IJERPH, MDPI, vol. 18(20), pages 1-14, October.
    13. Jones, Shayne & Dinkins, Barbara & Sleep, Chelsea E. & Lynam, Donald R. & Miller, Joshua D., 2021. "The Add Health psychopathy scale: Assessing its construct validity," Journal of Criminal Justice, Elsevier, vol. 72(C).
    14. Shuili Du & Mayowa T. Babalola & Premilla D’Cruz & Edina Dóci & Lucia Garcia-Lorenzo & Louise Hassan & Gazi Islam & Alexander Newman & Ernesto Noronha & Suzanne Gils, 2024. "The Ethical, Societal, and Global Implications of Crowdsourcing Research," Journal of Business Ethics, Springer, vol. 193(1), pages 1-16, August.
    15. Palan, Stefan & Schitter, Christian, 2018. "Prolific.ac—A subject pool for online experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 17(C), pages 22-27.
    16. Tan, Yuxuan & Gong, Yanping & Xie, Julan & Li, Jian & Liu, Yongdan, 2022. "More mindfulness, less conspicuous consumption? Evidence from middle-aged Chinese consumers," Journal of Retailing and Consumer Services, Elsevier, vol. 69(C).
    17. Rahwan, Zoe & Hauser, Oliver P. & Kochanowska, Ewa & Fasolo, Barbara, 2018. "High stakes: A little more cheating, a lot less charity," Journal of Economic Behavior & Organization, Elsevier, vol. 152(C), pages 276-295.
    18. Adrianela Angeles & Adriana Perez-Encinas & Cristian E. Villanueva, 2022. "Characterizing Organizational Lifecycle through Strategic and Structural Flexibility: Insights from MSMEs in Mexico," Global Journal of Flexible Systems Management, Springer;Global Institute of Flexible Systems Management, vol. 23(2), pages 271-290, June.
    19. Will Bryant & Stephanie M. Merritt, 2021. "Unethical Pro-organizational Behavior and Positive Leader–Employee Relationships," Journal of Business Ethics, Springer, vol. 168(4), pages 777-793, February.
    20. Junghyun Lee & Se-Hyung Oh & Sanghee Park, 2022. "Effects of Organizational Embeddedness on Unethical Pro-organizational Behavior: Roles of Perceived Status and Ethical Leadership," Journal of Business Ethics, Springer, vol. 176(1), pages 111-125, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:drugsa:v:44:y:2021:i:3:d:10.1007_s40264-020-01028-w. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com/economics/journal/40264 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.