IDEAS home Printed from https://ideas.repec.org/a/spr/drugsa/v44y2021i3d10.1007_s40264-020-01028-w.html
   My bibliography  Save this article

Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation

Author

Listed:
  • Alex Gartland

    (University of Central Florida)

  • Andrew Bate

    (GlaxoSmithKline)

  • Jeffery L. Painter

    (JiveCast)

  • Tim A. Casperson

    (GlaxoSmithKline)

  • Gregory Eugene Powell

    (GlaxoSmithKline)

Abstract

Introduction Machine learning offers an alluring solution to developing automated approaches to the increasing individual case safety report burden being placed upon pharmacovigilance. Leveraging crowdsourcing to annotate unstructured data may provide accurate, efficient, and contemporaneous training data sets in support of machine learning. Objective The objective of this study was to evaluate whether crowdsourcing can be used to accurately and efficiently develop training data sets in support of pharmacovigilance automation. Materials and Methods Pharmacovigilance experts created a reference dataset by reviewing 15,490 de-identified social media posts of narratives pertaining to 15 drugs and 22 medically relevant topics. A random sampling of posts from the reference dataset was published on Amazon Turk and its users (Turkers) were asked a series of questions about those same medical concepts. Accuracy, price elasticity, and time efficiency were evaluated. Results Accuracy of crowdsourced curation exceeded 90% when compared to the reference dataset and was completed in about 5% of the time. There was an increase in time efficiency with higher pay, but there was no significant difference in accuracy. Additionally, having a social media post reviewed by more than one Turker (using a voting system) did not offer significant improvements in terms of accuracy. Conclusions Crowdsourcing is an accurate and efficient method that can be used to develop training data sets in support of pharmacovigilance automation. More research is needed to better understand the breadth and depth of possible uses as well as strengths, limitations, and generalizability of results.

Suggested Citation

  • Alex Gartland & Andrew Bate & Jeffery L. Painter & Tim A. Casperson & Gregory Eugene Powell, 2021. "Developing Crowdsourced Training Data Sets for Pharmacovigilance Intelligent Automation," Drug Safety, Springer, vol. 44(3), pages 373-382, March.
    • Handle: RePEc:spr:drugsa:v:44:y:2021:i:3:d:10.1007_s40264-020-01028-w
    DOI: 10.1007/s40264-020-01028-w
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s40264-020-01028-w
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s40264-020-01028-w?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Landers, Richard N. & Behrend, Tara S., 2015. "An Inconvenient Truth: Arbitrary Distinctions Between Organizational, Mechanical Turk, and Other Convenience Samples," Industrial and Organizational Psychology, Cambridge University Press, vol. 8(2), pages 142-164, June.
    2. repec:cup:judgdm:v:5:y:2010:i:5:p:411-419 is not listed on IDEAS
    3. Shaun Comfort & Sujan Perera & Zoe Hudson & Darren Dorrell & Shawman Meireis & Meenakshi Nagarajan & Cartic Ramakrishnan & Jennifer Fine, 2018. "Sorting Through the Safety Data Haystack: Using Machine Learning to Identify Individual Case Safety Reports in Social-Digital Media," Drug Safety, Springer, vol. 41(6), pages 579-590, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Clarke, Samuel L. & Rhodes, Eric S., 2020. "Entrepreneurial apologies: The mediating role of forgiveness on future cooperation," Journal of Business Venturing Insights, Elsevier, vol. 13(C).
    2. Deena A. Isom & Hunter M. Boehme & Toniqua C. Mikell & Stephen Chicoine & Marion Renner, 2021. "Status Threat, Social Concerns, and Conservative Media: A Look at White America and the Alt-Right," Societies, MDPI, vol. 11(3), pages 1-20, July.
    3. Allison, Thomas H. & Davis, Blakley C. & Webb, Justin W. & Short, Jeremy C., 2017. "Persuasion in crowdfunding: An elaboration likelihood model of crowdfunding performance," Journal of Business Venturing, Elsevier, vol. 32(6), pages 707-725.
    4. Indy Wijngaards & Owen C. King & Martijn J. Burger & Job Exel, 2022. "Worker Well-Being: What it Is, and how it Should Be Measured," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 17(2), pages 795-832, April.
    5. Shanmugavel, Nagarajan & Balakrishnan, Janarthanan, 2023. "Influence of pro-environmental behaviour towards behavioural intention of electric vehicles," Technological Forecasting and Social Change, Elsevier, vol. 187(C).
    6. Paul Benjamin Lowry & Jun Zhang & Chuang Wang & Mikko Siponen, 2016. "Why Do Adults Engage in Cyberbullying on Social Media? An Integration of Online Disinhibition and Deindividuation Effects with the Social Structure and Social Learning Model," Information Systems Research, INFORMS, vol. 27(4), pages 962-986, December.
    7. Christopher M. Castille & John E. Buckner & Christian N. Thoroughgood, 2018. "Prosocial Citizens Without a Moral Compass? Examining the Relationship Between Machiavellianism and Unethical Pro-Organizational Behavior," Journal of Business Ethics, Springer, vol. 149(4), pages 919-930, June.
    8. Tan, Yuxuan & Gong, Yanping & Xie, Julan & Li, Jian & Liu, Yongdan, 2022. "More mindfulness, less conspicuous consumption? Evidence from middle-aged Chinese consumers," Journal of Retailing and Consumer Services, Elsevier, vol. 69(C).
    9. Adrianela Angeles & Adriana Perez-Encinas & Cristian E. Villanueva, 2022. "Characterizing Organizational Lifecycle through Strategic and Structural Flexibility: Insights from MSMEs in Mexico," Global Journal of Flexible Systems Management, Springer;Global Institute of Flexible Systems Management, vol. 23(2), pages 271-290, June.
    10. Will Bryant & Stephanie M. Merritt, 2021. "Unethical Pro-organizational Behavior and Positive Leader–Employee Relationships," Journal of Business Ethics, Springer, vol. 168(4), pages 777-793, February.
    11. Junghyun Lee & Se-Hyung Oh & Sanghee Park, 2022. "Effects of Organizational Embeddedness on Unethical Pro-organizational Behavior: Roles of Perceived Status and Ethical Leadership," Journal of Business Ethics, Springer, vol. 176(1), pages 111-125, February.
    12. Frederiks, Arjan J. & Englis, Basil G. & Ehrenhard, Michel L. & Groen, Aard J., 2019. "Entrepreneurial cognition and the quality of new venture ideas: An experimental approach to comparing future-oriented cognitive processes," Journal of Business Venturing, Elsevier, vol. 34(2), pages 327-347.
    13. Bissan Audeh & Florelle Bellet & Marie-Noëlle Beyens & Agnès Lillo-Le Louët & Cédric Bousquet, 2020. "Use of Social Media for Pharmacovigilance Activities: Key Findings and Recommendations from the Vigi4Med Project," Drug Safety, Springer, vol. 43(9), pages 835-851, September.
    14. Kristina S. Weißmüller & Lode Waele, 2022. "Would you Bribe your Lecturer? A Quasi-experimental Study on Burnout and Bribery in Higher Education," Research in Higher Education, Springer;Association for Institutional Research, vol. 63(5), pages 768-796, August.
    15. Andrew Bate & Steve F. Hobbiger, 2021. "Artificial Intelligence, Real-World Automation and the Safety of Medicines," Drug Safety, Springer, vol. 44(2), pages 125-132, February.
    16. Radtke, Robin R. & Speklé, Roland F. & Widener, Sally K., 2023. "Flourish or flounder: Do trust-centric management controls encourage knowledge sharing and team performance?," Accounting, Organizations and Society, Elsevier, vol. 107(C).
    17. James N. Maples & Michael J. Bradley, 2021. "Outdoor Recreation and Rural Transitions in Central Appalachia: Revisiting the Economic Impact of Rock Climbing in Kentucky's Red River Gorge," Journal of Economic Impact, Science Impact Publishers, vol. 3(3), pages 186-195.
    18. Richard N. Landers & Elena M. Auer & Andrew B. Collmus & Michael B. Armstrong, 2018. "Gamification Science, Its History and Future: Definitions and a Research Agenda," Simulation & Gaming, , vol. 49(3), pages 315-337, June.
    19. Trevor M. Spoelma & Nitya Chawla & Aleksander P. J. Ellis, 2021. "If You Can’t Join ‘Em, Report ‘Em: A Model of Ostracism and Whistleblowing in Teams," Journal of Business Ethics, Springer, vol. 173(2), pages 345-363, October.
    20. Kappes, Heather Barry & Balcetis, Emily & De Cremer, David, 2018. "Motivated reasoning during recruitment," LSE Research Online Documents on Economics 84093, London School of Economics and Political Science, LSE Library.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:drugsa:v:44:y:2021:i:3:d:10.1007_s40264-020-01028-w. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com/economics/journal/40264 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.