IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0100647.html
   My bibliography  Save this article

Using Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example

Author

Listed:
  • Andrew W Brown
  • David B Allison

Abstract

Systematically evaluating scientific literature is a time consuming endeavor that requires hours of coding and rating. Here, we describe a method to distribute these tasks across a large group through online crowdsourcing. Using Amazon's Mechanical Turk, crowdsourced workers (microworkers) completed four groups of tasks to evaluate the question, “Do nutrition-obesity studies with conclusions concordant with popular opinion receive more attention in the scientific community than do those that are discordant?” 1) Microworkers who passed a qualification test (19% passed) evaluated abstracts to determine if they were about human studies investigating nutrition and obesity. Agreement between the first two raters' conclusions was moderate (κ = 0.586), with consensus being reached in 96% of abstracts. 2) Microworkers iteratively synthesized free-text answers describing the studied foods into one coherent term. Approximately 84% of foods were agreed upon, with only 4 and 8% of ratings failing manual review in different steps. 3) Microworkers were asked to rate the perceived obesogenicity of the synthesized food terms. Over 99% of responses were complete and usable, and opinions of the microworkers qualitatively matched the authors' expert expectations (e.g., sugar-sweetened beverages were thought to cause obesity and fruits and vegetables were thought to prevent obesity). 4) Microworkers extracted citation counts for each paper through Google Scholar. Microworkers reached consensus or unanimous agreement for all successful searches. To answer the example question, data were aggregated and analyzed, and showed no significant association between popular opinion and attention the paper received as measured by Scimago Journal Rank and citation counts. Direct microworker costs totaled $221.75, (estimated cost at minimum wage: $312.61). We discuss important points to consider to ensure good quality control and appropriate pay for microworkers. With good reliability and low cost, crowdsourcing has potential to evaluate published literature in a cost-effective, quick, and reliable manner using existing, easily accessible resources.

Suggested Citation

  • Andrew W Brown & David B Allison, 2014. "Using Crowdsourcing to Evaluate Published Scientific Literature: Methods and Example," PLOS ONE, Public Library of Science, vol. 9(7), pages 1-9, July.
  • Handle: RePEc:plo:pone00:0100647
    DOI: 10.1371/journal.pone.0100647
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0100647
    Download Restriction: no

    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0100647&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pone.0100647?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Anne-Wil Harzing, 2013. "A preliminary test of Google Scholar as a source for citation data: a longitudinal study of Nobel prize winners," Scientometrics, Springer;Akadémiai Kiadó, vol. 94(3), pages 1057-1075, March.
    2. Amélie Yavchitz & Isabelle Boutron & Aida Bafeta & Ibrahim Marroun & Pierre Charles & Jean Mantz & Philippe Ravaud, 2012. "Misrepresentation of Randomized Controlled Trials in Press Releases and News Coverage: A Cohort Study," PLOS Medicine, Public Library of Science, vol. 9(9), pages 1-11, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nianhang Xu & Winnie P. H. Poon & Kam C. Chan, 2014. "Contributing Institutions and Authors in International Business Research: A Quality-Based Assessment," Management International Review, Springer, vol. 54(5), pages 735-755, October.
    2. Anne-Wil Harzing & Satu Alakangas, 2017. "Microsoft Academic is one year old: the Phoenix is ready to leave the nest," Scientometrics, Springer;Akadémiai Kiadó, vol. 112(3), pages 1887-1894, September.
    3. Jan Recker, 2013. "Of Hygiene and Motivator Factors: Views from “Down Under”," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 5(4), pages 287-288, August.
    4. Thomas W. Sanchez, 2021. "Urban Planning Academics: Tweets and Citations," Urban Planning, Cogitatio Press, vol. 6(1), pages 146-153.
    5. Chris Fields, 2015. "Close to the edge: co-authorship proximity of Nobel laureates in Physiology or Medicine, 1991–2010, to cross-disciplinary brokers," Scientometrics, Springer;Akadémiai Kiadó, vol. 103(1), pages 267-299, April.
    6. Anne-Wil Harzing & Wilfred Mijnhardt, 2015. "Proof over promise: towards a more inclusive ranking of Dutch academics in Economics & Business," Scientometrics, Springer;Akadémiai Kiadó, vol. 102(1), pages 727-749, January.
    7. Martin-Martin, Alberto & Orduna-Malea, Enrique & Harzing, Anne-Wil & Delgado López-Cózar, Emilio, 2017. "Can we use Google Scholar to identify highly-cited documents?," Journal of Informetrics, Elsevier, vol. 11(1), pages 152-163.
    8. Clément Bosquet & Pierre-Philippe Combes, 2013. "Are academics who publish more also more cited? Individual determinants of publication and citation records," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 831-857, December.
    9. Anne-Wil Harzing, 2014. "A longitudinal study of Google Scholar coverage between 2012 and 2013," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 565-575, January.
    10. Helene Tenzer & Siri Terjesen & Anne-Wil Harzing, 2017. "Language in International Business: A Review and Agenda for Future Research," Management International Review, Springer, vol. 57(6), pages 815-854, December.
    11. Samreen Ayaz & Muhammad Tanvir Afzal, 2016. "Identification of conversion factor for completing-h index for the field of mathematics," Scientometrics, Springer;Akadémiai Kiadó, vol. 109(3), pages 1511-1524, December.
    12. Russell Smyth & Vinod Mishra, 2014. "Academic inbreeding and research productivity and impact in Australian law schools," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(1), pages 583-618, January.
    13. Cohen, Scott & Stienmetz, Jason & Hanna, Paul & Humbracht, Michael & Hopkins, Debbie, 2020. "Shadowcasting tourism knowledge through media: Self-driving sex cars?," Annals of Tourism Research, Elsevier, vol. 85(C).
    14. Romana Haneef & Clement Lazarus & Philippe Ravaud & Amélie Yavchitz & Isabelle Boutron, 2015. "Interpretation of Results of Studies Evaluating an Intervention Highlighted in Google Health News: A Cross-Sectional Study of News," PLOS ONE, Public Library of Science, vol. 10(10), pages 1-15, October.
    15. Ilenia Ascani & Roberta Ciccola & Maria Serena Chiucchi, 2021. "A Structured Literature Review about the Role of Management Accountants in Sustainability Accounting and Reporting," Sustainability, MDPI, vol. 13(4), pages 1-25, February.
    16. John Mingers & Jesse R. O’Hanley & Musbaudeen Okunola, 2017. "Using Google Scholar institutional level data to evaluate the quality of university research," Scientometrics, Springer;Akadémiai Kiadó, vol. 113(3), pages 1627-1643, December.
    17. Joseph W Taylor & Marie Long & Elizabeth Ashley & Alex Denning & Beatrice Gout & Kayleigh Hansen & Thomas Huws & Leifa Jennings & Sinead Quinn & Patrick Sarkies & Alex Wojtowicz & Philip M Newton, 2015. "When Medical News Comes from Press Releases—A Case Study of Pancreatic Cancer and Processed Meat," PLOS ONE, Public Library of Science, vol. 10(6), pages 1-13, June.
    18. Enrique Orduña-Malea & Emilio Delgado López-Cózar, 2014. "Google Scholar Metrics evolution: an analysis according to languages," Scientometrics, Springer;Akadémiai Kiadó, vol. 98(3), pages 2353-2367, March.
    19. Betancourt, Nathan & Jochem, Torsten & Otner, Sarah M.G., 2023. "Standing on the shoulders of giants: How star scientists influence their coauthors," Research Policy, Elsevier, vol. 52(1).
    20. Keshra Sangwal, 2013. "Some citation-related characteristics of scientific journals published in individual countries," Scientometrics, Springer;Akadémiai Kiadó, vol. 97(3), pages 719-741, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0100647. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.