IDEAS home Printed from https://ideas.repec.org/a/plo/pbio00/3000125.html
   My bibliography  Save this article

Developing a modern data workflow for regularly updated data

Author

Listed:
  • Glenda M Yenni
  • Erica M Christensen
  • Ellen K Bledsoe
  • Sarah R Supp
  • Renata M Diaz
  • Ethan P White
  • S K Morgan Ernest

Abstract

Over the past decade, biology has undergone a data revolution in how researchers collect data and the amount of data being collected. An emerging challenge that has received limited attention in biology is managing, working with, and providing access to data under continual active collection. Regularly updated data present unique challenges in quality assurance and control, data publication, archiving, and reproducibility. We developed a workflow for a long-term ecological study that addresses many of the challenges associated with managing this type of data. We do this by leveraging existing tools to 1) perform quality assurance and control; 2) import, restructure, version, and archive data; 3) rapidly publish new data in ways that ensure appropriate credit to all contributors; and 4) automate most steps in the data pipeline to reduce the time and effort required by researchers. The workflow leverages tools from software development, including version control and continuous integration, to create a modern data management system that automates the pipeline.This Community Page article describes a data management workflow that can be readily implemented by small research teams and which solves the core challenges of managing regularly updating data. It includes a template repository and tutorial to assist others in setting up their own regularly updating data management systems.

Suggested Citation

  • Glenda M Yenni & Erica M Christensen & Ellen K Bledsoe & Sarah R Supp & Renata M Diaz & Ethan P White & S K Morgan Ernest, 2019. "Developing a modern data workflow for regularly updated data," PLOS Biology, Public Library of Science, vol. 17(1), pages 1-12, January.
  • Handle: RePEc:plo:pbio00:3000125
    DOI: 10.1371/journal.pbio.3000125
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3000125
    Download Restriction: no

    File URL: https://journals.plos.org/plosbiology/article/file?id=10.1371/journal.pbio.3000125&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pbio.3000125?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Greg Wilson & D A Aruliah & C Titus Brown & Neil P Chue Hong & Matt Davis & Richard T Guy & Steven H D Haddock & Kathryn D Huff & Ian M Mitchell & Mark D Plumbley & Ben Waugh & Ethan P White & Paul Wi, 2014. "Best Practices for Scientific Computing," PLOS Biology, Public Library of Science, vol. 12(1), pages 1-7, January.
    2. Yasset Perez-Riverol & Laurent Gatto & Rui Wang & Timo Sachsenberg & Julian Uszkoreit & Felipe da Veiga Leprevost & Christian Fufezan & Tobias Ternent & Stephen J Eglen & Daniel S Katz & Tom J Pollard, 2016. "Ten Simple Rules for Taking Advantage of Git and GitHub," PLOS Computational Biology, Public Library of Science, vol. 12(7), pages 1-11, July.
    3. Jennifer C Molloy, 2011. "The Open Knowledge Foundation: Open Data Means Better Science," PLOS Biology, Public Library of Science, vol. 9(12), pages 1-4, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Benjamin D Lee, 2018. "Ten simple rules for documenting scientific software," PLOS Computational Biology, Public Library of Science, vol. 14(12), pages 1-6, December.
    2. John D Blischak & Emily R Davenport & Greg Wilson, 2016. "A Quick Introduction to Version Control with Git and GitHub," PLOS Computational Biology, Public Library of Science, vol. 12(1), pages 1-18, January.
    3. Serghei Mangul & Thiago Mosqueiro & Richard J Abdill & Dat Duong & Keith Mitchell & Varuni Sarwal & Brian Hill & Jaqueline Brito & Russell Jared Littman & Benjamin Statz & Angela Ka-Mei Lam & Gargi Da, 2019. "Challenges and recommendations to improve the installability and archival stability of omics computational tools," PLOS Biology, Public Library of Science, vol. 17(6), pages 1-16, June.
    4. Sibele Fausto & Fabio A Machado & Luiz Fernando J Bento & Atila Iamarino & Tatiana R Nahas & David S Munger, 2012. "Research Blogging: Indexing and Registering the Change in Science 2.0," PLOS ONE, Public Library of Science, vol. 7(12), pages 1-10, December.
    5. Sean C Anderson & Cole C Monnahan & Kelli F Johnson & Kotaro Ono & Juan L Valero, 2014. "ss3sim: An R Package for Fisheries Stock Assessment Simulation with Stock Synthesis," PLOS ONE, Public Library of Science, vol. 9(4), pages 1-7, April.
    6. Markus List & Peter Ebert & Felipe Albrecht, 2017. "Ten Simple Rules for Developing Usable Software in Computational Biology," PLOS Computational Biology, Public Library of Science, vol. 13(1), pages 1-5, January.
    7. Daniel Nüst & Vanessa Sochat & Ben Marwick & Stephen J Eglen & Tim Head & Tony Hirst & Benjamin D Evans, 2020. "Ten simple rules for writing Dockerfiles for reproducible data science," PLOS Computational Biology, Public Library of Science, vol. 16(11), pages 1-24, November.
    8. Bastian Greshake & Philipp E Bayer & Helge Rausch & Julia Reda, 2014. "openSNP–A Crowdsourced Web Resource for Personal Genomics," PLOS ONE, Public Library of Science, vol. 9(3), pages 1-9, March.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pbio00:3000125. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosbiology (email available below). General contact details of provider: https://journals.plos.org/plosbiology/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.