IDEAS home Printed from https://ideas.repec.org/a/nat/natcom/v12y2021i1d10.1038_s41467-021-27222-7.html
   My bibliography  Save this article

Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks

Author

Listed:
  • Julia Koehler Leman

    (Flatiron Institute, Simons Foundation
    New York University)

  • Sergey Lyskov

    (Johns Hopkins University)

  • Steven M. Lewis

    (Cyrus Biotechnology)

  • Jared Adolf-Bryfogle

    (Scripps Research
    IAVI Neutralizing Antibody Center, Scripps Research)

  • Rebecca F. Alford

    (Johns Hopkins University)

  • Kyle Barlow

    (University of California San Francisco)

  • Ziv Ben-Aharon

    (Hebrew University, Hadassah Medical School)

  • Daniel Farrell

    (University of Washington
    University of Washington)

  • Jason Fell

    (University of California
    University of California
    University of California)

  • William A. Hansen

    (Rutgers, The State University of New Jersey
    Rutgers, The State University of New Jersey)

  • Ameya Harmalkar

    (Johns Hopkins University)

  • Jeliazko Jeliazkov

    (Johns Hopkins University)

  • Georg Kuenze

    (Vanderbilt University
    Vanderbilt University
    Medical School, Leipzig University)

  • Justyna D. Krys

    (University of Warsaw, Pasteura 1)

  • Ajasja Ljubetič

    (University of Washington
    University of Washington)

  • Amanda L. Loshbaugh

    (University of California San Francisco
    University of California San Francisco)

  • Jack Maguire

    (University of North Carolina at Chapel Hill)

  • Rocco Moretti

    (Vanderbilt University
    Vanderbilt University)

  • Vikram Khipple Mulligan

    (Flatiron Institute, Simons Foundation)

  • Morgan L. Nance

    (Johns Hopkins University)

  • Phuong T. Nguyen

    (University of California)

  • Shane Ó Conchúir

    (University of California San Francisco)

  • Shourya S. Roy Burman

    (Johns Hopkins University)

  • Rituparna Samanta

    (Johns Hopkins University)

  • Shannon T. Smith

    (Vanderbilt University
    Vanderbilt University)

  • Frank Teets

    (University of North Carolina at Chapel Hill)

  • Johanna K. S. Tiemann

    (University of Copenhagen)

  • Andrew Watkins

    (Stanford University School of Medicine)

  • Hope Woods

    (Vanderbilt University
    Vanderbilt University)

  • Brahm J. Yachnin

    (Rutgers, The State University of New Jersey
    Rutgers, The State University of New Jersey)

  • Christopher D. Bahl

    (Institute for Protein Innovation
    Boston Children’s Hospital
    Harvard Medical School)

  • Chris Bailey-Kellogg

    (Dartmouth)

  • David Baker

    (University of Washington
    University of Washington)

  • Rhiju Das

    (Stanford University School of Medicine)

  • Frank DiMaio

    (University of Washington
    University of Washington)

  • Sagar D. Khare

    (Rutgers, The State University of New Jersey
    Rutgers, The State University of New Jersey)

  • Tanja Kortemme

    (University of California San Francisco
    University of California San Francisco)

  • Jason W. Labonte

    (Johns Hopkins University)

  • Kresten Lindorff-Larsen

    (University of Copenhagen)

  • Jens Meiler

    (Vanderbilt University
    Vanderbilt University
    Medical School, Leipzig University)

  • William Schief

    (Scripps Research
    IAVI Neutralizing Antibody Center, Scripps Research)

  • Ora Schueler-Furman

    (Hebrew University, Hadassah Medical School)

  • Justin B. Siegel

    (University of California
    University of California
    University of California)

  • Amelie Stein

    (University of Copenhagen)

  • Vladimir Yarov-Yarovoy

    (University of California)

  • Brian Kuhlman

    (University of North Carolina at Chapel Hill)

  • Andrew Leaver-Fay

    (University of North Carolina at Chapel Hill)

  • Dominik Gront

    (University of Warsaw, Pasteura 1)

  • Jeffrey J. Gray

    (Johns Hopkins University)

  • Richard Bonneau

    (Flatiron Institute, Simons Foundation
    New York University
    New York University)

Abstract

Each year vast international resources are wasted on irreproducible research. The scientific community has been slow to adopt standard software engineering practices, despite the increases in high-dimensional data, complexities of workflows, and computational environments. Here we show how scientific software applications can be created in a reproducible manner when simple design goals for reproducibility are met. We describe the implementation of a test server framework and 40 scientific benchmarks, covering numerous applications in Rosetta bio-macromolecular modeling. High performance computing cluster integration allows these benchmarks to run continuously and automatically. Detailed protocol captures are useful for developers and users of Rosetta and other macromolecular modeling tools. The framework and design concepts presented here are valuable for developers and users of any type of scientific software and for the scientific community to create reproducible methods. Specific examples highlight the utility of this framework, and the comprehensive documentation illustrates the ease of adding new tests in a matter of hours.

Suggested Citation

  • Julia Koehler Leman & Sergey Lyskov & Steven M. Lewis & Jared Adolf-Bryfogle & Rebecca F. Alford & Kyle Barlow & Ziv Ben-Aharon & Daniel Farrell & Jason Fell & William A. Hansen & Ameya Harmalkar & Je, 2021. "Ensuring scientific reproducibility in bio-macromolecular modeling via extensive, automated benchmarks," Nature Communications, Nature, vol. 12(1), pages 1-15, December.
    • Handle: RePEc:nat:natcom:v:12:y:2021:i:1:d:10.1038_s41467-021-27222-7
    DOI: 10.1038/s41467-021-27222-7
    as

    Download full text from publisher

    File URL: https://www.nature.com/articles/s41467-021-27222-7
    File Function: Abstract
    Download Restriction: no
    File URL: https://libkey.io/10.1038/s41467-021-27222-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jeffrey M. Perkel, 2020. "Challenge to scientists: does your ten-year-old code still run?," Nature, Nature, vol. 584(7822), pages 656-658, August.
    2. Aroop Sircar & Jeffrey J Gray, 2010. "SnugDock: Paratope Structural Optimization during Antibody-Antigen Docking Compensates for Errors in Antibody Homology Models," PLOS Computational Biology, Public Library of Science, vol. 6(1), pages 1-13, January.
    3. Gaurav Bhardwaj & Vikram Khipple Mulligan & Christopher D. Bahl & Jason M. Gilmore & Peta J. Harvey & Olivier Cheneval & Garry W. Buchko & Surya V. S. R. K. Pulavarti & Quentin Kaas & Alexander Eletsk, 2016. "Accurate de novo design of hyperstable constrained peptides," Nature, Nature, vol. 538(7625), pages 329-335, October.
    4. Leonard P Freedman & Iain M Cockburn & Timothy S Simcoe, 2015. "The Economics of Reproducibility in Preclinical Research," PLOS Biology, Public Library of Science, vol. 13(6), pages 1-9, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Thillai V. Sekar & Eslam A. Elghonaimy & Katy L. Swancutt & Sebastian Diegeler & Isaac Gonzalez & Cassandra Hamilton & Peter Q. Leung & Jens Meiler & Cristina E. Martina & Michael Whitney & Todd A. Ag, 2023. "Simultaneous selection of nanobodies for accessible epitopes on immune cells in the tumor microenvironment," Nature Communications, Nature, vol. 14(1), pages 1-20, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dean A Fergusson & Marc T Avey & Carly C Barron & Mathew Bocock & Kristen E Biefer & Sylvain Boet & Stephane L Bourque & Isidora Conic & Kai Chen & Yuan Yi Dong & Grace M Fox & Ronald B George & Neil , 2019. "Reporting preclinical anesthesia study (REPEAT): Evaluating the quality of reporting in the preclinical anesthesiology literature," PLOS ONE, Public Library of Science, vol. 14(5), pages 1-15, May.
    2. Seibold, Heidi & Charlton, Alethea & Boulesteix, Anne-Laure & Hoffmann, Sabine, 2020. "Statisticians roll up your sleeves! There’s a crisis to be solved," MetaArXiv frta7, Center for Open Science.
    3. Watzinger, Martin & Schnitzer, Monika, 2019. "Standing on the Shoulders of Science," Rationality and Competition Discussion Paper Series 215, CRC TRR 190 Rationality and Competition.
    4. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    5. Vivian Leung & Frédérik Rousseau-Blass & Guy Beauchamp & Daniel S J Pang, 2018. "ARRIVE has not ARRIVEd: Support for the ARRIVE (Animal Research: Reporting of in vivo Experiments) guidelines does not improve the reporting quality of papers in animal welfare, analgesia or anesthesi," PLOS ONE, Public Library of Science, vol. 13(5), pages 1-13, May.
    6. Thillai V. Sekar & Eslam A. Elghonaimy & Katy L. Swancutt & Sebastian Diegeler & Isaac Gonzalez & Cassandra Hamilton & Peter Q. Leung & Jens Meiler & Cristina E. Martina & Michael Whitney & Todd A. Ag, 2023. "Simultaneous selection of nanobodies for accessible epitopes on immune cells in the tumor microenvironment," Nature Communications, Nature, vol. 14(1), pages 1-20, December.
    7. Massih Khorvash & Nick Blinov & Carol Ladner-Keay & Jie Lu & Judith M Silverman & Ebrima Gibbs & Yu Tian Wang & Andriy Kovalenko & David Wishart & Neil R Cashman, 2020. "Molecular interactions between monoclonal oligomer-specific antibody 5E3 and its amyloid beta cognates," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-35, May.
    8. Jordan R Willis & Bryan S Briney & Samuel L DeLuca & James E Crowe Jr & Jens Meiler, 2013. "Human Germline Antibody Gene Segments Encode Polyspecific Antibodies," PLOS Computational Biology, Public Library of Science, vol. 9(4), pages 1-14, April.
    9. Kiri, Bralind & Lacetera, Nicola & Zirulia, Lorenzo, 2018. "Above a swamp: A theory of high-quality scientific production," Research Policy, Elsevier, vol. 47(5), pages 827-839.
    10. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    11. Malika Ihle & Isabel S. Winney & Anna Krystalli & Michael Croucher, 2017. "Striving for transparent and credible research: practical guidelines for behavioral ecologists," Behavioral Ecology, International Society for Behavioral Ecology, vol. 28(2), pages 348-354.
    12. Bernhard Voelkl & Lucile Vogt & Emily S Sena & Hanno Würbel, 2018. "Reproducibility of preclinical animal research improves with heterogeneity of study samples," PLOS Biology, Public Library of Science, vol. 16(2), pages 1-13, February.
    13. Muradchanian, Jasmine & Hoekstra, Rink & Kiers, Henk & van Ravenzwaaij, Don, 2020. "How Best to Quantify Replication Success? A Simulation Study on the Comparison of Replication Success Metrics," MetaArXiv wvdjf, Center for Open Science.
    14. Wang, Xuefeng & Zhang, Shuo & Liu, Yuqin & Du, Jian & Huang, Heng, 2021. "How pharmaceutical innovation evolves: The path from science to technological development to marketable drugs," Technological Forecasting and Social Change, Elsevier, vol. 167(C).
    15. Martin Backfisch, 2018. "The Development of Firm Size and Innovativeness in the Pharmaceutical industry between 1989 and 2010," MAGKS Papers on Economics 201813, Philipps-Universität Marburg, Faculty of Business Administration and Economics, Department of Economics (Volkswirtschaftliche Abteilung).
    16. Matthias Steinfath & Silvia Vogl & Norman Violet & Franziska Schwarz & Hans Mielke & Thomas Selhorst & Matthias Greiner & Gilbert Schönfelder, 2018. "Simple changes of individual studies can improve the reproducibility of the biomedical scientific process as a whole," PLOS ONE, Public Library of Science, vol. 13(9), pages 1-20, September.
    17. Christopher Allen & David M A Mehler, 2019. "Open science challenges, benefits and tips in early career and beyond," PLOS Biology, Public Library of Science, vol. 17(5), pages 1-14, May.
    18. Jeff Miller & Rolf Ulrich, 2019. "The quest for an optimal alpha," PLOS ONE, Public Library of Science, vol. 14(1), pages 1-13, January.
    19. Jorge Roel-Touris & Marta Nadal & Enrique Marcos, 2023. "Single-chain dimers from de novo immunoglobulins as robust scaffolds for multiple binding loops," Nature Communications, Nature, vol. 14(1), pages 1-15, December.
    20. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nat:natcom:v:12:y:2021:i:1:d:10.1038_s41467-021-27222-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.nature.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.