IDEAS home Printed from https://ideas.repec.org/a/plo/pone00/0251194.html
   My bibliography  Save this article

A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses

Author

Listed:
  • Heidi Seibold
  • Severin Czerny
  • Siona Decke
  • Roman Dieterle
  • Thomas Eder
  • Steffen Fohr
  • Nico Hahn
  • Rabea Hartmann
  • Christoph Heindl
  • Philipp Kopper
  • Dario Lepke
  • Verena Loidl
  • Maximilian Mandl
  • Sarah Musiol
  • Jessica Peter
  • Alexander Piehler
  • Elio Rojas
  • Stefanie Schmid
  • Hannah Schmidt
  • Melissa Schmoll
  • Lennart Schneider
  • Xiao-Yin To
  • Viet Tran
  • Antje Völker
  • Moritz Wagner
  • Joshua Wagner
  • Maria Waize
  • Hannah Wecker
  • Rui Yang
  • Simone Zellner
  • Malte Nalenz

Abstract

Computational reproducibility is a corner stone for sound and credible research. Especially in complex statistical analyses—such as the analysis of longitudinal data—reproducing results is far from simple, especially if no source code is available. In this work we aimed to reproduce analyses of longitudinal data of 11 articles published in PLOS ONE. Inclusion criteria were the availability of data and author consent. We investigated the types of methods and software used and whether we were able to reproduce the data analysis using open source software. Most articles provided overview tables and simple visualisations. Generalised Estimating Equations (GEEs) were the most popular statistical models among the selected articles. Only one article used open source software and only one published part of the analysis code. Replication was difficult in most cases and required reverse engineering of results or contacting the authors. For three articles we were not able to reproduce the results, for another two only parts of them. For all but two articles we had to contact the authors to be able to reproduce the results. Our main learning is that reproducing papers is difficult if no code is supplied and leads to a high burden for those conducting the reproductions. Open data policies in journals are good, but to truly boost reproducibility we suggest adding open code policies.

Suggested Citation

  • Heidi Seibold & Severin Czerny & Siona Decke & Roman Dieterle & Thomas Eder & Steffen Fohr & Nico Hahn & Rabea Hartmann & Christoph Heindl & Philipp Kopper & Dario Lepke & Verena Loidl & Maximilian Ma, 2021. "A computational reproducibility study of PLOS ONE articles featuring longitudinal data analyses," PLOS ONE, Public Library of Science, vol. 16(6), pages 1-15, June.
    • Handle: RePEc:plo:pone00:0251194
    DOI: 10.1371/journal.pone.0251194
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0251194
    Download Restriction: no
    File URL: https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0251194&type=printable
    Download Restriction: no
    File URL: https://libkey.io/10.1371/journal.pone.0251194?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jessica L Couture & Rachael E Blake & Gavin McDonald & Colette L Ward, 2018. "A funder-imposed data publication requirement seldom inspired data sharing," PLOS ONE, Public Library of Science, vol. 13(7), pages 1-13, July.
    2. Kevin V Lemley & Serena M Bagnasco & Cynthia C Nast & Laura Barisoni & Catherine M Conway & Stephen M Hewitt & Peter X K Song, 2016. "Morphometry Predicts Early GFR Change in Primary Proteinuric Glomerulopathies: A Longitudinal Cohort Study Using Generalized Estimating Equations," PLOS ONE, Public Library of Science, vol. 11(6), pages 1-13, June.
    3. Victoria Stodden & Jennifer Seiler & Zhaokun Ma, 2018. "An empirical analysis of journal policy effectiveness for computational reproducibility," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(11), pages 2584-2589, March.
    4. Martí Casals & Montserrat Girabent-Farrés & Josep L Carrasco, 2014. "Methodological Quality and Reporting of Generalized Linear Mixed Models in Clinical Medicine (2000–2012): A Systematic Review," PLOS ONE, Public Library of Science, vol. 9(11), pages 1-10, November.
    5. Esther Maassen & Marcel A L M van Assen & Michèle B Nuijten & Anton Olsson-Collentine & Jelte M Wicherts, 2020. "Reproducibility of individual effect sizes in meta-analyses in psychology," PLOS ONE, Public Library of Science, vol. 15(5), pages 1-18, May.
    6. Maria Vivien Visaya & David Sherwell & Benn Sartorius & Fabien Cromieres, 2015. "Analysis of Binary Multivariate Longitudinal Data via 2-Dimensional Orbits: An Application to the Agincourt Health and Socio-Demographic Surveillance System in South Africa," PLOS ONE, Public Library of Science, vol. 10(4), pages 1-27, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Piers Steel & Sjoerd Beugelsdijk & Herman Aguinis, 2021. "The anatomy of an award-winning meta-analysis: Recommendations for authors, reviewers, and readers of meta-analytic reviews," Journal of International Business Studies, Palgrave Macmillan;Academy of International Business, vol. 52(1), pages 23-44, February.
    2. Troy J. Bouffard & Ekaterina Uryupova & Klaus Dodds & Vladimir E. Romanovsky & Alec P. Bennett & Dmitry Streletskiy, 2021. "Scientific Cooperation: Supporting Circumpolar Permafrost Monitoring and Data Sharing," Land, MDPI, vol. 10(6), pages 1-17, June.
    3. Antonio Páez, 2021. "Open spatial sciences: an introduction," Journal of Geographical Systems, Springer, vol. 23(4), pages 467-476, October.
    4. Byron C. Jaeger & Lloyd J. Edwards & Kalyan Das & Pranab K. Sen, 2017. "An statistic for fixed effects in the generalized linear mixed model," Journal of Applied Statistics, Taylor & Francis Journals, vol. 44(6), pages 1086-1105, April.
    5. Rat für Sozial- und Wirtschaftsdaten RatSWD (ed.), 2023. "Erhebung und Nutzung unstrukturierter Daten in den Sozial-, Verhaltens- und Wirtschaftswissenschaften," RatSWD Output Series, German Data Forum (RatSWD), volume 7, number 7-2de.
    6. Monica Billio & Roberto Casarin & Matteo Iacopini, 2024. "Bayesian Markov-Switching Tensor Regression for Time-Varying Networks," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 119(545), pages 109-121, January.
    7. Anders Wimo & Katrin Seeher & Rodrigo Cataldi & Eva Cyhlarova & Joseph L Dielemann & Oskar Frisell & Maëlenn Guerchet & Linus Jönsson & Angeladine Kenne Malaha & Emma Nichols & Paola Pedroza & Martin , 2023. "The worldwide costs of dementia in 2019," Post-Print ird-03944615, HAL.
    8. Sara Stoudt & Váleri N Vásquez & Ciera C Martinez, 2021. "Principles for data analysis workflows," PLOS Computational Biology, Public Library of Science, vol. 17(3), pages 1-26, March.
    9. Bettina Suhr & Johanna Dungl & Alexander Stocker, 2020. "Search, reuse and sharing of research data in materials science and engineering—A qualitative interview study," PLOS ONE, Public Library of Science, vol. 15(9), pages 1-26, September.
    10. Nikolas I Krieger & Adam T Perzynski & Jarrod E Dalton, 2019. "Facilitating reproducible project management and manuscript development in team science: The projects R package," PLOS ONE, Public Library of Science, vol. 14(7), pages 1-9, July.
    11. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    12. Felix Holzmeister & Magnus Johannesson & Robert Böhm & Anna Dreber & Jürgen Huber & Michael Kirchler, 2023. "Heterogeneity in effect size estimates: Empirical evidence and practical implications," Working Papers 2023-17, Faculty of Economics and Statistics, Universität Innsbruck.
    13. Thu-Mai Christian & Amanda Gooch & Todd Vision & Elizabeth Hull, 2020. "Journal data policies: Exploring how the understanding of editors and authors corresponds to the policies themselves," PLOS ONE, Public Library of Science, vol. 15(3), pages 1-15, March.
    14. Shawn J Leroux, 2019. "On the prevalence of uninformative parameters in statistical models applying model selection in applied ecology," PLOS ONE, Public Library of Science, vol. 14(2), pages 1-12, February.
    15. Vlaeminck, Sven, 2021. "Dawning of a New Age? Economics Journals’ Data Policies on the Test Bench," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 31(1), pages 1-29.
    16. Ruiz-Benito, Paloma & Vacchiano, Giorgio & Lines, Emily R. & Reyer, Christopher P.O. & Ratcliffe, Sophia & Morin, Xavier & Hartig, Florian & Mäkelä, Annikki & Yousefpour, Rasoul & Chaves, Jimena E. & , 2020. "Available and missing data to model impact of climate change on European forests," Ecological Modelling, Elsevier, vol. 416(C).
    17. Christophe Hurlin & Christophe Pérignon, 2020. "Reproducibility Certification in Economics Research," Working Papers hal-02896404, HAL.
    18. Wimo, Anders & Seeher, Katrin & Cataldi, Rodrigo & Cyhlarova, Eva & Dielemann, Joseph L. & Frisell, Oskar & Guerchet, Maëlenn & Jönsson, Linus & Malaha, Angeladine Kenne & Nichols, Emma & Pedroza, Pao, 2023. "The worldwide costs of dementia in 2019," LSE Research Online Documents on Economics 118062, London School of Economics and Political Science, LSE Library.
    19. Adler, Susanne Jana & Sharma, Pratyush Nidhi & Radomir, Lăcrămioara, 2023. "Toward open science in PLS-SEM: Assessing the state of the art and future perspectives," Journal of Business Research, Elsevier, vol. 169(C).
    20. Jessica L Couture & Rachael E Blake & Gavin McDonald & Colette L Ward, 2018. "A funder-imposed data publication requirement seldom inspired data sharing," PLOS ONE, Public Library of Science, vol. 13(7), pages 1-13, July.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pone00:0251194. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosone (email available below). General contact details of provider: https://journals.plos.org/plosone/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.