IDEAS home Printed from https://ideas.repec.org/a/plo/pbio00/3001009.html
   My bibliography  Save this article

Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research

Author

Listed:
  • Takuji Usui
  • Malcolm R Macleod
  • Sarah K McCann
  • Alistair M Senior
  • Shinichi Nakagawa

Abstract

The replicability of research results has been a cause of increasing concern to the scientific community. The long-held belief that experimental standardization begets replicability has also been recently challenged, with the observation that the reduction of variability within studies can lead to idiosyncratic, lab-specific results that cannot be replicated. An alternative approach is to, instead, deliberately introduce heterogeneity, known as “heterogenization” of experimental design. Here, we explore a novel perspective in the heterogenization program in a meta-analysis of variability in observed phenotypic outcomes in both control and experimental animal models of ischemic stroke. First, by quantifying interindividual variability across control groups, we illustrate that the amount of heterogeneity in disease state (infarct volume) differs according to methodological approach, for example, in disease induction methods and disease models. We argue that such methods may improve replicability by creating diverse and representative distribution of baseline disease state in the reference group, against which treatment efficacy is assessed. Second, we illustrate how meta-analysis can be used to simultaneously assess efficacy and stability (i.e., mean effect and among-individual variability). We identify treatments that have efficacy and are generalizable to the population level (i.e., low interindividual variability), as well as those where there is high interindividual variability in response; for these, latter treatments translation to a clinical setting may require nuance. We argue that by embracing rather than seeking to minimize variability in phenotypic outcomes, we can motivate the shift toward heterogenization and improve both the replicability and generalizability of preclinical research.A meta-analysis study of the variability in phenotypic outcomes in both control and experimental animal models of ischaemic stroke provides novel perspectives in which heterogeneity can be embraced to improve the reproducibility and translation of preclinical studies.

Suggested Citation

  • Takuji Usui & Malcolm R Macleod & Sarah K McCann & Alistair M Senior & Shinichi Nakagawa, 2021. "Meta-analysis of variation suggests that embracing variability improves both replicability and generalizability in preclinical research," PLOS Biology, Public Library of Science, vol. 19(5), pages 1-20, May.
  • Handle: RePEc:plo:pbio00:3001009
    DOI: 10.1371/journal.pbio.3001009
    as

    Download full text from publisher

    File URL: https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3001009
    Download Restriction: no

    File URL: https://journals.plos.org/plosbiology/article/file?id=10.1371/journal.pbio.3001009&type=printable
    Download Restriction: no

    File URL: https://libkey.io/10.1371/journal.pbio.3001009?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    2. van Buuren, Stef & Groothuis-Oudshoorn, Karin, 2011. "mice: Multivariate Imputation by Chained Equations in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 45(i03).
    3. Bernhard Voelkl & Lucile Vogt & Emily S Sena & Hanno Würbel, 2018. "Reproducibility of preclinical animal research improves with heterogeneity of study samples," PLOS Biology, Public Library of Science, vol. 16(2), pages 1-13, February.
    4. Constantin Volkmann & Alexander Volkmann & Christian A Müller, 2020. "On the treatment effect heterogeneity of antidepressants in major depression: A Bayesian meta-analysis and simulation study," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-22, November.
    5. Janine A. Clayton & Francis S. Collins, 2014. "Policy: NIH to balance sex in cell and animal studies," Nature, Nature, vol. 509(7500), pages 282-283, May.
    6. Cara Tannenbaum & Robert P. Ellis & Friederike Eyssel & James Zou & Londa Schiebinger, 2019. "Sex and gender analysis improves science and engineering," Nature, Nature, vol. 575(7781), pages 137-146, November.
    7. Nicholas J. Schork, 2015. "Personalized medicine: Time for one-person trials," Nature, Nature, vol. 520(7549), pages 609-611, April.
    8. Natasha A Karp & Anneliese O Speak & Jacqueline K White & David J Adams & Martin Hrabé de Angelis & Yann Hérault & Richard F Mott, 2014. "Impact of Temporal Variation on Design and Analysis of Mouse Knockout Phenotyping Studies," PLOS ONE, Public Library of Science, vol. 9(10), pages 1-10, October.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sadri, Arash, 2022. "The Ultimate Cause of the “Reproducibility Crisis”: Reductionist Statistics," MetaArXiv yxba5, Center for Open Science.
    2. Laura A. B. Wilson & Susanne R. K. Zajitschek & Malgorzata Lagisz & Jeremy Mason & Hamed Haselimashhadi & Shinichi Nakagawa, 2022. "Sex differences in allometry for phenotypic traits in mice indicate that females are not scaled males," Nature Communications, Nature, vol. 13(1), pages 1-12, December.
    3. Sadri, Arash, 2022. "Machine Learning Can Solve the Reproducibility Crisis by Supplanting Reductionist Statistics," MetaArXiv yxba5_v1, Center for Open Science.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bernhard Voelkl & Lucile Vogt & Emily S Sena & Hanno Würbel, 2018. "Reproducibility of preclinical animal research improves with heterogeneity of study samples," PLOS Biology, Public Library of Science, vol. 16(2), pages 1-13, February.
    2. Ballering, Aranka V. & Bonvanie, Irma J. & Olde Hartman, Tim C. & Monden, Rei & Rosmalen, Judith G.M., 2020. "Gender and sex independently associate with common somatic symptoms and lifetime prevalence of chronic disease," Social Science & Medicine, Elsevier, vol. 253(C).
    3. Lori van den Hurk & Sarah Hiltner & Sabine Oertelt-Prigione, 2022. "Operationalization and Reporting Practices in Manuscripts Addressing Gender Differences in Biomedical Research: A Cross-Sectional Bibliographical Study," IJERPH, MDPI, vol. 19(21), pages 1-13, November.
    4. Bettina Bert & Céline Heinl & Justyna Chmielewska & Franziska Schwarz & Barbara Grune & Andreas Hensel & Matthias Greiner & Gilbert Schönfelder, 2019. "Refining animal research: The Animal Study Registry," PLOS Biology, Public Library of Science, vol. 17(10), pages 1-12, October.
    5. Gillian L Currie & Helena N Angel-Scott & Lesley Colvin & Fala Cramond & Kaitlyn Hair & Laila Khandoker & Jing Liao & Malcolm Macleod & Sarah K McCann & Rosie Morland & Nicki Sherratt & Robert Stewart, 2019. "Animal models of chemotherapy-induced peripheral neuropathy: A machine-assisted systematic review and meta-analysis," PLOS Biology, Public Library of Science, vol. 17(5), pages 1-34, May.
    6. Abhilash Bandam & Eedris Busari & Chloi Syranidou & Jochen Linssen & Detlef Stolten, 2022. "Classification of Building Types in Germany: A Data-Driven Modeling Approach," Data, MDPI, vol. 7(4), pages 1-23, April.
    7. Boonstra Philip S. & Little Roderick J.A. & West Brady T. & Andridge Rebecca R. & Alvarado-Leiton Fernanda, 2021. "A Simulation Study of Diagnostics for Selection Bias," Journal of Official Statistics, Sciendo, vol. 37(3), pages 751-769, September.
    8. Jyotirmoy Sarkar, 2018. "Will P†Value Triumph over Abuses and Attacks?," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 7(4), pages 66-71, July.
    9. Christopher J Greenwood & George J Youssef & Primrose Letcher & Jacqui A Macdonald & Lauryn J Hagg & Ann Sanson & Jenn Mcintosh & Delyse M Hutchinson & John W Toumbourou & Matthew Fuller-Tyszkiewicz &, 2020. "A comparison of penalised regression methods for informing the selection of predictive markers," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-14, November.
    10. Norah Alyabs & Sy Han Chiou, 2022. "The Missing Indicator Approach for Accelerated Failure Time Model with Covariates Subject to Limits of Detection," Stats, MDPI, vol. 5(2), pages 1-13, May.
    11. Eunsil Seok & Akhgar Ghassabian & Yuyan Wang & Mengling Liu, 2024. "Statistical Methods for Modeling Exposure Variables Subject to Limit of Detection," Statistics in Biosciences, Springer;International Chinese Statistical Association, vol. 16(2), pages 435-458, July.
    12. Ida Kubiszewski & Kenneth Mulder & Diane Jarvis & Robert Costanza, 2022. "Toward better measurement of sustainable development and wellbeing: A small number of SDG indicators reliably predict life satisfaction," Sustainable Development, John Wiley & Sons, Ltd., vol. 30(1), pages 139-148, February.
    13. Georges Steffgen & Philipp E. Sischka & Martha Fernandez de Henestrosa, 2020. "The Quality of Work Index and the Quality of Employment Index: A Multidimensional Approach of Job Quality and Its Links to Well-Being at Work," IJERPH, MDPI, vol. 17(21), pages 1-31, October.
    14. Christopher Kath & Florian Ziel, 2018. "The value of forecasts: Quantifying the economic gains of accurate quarter-hourly electricity price forecasts," Papers 1811.08604, arXiv.org.
    15. Kevin J. Boyle & Mark Morrison & Darla Hatton MacDonald & Roderick Duncan & John Rose, 2016. "Investigating Internet and Mail Implementation of Stated-Preference Surveys While Controlling for Differences in Sample Frames," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 64(3), pages 401-419, July.
    16. J M van Niekerk & M C Vos & A Stein & L M A Braakman-Jansen & A F Voor in ‘t holt & J E W C van Gemert-Pijnen, 2020. "Risk factors for surgical site infections using a data-driven approach," PLOS ONE, Public Library of Science, vol. 15(10), pages 1-14, October.
    17. Stefkovics, Ádám & Krekó, Péter & Koltai, Júlia, 2024. "When reality knocks on the door. The effect of conspiracy beliefs on COVID-19 vaccine acceptance and the moderating role of experience with the virus," Social Science & Medicine, Elsevier, vol. 356(C).
    18. Joost R. Ginkel, 2020. "Standardized Regression Coefficients and Newly Proposed Estimators for $${R}^{{2}}$$R2 in Multiply Imputed Data," Psychometrika, Springer;The Psychometric Society, vol. 85(1), pages 185-205, March.
    19. Jelte M Wicherts & Marjan Bakker & Dylan Molenaar, 2011. "Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-7, November.
    20. Gerko Vink & Laurence E. Frank & Jeroen Pannekoek & Stef Buuren, 2014. "Predictive mean matching imputation of semicontinuous variables," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 68(1), pages 61-90, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:plo:pbio00:3001009. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: plosbiology (email available below). General contact details of provider: https://journals.plos.org/plosbiology/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.