IDEAS home Printed from https://ideas.repec.org/p/osf/thesis/kd6ja.html
   My bibliography  Save this paper

Statistical methods for evidence synthesis

Author

Listed:
  • Mathur, Maya B
  • VanderWeele, Tyler

Abstract

In many empirical disciplines, scientific discovery is modularized into discrete papers each investigating one or more hypotheses. Synthesizing these modules of evidence is critical to inform a balanced and appropriately evolving view of the overall evidence on a topic as well as to identify where substantial uncertainty remains. This dissertation considers three realms in which such synthesis can occur: (1) when meta-analyzing multiple studies; (2) when subjecting a single study to independent replications; and (3) when testing related hypotheses within a study. We consider specific methodological challenges within each of these realms and propose statistical methods to address each. All proposed methods are implemented in R packages.

Suggested Citation

  • Mathur, Maya B & VanderWeele, Tyler, 2018. "Statistical methods for evidence synthesis," Thesis Commons kd6ja, Center for Open Science.
    • Handle: RePEc:osf:thesis:kd6ja
    DOI: 10.31219/osf.io/kd6ja
    as

    Download full text from publisher

    File URL: https://osf.io/download/5b6ef3f890a7e0001886c4fd/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/kd6ja?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    2. Guildo W. Imbens, 2003. "Sensitivity to Exogeneity Assumptions in Program Evaluation," American Economic Review, American Economic Association, vol. 93(2), pages 126-132, May.
    3. Mathur, Maya B & VanderWeele, Tyler, 2017. "Sensitivity analysis for unmeasured confounding in meta-analyses," OSF Preprints jkhfg, Center for Open Science.
    4. Kurex Sidik & Jeffrey N. Jonkman, 2005. "Simple heterogeneity variance estimation for meta‐analysis," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 54(2), pages 367-384, April.
    5. Joseph P. Romano & Michael Wolf, "undated". "Control of Generalized Error Rates in Multiple Testing," IEW - Working Papers 245, Institute for Empirical Research in Economics - University of Zurich.
    6. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March.
    7. Viechtbauer, Wolfgang, 2010. "Conducting Meta-Analyses in R with the metafor Package," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i03).
    8. Anna, Petrenko, 2016. "Мaркування готової продукції як складова частина інформаційного забезпечення маркетингової діяльності підприємств овочепродуктового підкомплексу," Agricultural and Resource Economics: International Scientific E-Journal, Agricultural and Resource Economics: International Scientific E-Journal, vol. 2(01), March.
    9. Isaiah Andrews & Maximilian Kasy, 2019. "Identification of and Correction for Publication Bias," American Economic Review, American Economic Association, vol. 109(8), pages 2766-2794, August.
    10. N. J. Welton & A. E. Ades & J. B. Carlin & D. G. Altman & J. A. C. Sterne, 2009. "Models for potentially biased evidence in meta‐analysis using empirically based priors," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(1), pages 119-136, January.
    11. Alberto Abadie & Guido W. Imbens, 2008. "On the Failure of the Bootstrap for Matching Estimators," Econometrica, Econometric Society, vol. 76(6), pages 1537-1557, November.
    12. Keyes, C.L.M. & Simoes, E.J., 2012. "To flourish or not: Positive mental health and all-cause mortality," American Journal of Public Health, American Public Health Association, vol. 102(11), pages 2164-2172.
    13. van der Laan Mark J. & Birkner Merrill D. & Hubbard Alan E., 2005. "Empirical Bayes and Resampling Based Multiple Testing Procedure Controlling Tail Probability of the Proportion of False Positives," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 4(1), pages 1-32, October.
    14. Lynch, John G. & Bradlow, Eric T. & Huber, Joel C. & Lehmann, Donald R., 2015. "Reflections on the replication corner: In praise of conceptual replications," International Journal of Research in Marketing, Elsevier, vol. 32(4), pages 333-342.
    15. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    16. Alexander Etz & Joachim Vandekerckhove, 2016. "A Bayesian Perspective on the Reproducibility Project: Psychology," PLOS ONE, Public Library of Science, vol. 11(2), pages 1-12, February.
    17. van der Laan Mark J. & Dudoit Sandrine & Pollard Katherine S., 2004. "Augmentation Procedures for Control of the Generalized Family-Wise Error Rate and Tail Probabilities for the Proportion of False Positives," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 3(1), pages 1-27, June.
    18. Mark van der Laan & Sandrine Dudoit & Katherine Pollard, 2004. "Multiple Testing. Part III. Procedures for Control of the Generalized Family-Wise Error Rate and Proportion of False Positives," U.C. Berkeley Division of Biostatistics Working Paper Series 1140, Berkeley Electronic Press.
    19. Gavin B Stewart & Douglas G Altman & Lisa M Askie & Lelia Duley & Mark C Simmonds & Lesley A Stewart, 2012. "Statistical Analysis of Individual Participant Data Meta-Analyses: A Comparison of Methods and Recommendations for Practice," PLOS ONE, Public Library of Science, vol. 7(10), pages 1-8, October.
    20. Daniel Vaughan-Whitehead, 2014. "Introduction," Economia & lavoro, Carocci editore, issue 2, pages 5-8.
    21. McCandless Lawrence C., 2012. "Meta-Analysis of Observational Studies with Unmeasured Confounders," The International Journal of Biostatistics, De Gruyter, vol. 8(2), pages 1-31, January.
    22. Rebecca M. Turner & David J. Spiegelhalter & Gordon C. S. Smith & Simon G. Thompson, 2009. "Bias modelling in evidence synthesis," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 172(1), pages 21-47, January.
    23. Friguet, Chloé & Kloareg, Maela & Causeur, David, 2009. "A Factor Model Approach to Multiple Testing Under Dependence," Journal of the American Statistical Association, American Statistical Association, vol. 104(488), pages 1406-1415.
    24. Moulton, Lawrence H. & Zeger, Scott L., 1991. "Bootstrapping generalized linear models," Computational Statistics & Data Analysis, Elsevier, vol. 11(1), pages 53-63, January.
    25. Wenguang Sun & T. Tony Cai, 2009. "Large‐scale multiple testing under dependence," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 71(2), pages 393-424, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mathur, Maya B & VanderWeele, Tyler J., 2018. "Statistical methods for evidence synthesis," Thesis Commons kd6ja_v1, Center for Open Science.
    2. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    3. Pallavi Basu & Luella Fu & Alessio Saretto & Wenguang Sun, 2021. "Empirical Bayes Control of the False Discovery Exceedance," Working Papers 2115, Federal Reserve Bank of Dallas.
    4. Joseph P. Romano & Michael Wolf, 2008. "Balanced Control of Generalized Error Rates," IEW - Working Papers 379, Institute for Empirical Research in Economics - University of Zurich.
    5. Maya B. Mathur & Tyler J. VanderWeele, 2020. "New statistical metrics for multisite replication projects," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 183(3), pages 1145-1166, June.
    6. Wang, Li & Xu, Xingzhong, 2012. "Step-up procedure controlling generalized family-wise error rate," Statistics & Probability Letters, Elsevier, vol. 82(4), pages 775-782.
    7. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    8. Heath, Davidson & Ringgenberg, Matthew C. & Samadi, Mehrdad & Werner, Ingrid M., 2019. "Reusing Natural Experiments," Working Paper Series 2019-21, Ohio State University, Charles A. Dice Center for Research in Financial Economics.
    9. Joseph Romano & Azeem Shaikh & Michael Wolf, 2008. "Control of the false discovery rate under dependence using the bootstrap and subsampling," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(3), pages 417-442, November.
    10. Romano, Joseph P. & Shaikh, Azeem M. & Wolf, Michael, 2008. "Formalized Data Snooping Based On Generalized Error Rates," Econometric Theory, Cambridge University Press, vol. 24(2), pages 404-447, April.
    11. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    12. Christina C. Bartenschlager & Michael Krapp, 2015. "Theorie und Methoden multipler statistischer Vergleiche," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 9(2), pages 107-129, November.
    13. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    14. Sandner, Malte & Cornelissen, Thomas & Jungmann, Tanja & Herrmann, Peggy, 2018. "Evaluating the effects of a targeted home visiting program on maternal and child health outcomes," Journal of Health Economics, Elsevier, vol. 58(C), pages 269-283.
    15. Schumi Jennifer & DiRienzo A. Gregory & DeGruttola Victor, 2008. "Testing for Associations with Missing High-Dimensional Categorical Covariates," The International Journal of Biostatistics, De Gruyter, vol. 4(1), pages 1-19, September.
    16. Alessio Farcomeni, 2009. "Generalized Augmentation to Control the False Discovery Exceedance in Multiple Testing," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 36(3), pages 501-517, September.
    17. Rute M. Caeiro & Pedro C. Vicente, 2020. "Knowledge of vitamin A deficiency and crop adoption: Evidence from a field experiment in Mozambique," Agricultural Economics, International Association of Agricultural Economists, vol. 51(2), pages 175-190, March.
    18. Alexander Frankel & Maximilian Kasy, 2022. "Which Findings Should Be Published?," American Economic Journal: Microeconomics, American Economic Association, vol. 14(1), pages 1-38, February.
    19. Jaschke Philipp & Sulin Sardoschau & Marco Tabellini, 2021. "Scared Straight? Threat and Assimilation of Refugees in Germany," RF Berlin - CReAM Discussion Paper Series 2136, Rockwool Foundation Berlin (RF Berlin) - Centre for Research and Analysis of Migration (CReAM).
    20. Grácio, Matilde & Vicente, Pedro C., 2021. "Information, get-out-the-vote messages, and peer influence: Causal effects on political behavior in Mozambique," Journal of Development Economics, Elsevier, vol. 151(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:thesis:kd6ja. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://thesiscommons.org .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.