IDEAS home Printed from https://ideas.repec.org/a/zbw/ifweej/201853.html
   My bibliography  Save this article

Which tests not witch hunts: A diagnostic approach for conducting replication research

Author

Listed:
  • Brown, Annette N.
  • Wood, Benjamin Douglas Kuflick

Abstract

Replication research can be used to explore original study results that researchers consider questionable, but it should also be a tool for reinforcing the credibility of results that are important to policies and programs. The challenge is to design a replication plan open to both supporting the original findings and uncovering potential problems. The purpose of this paper is to provide replication researchers with an objective list of checks or tests to consider when planning a replication study. The authors present tips for diagnostic replication exercises in four groups: validity of assumptions, data transformations, estimation methods, and heterogeneous impacts. For each group, the authors present an introduction to the issues, a list of replication tests and checks, some examples of how these checks are employed in replication studies of development impact evaluations, and a set of resources that provide statistical and econometric details. The authors also provide a list of don'ts for how to conduct and report replication research.

Suggested Citation

  • Brown, Annette N. & Wood, Benjamin Douglas Kuflick, 2018. "Which tests not witch hunts: A diagnostic approach for conducting replication research," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-26.
    • Handle: RePEc:zbw:ifweej:201853
    DOI: 10.5018/economics-ejournal.ja.2018-53
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.5018/economics-ejournal.ja.2018-53
    Download Restriction: no
    File URL: https://www.econstor.eu/bitstream/10419/181452/1/1029195676.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.5018/economics-ejournal.ja.2018-53?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Anderson, Michael L, 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Department of Agricultural & Resource Economics, UC Berkeley, Working Paper Series qt15n8j26f, Department of Agricultural & Resource Economics, UC Berkeley.
    2. David McKenzie, 2017. "How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence," The World Bank Research Observer, World Bank, vol. 32(2), pages 127-154.
    3. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    4. Cummins, Joseph R., 2017. "Heterogeneous treatment effects in the low track: Revisiting the Kenyan primary school experiment," Economics of Education Review, Elsevier, vol. 56(C), pages 40-51.
    5. Nava Ashraf & Xavier Giné & Dean Karlan, 2009. "Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 973-990.
    6. Daron Acemoglu & Simon Johnson & James A. Robinson & Pierre Yared, 2008. "Income and Democracy," American Economic Review, American Economic Association, vol. 98(3), pages 808-842, June.
    7. Robert Jensen & Emily Oster, 2009. "The Power of TV: Cable Television and Women's Status in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 124(3), pages 1057-1094.
    8. Paul J. Gertler & Sebastian Martinez & Patrick Premand & Laura B. Rawlings & Christel M. J. Vermeersch, 2016. "Impact Evaluation in Practice, Second Edition," World Bank Publications - Books, The World Bank Group, number 25030.
    9. Kilic, Talip & Zezza, Alberto & Carletto, Calogero & Savastano, Sara, 2017. "Missing(ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements," World Development, Elsevier, vol. 92(C), pages 143-157.
    10. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    11. repec:diw:diwwpp:dp485 is not listed on IDEAS
    12. W. Robert Reed, 2017. "Replication in labor economics," IZA World of Labor, Institute of Labor Economics (IZA), pages 413-413, December.
    13. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    14. David Roodman, 2009. "A Note on the Theme of Too Many Instruments," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 71(1), pages 135-158, February.
    15. Jesse Rothstein, 2017. "Measuring the Impacts of Teachers: Comment," American Economic Review, American Economic Association, vol. 107(6), pages 1656-1684, June.
    16. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    17. Pradyot Ranjan Jena & Bezawit Beyene Chichaibelu & Till Stellmacher & Ulrike Grote, 2012. "The impact of coffee certification on small-scale producers’ livelihoods: a case study from the Jimma Zone, Ethiopia," Agricultural Economics, International Association of Agricultural Economists, vol. 43(4), pages 429-440, July.
    18. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    19. Christian Zimmermann, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    20. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 175-214.
    21. Matias D. Cattaneo & Sebastian Galiani & Paul J. Gertler & Sebastian Martinez & Rocio Titiunik, 2009. "Housing, Health, and Happiness," American Economic Journal: Economic Policy, American Economic Association, vol. 1(1), pages 75-105, February.
    22. Natalie Carvalho & Slawa Rokicki, 2019. "The Impact of India’s Janani Suraksha Yojana Conditional Cash Transfer Programme: A Replication Study," Journal of Development Studies, Taylor & Francis Journals, vol. 55(5), pages 989-1006, May.
    23. Stefan Dercon & Daniel O. Gilligan & John Hoddinott & Tassew Woldehanna, 2009. "The Impact of Agricultural Extension and Roads on Poverty and Consumption Growth in Fifteen Ethiopian Villages," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 1007-1021.
    24. Galiani, Sebastian & Schargrodsky, Ernesto, 2010. "Property rights for the poor: Effects of land titling," Journal of Public Economics, Elsevier, vol. 94(9-10), pages 700-729, October.
    25. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    26. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2017. "Measuring the Impacts of Teachers: Reply," American Economic Review, American Economic Association, vol. 107(6), pages 1685-1717, June.
    27. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 79(3), pages 933-959.
    28. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    29. Lall, Ranjit, 2016. "How Multiple Imputation Makes a Difference," Political Analysis, Cambridge University Press, vol. 24(4), pages 414-433.
    30. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    31. Imai, Kosuke & Strauss, Aaron, 2011. "Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign," Political Analysis, Cambridge University Press, vol. 19(1), pages 1-19, January.
    32. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    33. Wood, Benjamin D.K. & Dong, Michell, 2015. "Recalling Extra Data: A Replication Study of Finding Missing Markets," 2014: Food, Resources and Conflict, December 7-9, 2014. San Diego, California 206225, International Agricultural Trade Research Consortium.
    34. Sebastian Calonico & Matias D. Cattaneo & Rocío Titiunik, 2015. "Optimal Data-Driven Regression Discontinuity Plots," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1753-1769, December.
    35. David S. Lee & Enrico Moretti & Matthew J. Butler, 2004. "Do Voters Affect or Elect Policies? Evidence from the U. S. House," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(3), pages 807-859.
    36. Patrick Button, 2015. "A Replication of 'Do Voters Affect or Elect Policies? Evidence from the U.S. House' (The Quarterly Journal of Economics, 2004)," Working Papers 1518, Tulane University, Department of Economics.
    37. Edward M. Whitney & Drew B. Cameron & Paul C. Winters, 2019. "Heterogeneous Effects of Urban Land Titling: A Replication of ‘Property Rights for the Poor’," Journal of Development Studies, Taylor & Francis Journals, vol. 55(5), pages 1030-1033, May.
    38. Shahidur R. Khandker & Gayatri B. Koolwal & Hussain A. Samad, . "Handbook on Impact Evaluation : Quantitative Methods and Practices," World Bank Publications, The World Bank, number 2693, September.
    39. Rajeev K. Goel & Ummad Mazhar, 2015. "A Replication of “Corruption and Elections," Public Finance Review, , vol. 43(2), pages 143-154, March.
    40. Matteo Cervellati & Florian Jung & Uwe Sunde & Thomas Vischer, 2014. "Income and Democracy: Comment," American Economic Review, American Economic Association, vol. 104(2), pages 707-719, February.
    41. Sebastian Galiani & Paul Gertler & Mauricio Romero, 2017. "Incentives for Replication in Economics," NBER Working Papers 23576, National Bureau of Economic Research, Inc.
    42. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    43. Ritva Reinikka & Jakob Svensson, 2005. "Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda," Journal of the European Economic Association, MIT Press, vol. 3(2-3), pages 259-267, 04/05.
    44. Sandip Sukhtankar, 2017. "Replications in Development Economics," American Economic Review, American Economic Association, vol. 107(5), pages 32-36, May.
    45. Helland Eric & Tabarrok Alexander, 2004. "Using Placebo Laws to Test "More Guns, Less Crime"," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 4(1), pages 1-9, January.
    46. Ashraf, Nava & Giné, Xavier & Karlan, Dean S., 2009. "AJAE appendix for “Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya”," American Journal of Agricultural Economics APPENDICES, Agricultural and Applied Economics Association, vol. 91(4), pages 1-9, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wood, Benjamin Douglas Kuflick & Vasquez, Maria, 2018. "Microplots and food security: Encouraging replication studies of policy relevant research," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-12.
    2. Reed, W. Robert, 2019. "Takeaways from the special issue on The Practice of Replication," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 13, pages 1-11.
    3. Owen, P. Dorian, 2018. "Replication to assess statistical adequacy," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-16.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    2. Rothstein, Jesse & Von Wachter, Till, 2016. "Social Experiments in the Labor Market," Department of Economics, Working Paper Series qt7957p9g6, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    3. Chakravarty, Shubha & Lundberg, Mattias & Nikolov, Plamen & Zenker, Juliane, 2019. "Vocational training programs and youth labor market outcomes: Evidence from Nepal," Journal of Development Economics, Elsevier, vol. 136(C), pages 71-110.
    4. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    5. Ari Hyytinen & Jaakko Meriläinen & Tuukka Saarimaa & Otto Toivanen & Janne Tukiainen, 2018. "When does regression discontinuity design work? Evidence from random election outcomes," Quantitative Economics, Econometric Society, vol. 9(2), pages 1019-1051, July.
    6. Arouna, Aminou & Michler, Jeffrey D. & Lokossou, Jourdain C., 2021. "Contract farming and rural transformation: Evidence from a field experiment in Benin," Journal of Development Economics, Elsevier, vol. 151(C).
    7. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    8. Brendon McConnell & Marcos Vera-Hernandez, 2015. "Going beyond simple sample size calculations: a practitioner's guide," IFS Working Papers W15/17, Institute for Fiscal Studies.
    9. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    10. Apps, Patricia & Mendolia, Silvia & Walker, Ian, 2013. "The impact of pre-school on adolescents’ outcomes: Evidence from a recent English cohort," Economics of Education Review, Elsevier, vol. 37(C), pages 183-199.
    11. Cygan-Rehm, Kamila & Karbownik, Krzysztof, 2022. "The effects of incentivizing early prenatal care on infant health," Journal of Health Economics, Elsevier, vol. 83(C).
    12. Fitzsimons, Emla & Malde, Bansi & Mesnard, Alice & Vera-Hernández, Marcos, 2016. "Nutrition, information and household behavior: Experimental evidence from Malawi," Journal of Development Economics, Elsevier, vol. 122(C), pages 113-126.
    13. Ashimwe, Olive, 2016. "An Economic Analysis Of Impact Of Weather Index-Based Crop Insurance On Household Income In Huye District Of Rwanda," Research Theses 265675, Collaborative Masters Program in Agricultural and Applied Economics.
    14. Del Prete, Davide & Ghins, Léopold & Magrini, Emiliano & Pauw, Karl, 2019. "Land consolidation, specialization and household diets: Evidence from Rwanda," Food Policy, Elsevier, vol. 83(C), pages 139-149.
    15. Magnan, Nicholas & Hoffmann, Vivian & Opoku, Nelson & Gajate Garrido, Gissele & Kanyam, Daniel Akwasi, 2021. "Information, technology, and market rewards: Incentivizing aflatoxin control in Ghana," Journal of Development Economics, Elsevier, vol. 151(C).
    16. Bjorn Van Campenhout & David J. Spielman & Els Lecoutere, 2021. "Information and Communication Technologies to Provide Agricultural Advice to Smallholder Farmers: Experimental Evidence from Uganda," American Journal of Agricultural Economics, John Wiley & Sons, vol. 103(1), pages 317-337, January.
    17. Pedro Carneiro & Oswald Koussihouèdé & Nathalie Lahire & Costas Meghir & Corina Mommaerts, 2020. "School Grants and Education Quality: Experimental Evidence from Senegal," Economica, London School of Economics and Political Science, vol. 87(345), pages 28-51, January.
    18. Depew, Briggs & Eren, Ozkan, 2016. "Born on the wrong day? School entry age and juvenile crime," Journal of Urban Economics, Elsevier, vol. 96(C), pages 73-90.
    19. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    20. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.

    More about this item

    Keywords

    replication; validation; data transformation; estimation methods; reanalysis; theory of change; assumptions;
    All these keywords.

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • A20 - General Economics and Teaching - - Economic Education and Teaching of Economics - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:ifweej:201853. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/iwkiede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.