IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/9a3rw_v1.html
   My bibliography  Save this paper

Transparency, Reproducibility, and the Credibility of Economics Research

Author

Listed:
  • Christensen, Garret

    (FDIC)

  • Miguel, Edward

    (University of California, Berkeley)

Abstract

There is growing interest in enhancing research transparency and reproducibility in economics and other scientific fields. We survey existing work on these topics within economics, and discuss the evidence suggesting that publication bias, inability to replicate, and specification searching remain widespread in the discipline. We next discuss recent progress in this area, including through improved research design, study registration and pre-analysis plans, disclosure standards, and open sharing of data and materials, drawing on experiences in both economics and other social sciences. We discuss areas where consensus is emerging on new practices, as well as approaches that remain controversial, and speculate about the most effective ways to make economics research more credible in the future.

Suggested Citation

  • Christensen, Garret & Miguel, Edward, 2017. "Transparency, Reproducibility, and the Credibility of Economics Research," MetaArXiv 9a3rw_v1, Center for Open Science.
    • Handle: RePEc:osf:metaar:9a3rw_v1
    DOI: 10.31219/osf.io/9a3rw_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/58b9dafd594d9001ef98141c/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/9a3rw_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. T.D. Stanley & Hristos Doucouliagos, 2010. "Picture This: A Simple Graph That Reveals Much Ado About Research," Journal of Economic Surveys, Wiley Blackwell, vol. 24(1), pages 170-191, February.
    2. Heléne Lundqvist & Matz Dahlberg & Eva Mörk, 2014. "Stimulating Local Public Employment: Do General Grants Work?," American Economic Journal: Economic Policy, American Economic Association, vol. 6(1), pages 167-192, February.
    3. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    4. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    5. Antonio Ciccone, 2013. "Estimating the Effect of Transitory Economic Shocks on Civil Conflict," Review of Economics and Institutions, Università di Perugia, vol. 4(2).
    6. G�nther Fink & Margaret McConnell & Sebastian Vollmer, 2014. "Testing for heterogeneous treatment effects in experimental data: false discovery risks and correction procedures," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(1), pages 44-57, January.
    7. Chris Doucouliagos, 2005. "Publication Bias in the Economic Freedom and Economic Growth Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 367-387, July.
    8. Card, David & Krueger, Alan B, 1994. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania," American Economic Review, American Economic Association, vol. 84(4), pages 772-793, September.
    9. Bellavance, Franois & Dionne, Georges & Lebeau, Martin, 2009. "The value of a statistical life: A meta-analysis with a mixed effects regression model," Journal of Health Economics, Elsevier, vol. 28(2), pages 444-464, March.
    10. Mansfield, Edwin & Schwartz, Mark & Wagner, Samuel, 1981. "Imitation Costs and Patents: An Empirical Study," Economic Journal, Royal Economic Society, vol. 91(364), pages 907-918, December.
    11. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    12. David M. Drukker & Vince Wiggins, 2004. "Verifying the Solution from a Nonlinear Solver: A Case Study: Comment," American Economic Review, American Economic Association, vol. 94(1), pages 397-399, March.
    13. Daron Acemoglu & Simon Johnson & James A. Robinson, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Reply," American Economic Review, American Economic Association, vol. 102(6), pages 3077-3110, October.
    14. Mitch Kunce & Shelby Gerking & William Morgan, 2002. "Effects of Environmental and Land Use Regulation in the Oil and Gas Industry Using the Wyoming Checkerboard as an Experimental Design," American Economic Review, American Economic Association, vol. 92(5), pages 1588-1593, December.
    15. Lucas C. Coffman & Muriel Niederle, 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 81-98, Summer.
    16. Holger Görg & Eric Strobl, 2016. "Multinational Companies And Productivity Spillovers: A Meta-Analysis," World Scientific Book Chapters, in: MULTINATIONAL ENTERPRISES AND HOST COUNTRY DEVELOPMENT Volume 53: World Scientific Studies in International Economics, chapter 8, pages 145-161, World Scientific Publishing Co. Pte. Ltd..
    17. Amy Finkelstein & Sarah Taubman & Bill Wright & Mira Bernstein & Jonathan Gruber & Joseph P. Newhouse & Heidi Allen & Katherine Baicker, 2012. "The Oregon Health Insurance Experiment: Evidence from the First Year," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(3), pages 1057-1106.
    18. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    19. Orley Ashenfelter & Michael Greenstone, 2004. "Estimating the Value of a Statistical Life: The Importance of Omitted Variables and Publication Bias," American Economic Review, American Economic Association, vol. 94(2), pages 454-460, May.
    20. Arindrajit Dube & T. William Lester & Michael Reich, 2010. "Minimum Wage Effects Across State Borders: Estimates Using Contiguous Counties," The Review of Economics and Statistics, MIT Press, vol. 92(4), pages 945-964, November.
    21. W. Kip Viscusi, 2015. "The Role of Publication Selection Bias in Estimates of the Value of a Statistical Life," American Journal of Health Economics, University of Chicago Press, vol. 1(1), pages 27-52, Winter.
    22. Don Husereau & Michael Drummond & Stavros Petrou & Chris Carswell & David Moher & Dan Greenberg & Federico Augustovski & Andrew Briggs & Josephine Mauskopf & Elizabeth Loder, 2013. "Consolidated Health Economic Evaluation Reporting Standards (CHEERS) statement," The European Journal of Health Economics, Springer;Deutsche Gesellschaft für Gesundheitsökonomie (DGGÖ), vol. 14(3), pages 367-372, June.
    23. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    24. Rachel Glennerster & Kudzai Takavarasha, 2013. "Running Randomized Evaluations: A Practical Guide," Economics Books, Princeton University Press, edition 1, number 10085.
    25. Ron Shachar & Barry Nalebuff, 2004. "Verifying the Solution from a Nonlinear Solver: A Case Study: Comment," American Economic Review, American Economic Association, vol. 94(1), pages 382-390, March.
    26. Peter Nijkamp & Jacques Poot, 2005. "The Last Word on the Wage Curve?," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 421-450, July.
    27. Christophe Bontemps & Grayham E. Mizon, 2008. "Encompassing: Concepts and Implementation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 70(s1), pages 721-750, December.
    28. Orley Ashenfelter & Colm Harmon & Hessel Oosterbeek, 1999. "A Review of Estimates of the Schooling/Earnings Relationship, with Tests for Publication Bias," Working Papers 804, Princeton University, Department of Economics, Industrial Relations Section..
    29. Hashem Dezhbakhsh & Paul H. Rubin & Joanna M. Shepherd, 2003. "Does Capital Punishment Have a Deterrent Effect? New Evidence from Postmoratorium Panel Data," American Law and Economics Review, American Law and Economics Association, vol. 5(2), pages 344-376, August.
    30. John Ifcher & Homa Zarghamee, 2011. "Happiness and Time Preference: The Effect of Positive Affect in a Random-Assignment Experiment," American Economic Review, American Economic Association, vol. 101(7), pages 3109-3129, December.
    31. Walter Enders & Gary Hoover, 2006. "Plagiarism in the Economics Profession: A Survey," Challenge, Taylor & Francis Journals, vol. 49(5), pages 92-107.
    32. Katherine Casey & Rachel Glennerster & Edward Miguel, 2012. "Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(4), pages 1755-1812.
    33. Daniele Fanelli, 2009. "How Many Scientists Fabricate and Falsify Research? A Systematic Review and Meta-Analysis of Survey Data," PLOS ONE, Public Library of Science, vol. 4(5), pages 1-11, May.
    34. Gerber, Alan & Arceneaux, Kevin & Boudreau, Cheryl & Dowling, Conor & Hillygus, Sunshine & Palfrey, Thomas & Biggers, Daniel R. & Hendry, David J., 2014. "Reporting Guidelines for Experimental Research: A Report from the Experimental Research Section Standards Committee," Journal of Experimental Political Science, Cambridge University Press, vol. 1(1), pages 81-98, April.
    35. Xavier Sala-I-Martin & Gernot Doppelhofer & Ronald I. Miller, 2004. "Determinants of Long-Term Growth: A Bayesian Averaging of Classical Estimates (BACE) Approach," American Economic Review, American Economic Association, vol. 94(4), pages 813-835, September.
    36. Edward Miguel & Shanker Satyanath & Ernest Sergenti, 2004. "Economic Shocks and Civil Conflict: An Instrumental Variables Approach," Journal of Political Economy, University of Chicago Press, vol. 112(4), pages 725-753, August.
    37. Shelby Gerking & William E. Morgan, 2007. "Effects of Environmental and Land Use Regulation in the Oil and Gas Industry Using the Wyoming Checkerboard as a Natural Experiment: Retraction," American Economic Review, American Economic Association, vol. 97(3), pages 1032-1032, June.
    38. Hunter, John E, 2001. "The Desperate Need for Replications," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 28(1), pages 149-158, June.
    39. Yoav Benjamini & Abba M. Krieger & Daniel Yekutieli, 2006. "Adaptive linear step-up procedures that control the false discovery rate," Biometrika, Biometrika Trust, vol. 93(3), pages 491-507, September.
    40. Hunt Allcott & Dmitry Taubinsky, 2015. "Evaluating Behaviorally Motivated Policy: Experimental Evidence from the Lightbulb Market," American Economic Review, American Economic Association, vol. 105(8), pages 2501-2538, August.
    41. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    42. Benjamin A. Olken, 2015. "Promises and Perils of Pre-analysis Plans," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 61-80, Summer.
    43. Gary A. Hoover, 2004. "Whose Line Is It? Plagiarism in Economics," Journal of Economic Literature, American Economic Association, vol. 42(2), pages 487-493, June.
    44. Mizon, Grayham E & Richard, Jean-Francois, 1986. "The Encompassing Principle and Its Application to Testing Non-nested Hypotheses," Econometrica, Econometric Society, vol. 54(3), pages 657-678, May.
    45. Graham A. Davis, 2013. "Replicating Sachs and Warner's Working Papers on the Resource Curse," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1615-1630, December.
    46. Ziliak, Stephen T. & McCloskey, Deirdre N., 2004. "Size matters: the standard error of regressions in the American Economic Review," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 527-546, November.
    47. Ori Heffetz & Katrina Ligett, 2014. "Privacy and Data-Based Research," Journal of Economic Perspectives, American Economic Association, vol. 28(2), pages 75-98, Spring.
    48. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    49. Hristos Doucouliagos & T. D. Stanley, 2009. "Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 47(2), pages 406-428, June.
    50. Orley Ashenfelter & Michael Greenstone, 2004. "Estimating the Value of a Statistical Life: The Importance of Omitted Variables and Publication Bias," American Economic Review, American Economic Association, vol. 94(2), pages 454-460, May.
    51. Jonathan A. Schwabish, 2014. "An Economist's Guide to Visualizing Data," Journal of Economic Perspectives, American Economic Association, vol. 28(1), pages 209-234, Winter.
    52. Antonio Ciccone, 2011. "Economic Shocks and Civil Conflict: A Comment," American Economic Journal: Applied Economics, American Economic Association, vol. 3(4), pages 215-227, October.
    53. Maria Abreu & Henri L. F. de Groot & Raymond J. G. M. Florax, 2005. "A Meta‐Analysis of β‐Convergence: the Legendary 2%," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 389-420, July.
    54. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    55. Welch, Finis, 1974. "Minimum Wage Legislation in the United States," Economic Inquiry, Western Economic Association International, vol. 12(3), pages 285-318, September.
    56. Leimer, Dean R & Lesnoy, Selig D, 1982. "Social Security and Private Saving: New Time-Series Evidence," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 606-629, June.
    57. B. D. McCullough & H. D. Vinod, 2004. "Verifying the Solution from a Nonlinear Solver: A Case Study: Reply," American Economic Review, American Economic Association, vol. 94(1), pages 400-406, March.
    58. Leamer, Edward E & Leonard, Herman B, 1983. "Reporting the Fragility of Regression Estimates," The Review of Economics and Statistics, MIT Press, vol. 65(2), pages 306-317, May.
    59. Orley Ashenfelter & Michael Greenstone, 2004. "Estimating the Value of a Statistical Life: The Importance of Omitted Variables and Publication Bias," Working Papers 858, Princeton University, Department of Economics, Industrial Relations Section..
    60. Robert MacCoun & Saul Perlmutter, 2015. "Blind analysis: Hide results to seek the truth," Nature, Nature, vol. 526(7572), pages 187-189, October.
    61. Katherine Baicker & Amy Finkelstein & Jae Song & Sarah Taubman, 2014. "The Impact of Medicaid on Labor Market Activity and Program Participation: Evidence from the Oregon Health Insurance Experiment," American Economic Review, American Economic Association, vol. 104(5), pages 322-328, May.
    62. Larry A. Chenault, 1984. "A Note on the Stability Limitations in "A Stable Price Adjustment Process"," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 99(2), pages 385-386.
    63. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    2. Opoku-Agyemang, Kweku A., 2017. "A Human-Computer Interaction Approach for Integrity in Economics," SocArXiv ra3cs, Center for Open Science.
    3. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    4. Cristina Blanco-Perez & Abel Brodeur, 2020. "Publication Bias and Editorial Statement on Negative Findings," The Economic Journal, Royal Economic Society, vol. 130(629), pages 1226-1247.
    5. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    6. Stanley, T.D. & Doucouliagos, Chris & Jarrell, Stephen B., 2008. "Meta-regression analysis as the socio-economics of economics research," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 37(1), pages 276-292, February.
    7. Brodeur, Abel & Blanco-Perez, Cristina, 2017. "Publication Bias and Editorial Statement on Negative Findings," MetaArXiv xq9nt_v1, Center for Open Science.
    8. Marek Rusnak & Tomas Havranek & Roman Horvath, 2013. "How to Solve the Price Puzzle? A Meta‐Analysis," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 45(1), pages 37-70, February.
    9. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    10. Havranek, Tomas & Irsova, Zuzana & Janda, Karel, 2012. "Demand for gasoline is more price-inelastic than commonly thought," Energy Economics, Elsevier, vol. 34(1), pages 201-207.
    11. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    12. Alex Eble & Peter Boone & Diana Elbourne, 2017. "On Minimizing the Risk of Bias in Randomized Controlled Trials in Economics," The World Bank Economic Review, World Bank, vol. 31(3), pages 687-707.
    13. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    14. Chris Doucouliagos & T.D. Stanley, 2013. "Are All Economic Facts Greatly Exaggerated? Theory Competition And Selectivity," Journal of Economic Surveys, Wiley Blackwell, vol. 27(2), pages 316-339, April.
    15. Amin Haghnejad & Mahboobeh Farahati, 2024. "An overview of meta-analytic methods for economic research," Papers 2412.10608, arXiv.org, revised Mar 2025.
    16. Havranek, Tomas & Irsova, Zuzana, 2011. "Estimating vertical spillovers from FDI: Why results vary and what the true effect is," Journal of International Economics, Elsevier, vol. 85(2), pages 234-244.
    17. Roman Horvath & Ali Elminejad & Tomas Havranek, 2020. "Publication and Identification Biases in Measuring the Intertemporal Substitution of Labor Supply," Working Papers IES 2020/32, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Sep 2020.
    18. Burlig, Fiona, 2018. "Improving transparency in observational social science research: A pre-analysis plan approach," Economics Letters, Elsevier, vol. 168(C), pages 56-60.
    19. Tomas Havranek, 2013. "Publication Bias in Measuring Intertemporal Substitution," Working Papers IES 2013/15, Charles University Prague, Faculty of Social Sciences, Institute of Economic Studies, revised Oct 2013.
    20. Jarle Møen & Helge Sandvig Thorsen, 2017. "Publication Bias in the Returns to R&D Literature," Journal of the Knowledge Economy, Springer;Portland International Center for Management of Engineering and Technology (PICMET), vol. 8(3), pages 987-1013, September.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:9a3rw_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.