IDEAS home Printed from https://ideas.repec.org/a/eee/jobhdp/v164y2021icp192-202.html
   My bibliography  Save this article

Expanding the scope of reproducibility research through data analysis replications

Author

Listed:
  • Hofman, Jake M.
  • Goldstein, Daniel G.
  • Sen, Siddhartha
  • Poursabzi-Sangdeh, Forough
  • Allen, Jennifer
  • Dong, Ling Liang
  • Fried, Brenda
  • Gaur, Harpreet
  • Hoq, Adnan
  • Mbazor, Emeka
  • Moreira, Naomi
  • Muso, Cindy
  • Rapp, Etta
  • Terrero, Roymil

Abstract

In recent years, researchers in several scientific disciplines have become concerned with published studies replicating less often than expected. A positive side effect of this concern is an appreciation that replicating other researchers’ work is an essential part of the scientific process. To date, many such efforts have come from the experimental sciences, where replication entails running new experiments, generating new data, and analyzing it. In this article, we emphasize not experimental replications but data analysis replications. We do so for three reasons. First, experimental replication excludes entire classes of publications that do not run experiments or even collect original data (e.g., archival data analysis). Second, experimental replication may in some cases be a needlessly high bar: there is great value in replicating just the data analyses of published experimental work. As data analysis replications require a lower investment of resources than experimental replications, their adoption should expand the number and variety of scientific reproducibility studies undertaken. Third, we propose that teaching undergraduate students to perform data analysis replications will greatly increase the number of replications done while providing them with research experience that should inform their decisions to pursue research or to attend graduate school. Towards this end, we provide details of a pilot program we created to teach undergraduates the skills necessary to conduct data analysis replications, and include a case study of the first set of students who completed this program and attempted to replicate the data analyses in a widely-cited social science paper on policing. In addition, we present a summary of ten additional data analysis replications carried out entirely by students in a university course.

Suggested Citation

  • Hofman, Jake M. & Goldstein, Daniel G. & Sen, Siddhartha & Poursabzi-Sangdeh, Forough & Allen, Jennifer & Dong, Ling Liang & Fried, Brenda & Gaur, Harpreet & Hoq, Adnan & Mbazor, Emeka & Moreira, Naom, 2021. "Expanding the scope of reproducibility research through data analysis replications," Organizational Behavior and Human Decision Processes, Elsevier, vol. 164(C), pages 192-202.
  • Handle: RePEc:eee:jobhdp:v:164:y:2021:i:c:p:192-202
    DOI: 10.1016/j.obhdp.2020.11.003
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0749597820304076
    Download Restriction: Full text for ScienceDirect subscribers only

    File URL: https://libkey.io/10.1016/j.obhdp.2020.11.003?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Fearon, James D. & Laitin, David D., 2003. "Ethnicity, Insurgency, and Civil War," American Political Science Review, Cambridge University Press, vol. 97(1), pages 75-90, February.
    2. Matias D. Cattaneo & Sebastian Galiani & Paul J. Gertler & Sebastian Martinez & Rocio Titiunik, 2009. "Housing, Health, and Happiness," American Economic Journal: Economic Policy, American Economic Association, vol. 1(1), pages 75-105, February.
    3. Paul Collier & Anke Hoeffler, 2004. "Greed and grievance in civil war," Oxford Economic Papers, Oxford University Press, vol. 56(4), pages 563-595, October.
    4. Katherine Button, 2018. "Reboot undergraduate courses for reproducibility," Nature, Nature, vol. 561(7723), pages 287-287, September.
    5. Nosek, Brian A. & Errington, Timothy M., 2019. "What is replication?," MetaArXiv u4g6t, Center for Open Science.
    6. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235, October.
    7. Neunhoeffer, Marcel & Sternberg, Sebastian, 2019. "How Cross-Validation Can Go Wrong and What to Do About It," Political Analysis, Cambridge University Press, vol. 27(1), pages 101-106, January.
    8. Roland G. Fryer Jr., 2019. "An Empirical Analysis of Racial Differences in Police Use of Force," Journal of Political Economy, University of Chicago Press, vol. 127(3), pages 1210-1261.
    9. Hyunyoung Choi & Hal Varian, 2012. "Predicting the Present with Google Trends," The Economic Record, The Economic Society of Australia, vol. 88(s1), pages 2-9, June.
    10. Nosek, Brian A. & Ebersole, Charles R. & DeHaven, Alexander Carl & Mellor, David Thomas, 2018. "The Preregistration Revolution," OSF Preprints 2dxu5, Center for Open Science.
    11. Coupé, Tom, 2018. "Replicating "Predicting the present with Google trends" by Hyunyoung Choi and Hal Varian (The Economic Record, 2012)," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-8.
    12. Muchlinski, David & Siroky, David & He, Jingrui & Kocher, Matthew, 2016. "Comparing Random Forest with Logistic Regression for Predicting Class-Imbalanced Civil War Onset Data," Political Analysis, Cambridge University Press, vol. 24(1), pages 87-103, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Brykman, Kyle M. & O'Neill, Thomas A., 2023. "How conflict expressions affect recipients’ conflict management behaviors," Organizational Behavior and Human Decision Processes, Elsevier, vol. 174(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. David Siroky & Carolyn M. Warner & Gabrielle Filip-Crawford & Anna Berlin & Steven L. Neuberg, 2020. "Grievances and rebellion: Comparing relative deprivation and horizontal inequality," Conflict Management and Peace Science, Peace Science Society (International), vol. 37(6), pages 694-715, November.
    2. Vestby, Jonas & Buhaug, Halvard & von Uexkull, Nina, 2021. "Why do some poor countries see armed conflict while others do not? A dual sector approach," World Development, Elsevier, vol. 138(C).
    3. Robert MacCulloch & Silvia Pezzini, 2010. "The Roles of Freedom, Growth, and Religion in the Taste for Revolution," Journal of Law and Economics, University of Chicago Press, vol. 53(2), pages 329-358, May.
    4. Kjetil Bjorvatn & Mohammad Reza Farzanegan, 2014. "Resource Rents, Power, and Political Stability," CESifo Working Paper Series 4727, CESifo.
    5. Thiemo Fetzer & Samuel Marden, 2017. "Take What You Can: Property Rights, Contestability and Conflict," Economic Journal, Royal Economic Society, vol. 0(601), pages 757-783, May.
    6. Benjamin Crost & Joseph H Felter, 2020. "Export Crops and Civil Conflict," Journal of the European Economic Association, European Economic Association, vol. 18(3), pages 1484-1520.
    7. Janus, Thorsten & Riera-Crichton, Daniel, 2015. "Economic shocks, civil war and ethnicity," Journal of Development Economics, Elsevier, vol. 115(C), pages 32-44.
    8. Victor Ginsburgh & Shlomo Weber, 2020. "The Economics of Language," Journal of Economic Literature, American Economic Association, vol. 58(2), pages 348-404, June.
    9. Elisabeth Gilmore & Nils Petter Gleditsch & Päivi Lujala & Jan Ketil Rod, 2005. "Conflict Diamonds: A New Dataset," Conflict Management and Peace Science, Peace Science Society (International), vol. 22(3), pages 257-272, July.
    10. Ang, James B. & Gupta, Satyendra Kumar, 2018. "Agricultural yield and conflict," Journal of Environmental Economics and Management, Elsevier, vol. 92(C), pages 397-417.
    11. Timothy Besley & Torsten Persson, 2011. "Pillars of Prosperity: The Political Economics of Development Clusters," Economics Books, Princeton University Press, edition 1, number 9624.
    12. Loaiza, Osmar & Muñetón, Guberney & Vanegas, Gabriel, 2014. "The relationship between multidimensional poverty and armed conflict: the case of Antioquia, Colombia," MPRA Paper 62462, University Library of Munich, Germany, revised 30 Jan 2015.
    13. Alessandro Moro, 2016. "Understanding the Dynamics of Violent Political Revolutions in an Agent-Based Framework," PLOS ONE, Public Library of Science, vol. 11(4), pages 1-17, April.
    14. Rigterink, Anouk S., 2012. "New Wars in Numbers. An exploration of various datasets on intra-state violence," MPRA Paper 45264, University Library of Munich, Germany.
    15. Mehlum, Halvor & Ove Moene, Karl, 2011. "Aggressive elites and vulnerable entrepreneurs - trust and cooperation in the shadow of conflict," Memorandum 16/2010, Oslo University, Department of Economics.
    16. Victor Ginsburgh & Shlomo Weber, 2016. "Linguistic Diversity, Standardization, and Disenfranchisement: Measurement and Consequences," ULB Institutional Repository 2013/277407, ULB -- Universite Libre de Bruxelles.
    17. Vusal Musayev, 2016. "Externalities in Military Spending and Growth: The Role of Natural Resources as a Channel through Conflict," Defence and Peace Economics, Taylor & Francis Journals, vol. 27(3), pages 378-391, June.
    18. Matthew Fuhrmann & Jaroslav Tir, 2009. "Territorial Dimensions of Enduring Internal Rivalries," Conflict Management and Peace Science, Peace Science Society (International), vol. 26(4), pages 307-329, September.
    19. Idean Salehyan, 2010. "The Delegation of War to Rebel Organizations," Journal of Conflict Resolution, Peace Science Society (International), vol. 54(3), pages 493-515, June.
    20. Axel Dreher & Merle Kreibaum, 2016. "Weapons of choice," Journal of Peace Research, Peace Research Institute Oslo, vol. 53(4), pages 539-553, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jobhdp:v:164:y:2021:i:c:p:192-202. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/obhdp .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.