IDEAS home Printed from https://ideas.repec.org/a/spr/qualqt/v58y2024i1d10.1007_s11135-023-01657-0.html
   My bibliography  Save this article

Towards more credible conceptual replications under heteroscedasticity and unbalanced designs

Author

Listed:
  • Pedro Mateu

    (Universidad del Pacífico)

  • Brooks Applegate

    (Western Michigan University)

  • Chris L. Coryn

    (Western Michigan University)

Abstract

Theory cannot be fully validated unless original results have been replicated, resulting in conclusion consistency. Replications are the strongest source of evidence to verify research findings and knowledge claims. In the social sciences, replication studies often fail and thus a continuing need for replication studies to confirm tentative facts, expand knowledge to gain new understanding, and verify hypotheses. Failure to replicate in the social and behavioral sciences sometimes arises due to dissimilarity between hypotheses formulated in original and replication studies. Alternatively, failure to replicate also occurs when the same hypothesis is tested; but done so in the absence of knowledge from previous investigations, as when original study effect sizes are not considered in replication studies. To increase replicability of research findings, this paper demonstrates that the application of two one-sided tests to evaluate a replication question provides a superior means for conducting replications, assuming all other methodological procedures remained as similar as possible. Furthermore, this paper sought to explore the impact of heteroscedasticity and unbalanced designs in replication studies in four paired conditions of variance and sample size. Two Monte Carlo simulations, each with two stages, were conducted to investigate conclusion consistency among different replication procedures to determine the repeatability of an observed effect. Overall, the proposed approach yielded a higher proportion of successful replications than the conventional approach (testing the original null hypothesis of no effect). Thus, findings can be confirmed by replications and in the absence of confirmation, there cannot be a final statement about any theory.

Suggested Citation

  • Pedro Mateu & Brooks Applegate & Chris L. Coryn, 2024. "Towards more credible conceptual replications under heteroscedasticity and unbalanced designs," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(1), pages 723-751, February.
  • Handle: RePEc:spr:qualqt:v:58:y:2024:i:1:d:10.1007_s11135-023-01657-0
    DOI: 10.1007/s11135-023-01657-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11135-023-01657-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11135-023-01657-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    2. Blakeley B. McShane & Jennifer L. Tackett & Ulf Böckenholt & Andrew Gelman, 2019. "Large-Scale Replication Projects in Contemporary Psychological Research," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 99-105, March.
    3. Monya Baker, 2016. "1,500 scientists lift the lid on reproducibility," Nature, Nature, vol. 533(7604), pages 452-454, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hussinger, Katrin & Pellens, Maikel, 2019. "Guilt by association: How scientific misconduct harms prior collaborators," Research Policy, Elsevier, vol. 48(2), pages 516-530.
    2. Vaidya, Shalvaree, 2021. "The impact of premium subsidies on health plan choices in Switzerland: Who responds to the incentives set by in-kind as opposed to cash transfers?," Health Policy, Elsevier, vol. 125(6), pages 675-684.
    3. Grolleau, Gilles & Ibanez, Lisette & Mzoughi, Naoufel, 2020. "Moral judgment of environmental harm caused by a single versus multiple wrongdoers: A survey experiment," Ecological Economics, Elsevier, vol. 170(C).
    4. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    5. Kaiser, Caspar, 2020. "People do not adapt. New analyses of the dynamic effects of own and reference income on life satisfaction," Journal of Economic Behavior & Organization, Elsevier, vol. 177(C), pages 494-513.
    6. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    7. Robert A. Moffitt, 2011. "Report of the Editor: American Economic Review (with Appendix by Philip J. Glandon)," American Economic Review, American Economic Association, vol. 101(3), pages 684-693, May.
    8. Dennis Bontempi & Leonard Nuernberg & Suraj Pai & Deepa Krishnaswamy & Vamsi Thiriveedhi & Ahmed Hosny & Raymond H. Mak & Keyvan Farahani & Ron Kikinis & Andrey Fedorov & Hugo J. W. L. Aerts, 2024. "End-to-end reproducible AI pipelines in radiology using the cloud," Nature Communications, Nature, vol. 15(1), pages 1-9, December.
    9. Steffen Mueller & Jens Stegmaier, 2017. "The Dynamic Effects of Works Councils on Labour Productivity: First Evidence from Panel Data," British Journal of Industrial Relations, London School of Economics, vol. 55(2), pages 372-395, June.
    10. Roy Chen & Yan Chen & Yohanes E. Riyanto, 2021. "Best practices in replication: a case study of common information in coordination games," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 2-30, March.
    11. Daniel S. Hamermesh & Katie R. Genadek & Michael C. Burda, 2022. "Reply to “Racial Differences in Time at Work Not Working†by William A. Darity Jr. et al," ILR Review, Cornell University, ILR School, vol. 75(3), pages 573-577, May.
    12. Raymond, Wladimir & Mairesse, Jacques & Mohnen, Pierre & Palm, Franz, 2015. "Dynamic models of R & D, innovation and productivity: Panel data evidence for Dutch and French manufacturing," European Economic Review, Elsevier, vol. 78(C), pages 285-306.
    13. Niklas Amberg & Thomas Jansson & Mathias Klein & Anna Rogantini Picco, 2022. "Five Facts about the Distributional Income Effects of Monetary Policy Shocks," American Economic Review: Insights, American Economic Association, vol. 4(3), pages 289-304, September.
    14. Braakmann Nils, 2009. "The Impact of September 11th, 2001 on the Employment Prospects of Arabs and Muslims in the German Labor Market," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 229(1), pages 2-21, February.
    15. Markowsky, Eva, 2022. "Culture, Female Labour Force Participation, and Selective Migrationː New Meta-Analytic Evidence," WiSo-HH Working Paper Series 65, University of Hamburg, Faculty of Business, Economics and Social Sciences, WISO Research Laboratory.
    16. Robert Inklaar & Marcel P. Timmer, 2012. "Productivity Convergence Across Industries and Countries: The Importance of Theory-based Measurement," Chapters, in: Matilde Mas & Robert Stehrer (ed.), Industrial Productivity in Europe, chapter 11, Edward Elgar Publishing.
    17. Dreber, Anna & Johannesson, Magnus, 2023. "A framework for evaluating reproducibility and replicability in economics," Ruhr Economic Papers 1055, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    18. Kaplan, Greg & Mitman, Kurt & Violante, Giovanni L., 2020. "Non-durable consumption and housing net worth in the Great Recession: Evidence from easily accessible data," Journal of Public Economics, Elsevier, vol. 189(C).
    19. Robert Bloomfield & Kristina Rennekamp & Blake Steenhoven, 2018. "No System Is Perfect: Understanding How Registration‐Based Editorial Processes Affect Reproducibility and Investment in Research Quality," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 313-362, May.
    20. Peter Harremoës, 2019. "Replication Papers," Publications, MDPI, vol. 7(3), pages 1-8, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:qualqt:v:58:y:2024:i:1:d:10.1007_s11135-023-01657-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.