IDEAS home Printed from https://ideas.repec.org/a/bla/joares/v56y2018i2p313-362.html
   My bibliography  Save this article

No System Is Perfect: Understanding How Registration‐Based Editorial Processes Affect Reproducibility and Investment in Research Quality

Author

Listed:
  • ROBERT BLOOMFIELD
  • KRISTINA RENNEKAMP
  • BLAKE STEENHOVEN

Abstract

The papers in this volume were published through a Registration‐based Editorial Process (REP). Authors submitted proposals to gather and analyze data; successful proposals were guaranteed publication as long as the authors lived up to their commitments, regardless of whether results supported their predictions. To understand how REP differs from the Traditional Editorial Process (TEP), we analyze the papers themselves; conference comments; a survey of conference authors, reviewers, and attendees; and a survey of authors who have successfully published under TEP. We find that REP increases up‐front investment in planning, data gathering, and analysis, but reduces follow‐up investment after results are known. This shift in investment makes individual results more reproducible, but leaves articles less thorough and refined. REP could be improved by encouraging selected forms of follow‐up investment that survey respondents believe are usually used under TEP to make papers more informative, focused, and accurate at little risk of overstatement.

Suggested Citation

  • Robert Bloomfield & Kristina Rennekamp & Blake Steenhoven, 2018. "No System Is Perfect: Understanding How Registration‐Based Editorial Processes Affect Reproducibility and Investment in Research Quality," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 313-362, May.
    • Handle: RePEc:bla:joares:v:56:y:2018:i:2:p:313-362
    DOI: 10.1111/1475-679X.12208
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/1475-679X.12208
    Download Restriction: no
    File URL: https://libkey.io/10.1111/1475-679X.12208?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sanne R. Van Duin & Henri C. Dekker & Jacco L. Wielhouwer & Juan P. Mendoza, 2018. "The Tone from Above: The Effect of Communicating a Supportive Regulatory Strategy on Reporting Quality," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 467-519, May.
    2. Vatter, Wj, 1966. "Critical Synthesis Of Conference Papers," Journal of Accounting Research, Wiley Blackwell, vol. 4, pages 228-233.
    3. Robert Bloomfield & Mark W. Nelson & Eugene Soltes, 2016. "Gathering Data for Archival, Field, Survey, and Experimental Accounting Research," Journal of Accounting Research, Wiley Blackwell, vol. 54(2), pages 341-395, May.
    4. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    5. Gigerenzer, Gerd, 2004. "Mindless statistics," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 587-606, November.
    6. Henry Eyring & V. G. Narayanan, 2018. "Performance Effects of Setting a High Reference Point for Peer‐Performance Comparison," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 581-615, May.
    7. Eyring, Henry & Narayanan, V.G., 2018. "Performance effects of setting a high reference point for peer-performance comparison," LSE Research Online Documents on Economics 86732, London School of Economics and Political Science, LSE Library.
    8. Shelley Xin Li & Tatiana Sandino, 2018. "Effects of an Information Sharing System on Employee Creativity, Engagement, and Performance," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 713-747, May.
    9. Hamermesh, Daniel S., 2007. "Replication in Economics," IZA Discussion Papers 2760, Institute of Labor Economics (IZA).
    10. Yonca Ertimur & Caleb Rawson & Jonathan L. Rogers & Sarah L. C. Zechman, 2018. "Bridging the Gap: Evidence from Externally Hired CEOs," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 521-579, May.
    11. Darren Bernard & Nicole L. Cade & Frank Hodge, 2018. "Investor Behavior and the Benefits of Direct Stock Ownership," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 431-466, May.
    12. Luzi Hail & Ahmed Tahoun & Clare Wang, 2018. "Corporate Scandals and Regulation," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 617-671, May.
    13. Zachary T. Kowaleski & Brian W. Mayhew & Amy C. Tegeler, 2018. "The Impact of Consulting Services on Audit Quality: An Experimental Approach," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 673-711, May.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christian Leuz, 2018. "Evidence-based policymaking: promise, challenges and opportunities for accounting and financial markets research," Accounting and Business Research, Taylor & Francis Journals, vol. 48(5), pages 582-608, July.
    2. Stephen A. Zeff, 2019. "A Personal View of the Evolution of the Accounting Professoriate," Accounting Perspectives, John Wiley & Sons, vol. 18(3), pages 159-185, September.
    3. Armstrong, Christopher & Kepler, John D. & Samuels, Delphine & Taylor, Daniel, 2022. "Causality redux: The evolution of empirical methods in accounting research and the growth of quasi-experiments," Journal of Accounting and Economics, Elsevier, vol. 74(2).
    4. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    5. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    6. Christof Weinhardt & Wil M. P. Aalst & Oliver Hinz, 2019. "Introducing Registered Reports to the Information Systems Community," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 61(4), pages 381-384, August.
    7. H. Latan & C.J. Chiappetta Jabbour & Ana Beatriz Lopes de Sousa Jabbour & M. Ali, 2023. "Crossing the Red Line? Empirical Evidence and Useful Recommendations on Questionable Research Practices among Business Scholars," Post-Print hal-04276024, HAL.
    8. Aman, Hiroyuki & Beekes, Wendy & Berkman, Henk & Bohmann, Marc & Bradbury, Michael & Chapple, Larelle & Chang, Millicent & Clout, Victoria & Faff, Robert & Han, Jianlei & Hillier, David & Hodgson, All, 2019. "Responsible science: Celebrating the 50-year legacy of Ball and Brown (1968) using a registration-based framework," Pacific-Basin Finance Journal, Elsevier, vol. 56(C), pages 129-150.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    2. Vlaeminck, Sven, 2013. "Data Management in Scholarly Journals and Possible Roles for Libraries - Some Insights from EDaWaX," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 23(1), pages 49-79.
    3. Hussinger, Katrin & Pellens, Maikel, 2019. "Guilt by association: How scientific misconduct harms prior collaborators," Research Policy, Elsevier, vol. 48(2), pages 516-530.
    4. Labro, Eva & Lang, Mark & Omartian, James D., 2023. "Predictive analytics and centralization of authority," Journal of Accounting and Economics, Elsevier, vol. 75(1).
    5. Vaidya, Shalvaree, 2021. "The impact of premium subsidies on health plan choices in Switzerland: Who responds to the incentives set by in-kind as opposed to cash transfers?," Health Policy, Elsevier, vol. 125(6), pages 675-684.
    6. Grolleau, Gilles & Ibanez, Lisette & Mzoughi, Naoufel, 2020. "Moral judgment of environmental harm caused by a single versus multiple wrongdoers: A survey experiment," Ecological Economics, Elsevier, vol. 170(C).
    7. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    8. Joachim Wagner, 2016. "From Estimation Results to Stylized Facts: Twelve Recommendations for Empirical Research in International Activities of Heterogeneous Firms," World Scientific Book Chapters, in: Microeconometrics of International Trade, chapter 15, pages 479-514, World Scientific Publishing Co. Pte. Ltd..
    9. Temouri, Yama & Vogel, Alexander & Wagner, Joachim, 2013. "Self-selection into export markets by business services firms – Evidence from France, Germany and the United Kingdom," Structural Change and Economic Dynamics, Elsevier, vol. 25(C), pages 146-158.
    10. Kaiser, Caspar, 2020. "People do not adapt. New analyses of the dynamic effects of own and reference income on life satisfaction," Journal of Economic Behavior & Organization, Elsevier, vol. 177(C), pages 494-513.
    11. Fišar, Miloš & Greiner, Ben & Huber, Christoph & Katok, Elena & Ozkes, Ali & Management Science Reproducibility Collaboration, 2023. "Reproducibility in Management Science," Department for Strategy and Innovation Working Paper Series 03/2023, WU Vienna University of Economics and Business.
    12. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    13. Robert A. Moffitt, 2011. "Report of the Editor: American Economic Review (with Appendix by Philip J. Glandon)," American Economic Review, American Economic Association, vol. 101(3), pages 684-693, May.
    14. Daniel S. Hamermesh, 2017. "Replication in Labor Economics: Evidence from Data, and What It Suggests," American Economic Review, American Economic Association, vol. 107(5), pages 37-40, May.
    15. Joachim Wagner, 2016. "Credit Constraints and Exports: A Survey of Empirical Studies Using Firm Level Data," World Scientific Book Chapters, in: Microeconometrics of International Trade, chapter 12, pages 401-421, World Scientific Publishing Co. Pte. Ltd..
    16. Chai, Daniel & Ali, Searat & Brosnan, Mark & Hasso, Tim, 2024. "Understanding researchers' perceptions and experiences in finance research replication studies: A pre-registered report," Pacific-Basin Finance Journal, Elsevier, vol. 86(C).
    17. Klaus Wersching, 2010. "Schumpeterian Competition, Technological Regimes and Learning through Knowledge Spillover," Post-Print hal-00849408, HAL.
    18. Steffen Mueller & Jens Stegmaier, 2017. "The Dynamic Effects of Works Councils on Labour Productivity: First Evidence from Panel Data," British Journal of Industrial Relations, London School of Economics, vol. 55(2), pages 372-395, June.
    19. Roy Chen & Yan Chen & Yohanes E. Riyanto, 2021. "Best practices in replication: a case study of common information in coordination games," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 2-30, March.
    20. Daniel S. Hamermesh & Katie R. Genadek & Michael C. Burda, 2022. "Reply to “Racial Differences in Time at Work Not Working†by William A. Darity Jr. et al," ILR Review, Cornell University, ILR School, vol. 75(3), pages 573-577, May.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:joares:v:56:y:2018:i:2:p:313-362. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0021-8456 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.