IDEAS home Printed from https://ideas.repec.org/a/bla/joares/v56y2018i2p313-362.html
   My bibliography  Save this article

No System Is Perfect: Understanding How Registration‐Based Editorial Processes Affect Reproducibility and Investment in Research Quality

Author

Listed:
  • ROBERT BLOOMFIELD
  • KRISTINA RENNEKAMP
  • BLAKE STEENHOVEN

Abstract

The papers in this volume were published through a Registration‐based Editorial Process (REP). Authors submitted proposals to gather and analyze data; successful proposals were guaranteed publication as long as the authors lived up to their commitments, regardless of whether results supported their predictions. To understand how REP differs from the Traditional Editorial Process (TEP), we analyze the papers themselves; conference comments; a survey of conference authors, reviewers, and attendees; and a survey of authors who have successfully published under TEP. We find that REP increases up‐front investment in planning, data gathering, and analysis, but reduces follow‐up investment after results are known. This shift in investment makes individual results more reproducible, but leaves articles less thorough and refined. REP could be improved by encouraging selected forms of follow‐up investment that survey respondents believe are usually used under TEP to make papers more informative, focused, and accurate at little risk of overstatement.

Suggested Citation

  • Robert Bloomfield & Kristina Rennekamp & Blake Steenhoven, 2018. "No System Is Perfect: Understanding How Registration‐Based Editorial Processes Affect Reproducibility and Investment in Research Quality," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 313-362, May.
    • Handle: RePEc:bla:joares:v:56:y:2018:i:2:p:313-362
    DOI: 10.1111/1475-679X.12208
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/1475-679X.12208
    Download Restriction: no
    File URL: https://libkey.io/10.1111/1475-679X.12208?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Shelley Xin Li & Tatiana Sandino, 2018. "Effects of an Information Sharing System on Employee Creativity, Engagement, and Performance," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 713-747, May.
    2. Sanne R. Van Duin & Henri C. Dekker & Jacco L. Wielhouwer & Juan P. Mendoza, 2018. "The Tone from Above: The Effect of Communicating a Supportive Regulatory Strategy on Reporting Quality," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 467-519, May.
    3. Vatter, Wj, 1966. "Critical Synthesis Of Conference Papers," Journal of Accounting Research, Wiley Blackwell, vol. 4, pages 228-233.
    4. Hamermesh, Daniel S., 2007. "Replication in Economics," IZA Discussion Papers 2760, Institute of Labor Economics (IZA).
    5. Robert Bloomfield & Mark W. Nelson & Eugene Soltes, 2016. "Gathering Data for Archival, Field, Survey, and Experimental Accounting Research," Journal of Accounting Research, Wiley Blackwell, vol. 54(2), pages 341-395, May.
    6. Yonca Ertimur & Caleb Rawson & Jonathan L. Rogers & Sarah L. C. Zechman, 2018. "Bridging the Gap: Evidence from Externally Hired CEOs," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 521-579, May.
    7. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    8. Darren Bernard & Nicole L. Cade & Frank Hodge, 2018. "Investor Behavior and the Benefits of Direct Stock Ownership," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 431-466, May.
    9. Gigerenzer, Gerd, 2004. "Mindless statistics," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 33(5), pages 587-606, November.
    10. Luzi Hail & Ahmed Tahoun & Clare Wang, 2018. "Corporate Scandals and Regulation," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 617-671, May.
    11. Zachary T. Kowaleski & Brian W. Mayhew & Amy C. Tegeler, 2018. "The Impact of Consulting Services on Audit Quality: An Experimental Approach," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 673-711, May.
    12. Henry Eyring & V. G. Narayanan, 2018. "Performance Effects of Setting a High Reference Point for Peer‐Performance Comparison," Journal of Accounting Research, Wiley Blackwell, vol. 56(2), pages 581-615, May.
    13. Eyring, Henry & Narayanan, V.G., 2018. "Performance effects of setting a high reference point for peer-performance comparison," LSE Research Online Documents on Economics 86732, London School of Economics and Political Science, LSE Library.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christian Leuz, 2018. "Evidence-based policymaking: promise, challenges and opportunities for accounting and financial markets research," Accounting and Business Research, Taylor & Francis Journals, vol. 48(5), pages 582-608, July.
    2. H. Latan & C.J. Chiappetta Jabbour & Ana Beatriz Lopes de Sousa Jabbour & M. Ali, 2023. "Crossing the Red Line? Empirical Evidence and Useful Recommendations on Questionable Research Practices among Business Scholars," Post-Print hal-04276024, HAL.
    3. Aman, Hiroyuki & Beekes, Wendy & Berkman, Henk & Bohmann, Marc & Bradbury, Michael & Chapple, Larelle & Chang, Millicent & Clout, Victoria & Faff, Robert & Han, Jianlei & Hillier, David & Hodgson, All, 2019. "Responsible science: Celebrating the 50-year legacy of Ball and Brown (1968) using a registration-based framework," Pacific-Basin Finance Journal, Elsevier, vol. 56(C), pages 129-150.
    4. Stephen A. Zeff, 2019. "A Personal View of the Evolution of the Accounting Professoriate," Accounting Perspectives, John Wiley & Sons, vol. 18(3), pages 159-185, September.
    5. Johnstone, David, 2022. "Accounting research and the significance test crisis," CRITICAL PERSPECTIVES ON ACCOUNTING, Elsevier, vol. 89(C).
    6. Thomas R. Dyckman & Stephen A. Zeff, 2019. "Important Issues in Statistical Testing and Recommended Improvements in Accounting Research," Econometrics, MDPI, vol. 7(2), pages 1-11, May.
    7. Armstrong, Christopher & Kepler, John D. & Samuels, Delphine & Taylor, Daniel, 2022. "Causality redux: The evolution of empirical methods in accounting research and the growth of quasi-experiments," Journal of Accounting and Economics, Elsevier, vol. 74(2).
    8. Christof Weinhardt & Wil M. P. Aalst & Oliver Hinz, 2019. "Introducing Registered Reports to the Information Systems Community," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 61(4), pages 381-384, August.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hussinger, Katrin & Pellens, Maikel, 2019. "Guilt by association: How scientific misconduct harms prior collaborators," Research Policy, Elsevier, vol. 48(2), pages 516-530.
    2. Labro, Eva & Lang, Mark & Omartian, James D., 2023. "Predictive analytics and centralization of authority," Journal of Accounting and Economics, Elsevier, vol. 75(1).
    3. Vaidya, Shalvaree, 2021. "The impact of premium subsidies on health plan choices in Switzerland: Who responds to the incentives set by in-kind as opposed to cash transfers?," Health Policy, Elsevier, vol. 125(6), pages 675-684.
    4. Grolleau, Gilles & Ibanez, Lisette & Mzoughi, Naoufel, 2020. "Moral judgment of environmental harm caused by a single versus multiple wrongdoers: A survey experiment," Ecological Economics, Elsevier, vol. 170(C).
    5. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    6. Kaiser, Caspar, 2020. "People do not adapt. New analyses of the dynamic effects of own and reference income on life satisfaction," Journal of Economic Behavior & Organization, Elsevier, vol. 177(C), pages 494-513.
    7. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    8. Robert A. Moffitt, 2011. "Report of the Editor: American Economic Review (with Appendix by Philip J. Glandon)," American Economic Review, American Economic Association, vol. 101(3), pages 684-693, May.
    9. Chai, Daniel & Ali, Searat & Brosnan, Mark & Hasso, Tim, 2024. "Understanding researchers' perceptions and experiences in finance research replication studies: A pre-registered report," Pacific-Basin Finance Journal, Elsevier, vol. 86(C).
    10. Steffen Mueller & Jens Stegmaier, 2017. "The Dynamic Effects of Works Councils on Labour Productivity: First Evidence from Panel Data," British Journal of Industrial Relations, London School of Economics, vol. 55(2), pages 372-395, June.
    11. Roy Chen & Yan Chen & Yohanes E. Riyanto, 2021. "Best practices in replication: a case study of common information in coordination games," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 2-30, March.
    12. Daniel S. Hamermesh & Katie R. Genadek & Michael C. Burda, 2022. "Reply to “Racial Differences in Time at Work Not Working†by William A. Darity Jr. et al," ILR Review, Cornell University, ILR School, vol. 75(3), pages 573-577, May.
    13. Lisa-Marie Wibbeke & Maik Lachmann, 2020. "Psychology in management accounting and control research: an overview of the recent literature," Journal of Management Control: Zeitschrift für Planung und Unternehmenssteuerung, Springer, vol. 31(3), pages 275-328, September.
    14. Niklas Amberg & Thomas Jansson & Mathias Klein & Anna Rogantini Picco, 2022. "Five Facts about the Distributional Income Effects of Monetary Policy Shocks," American Economic Review: Insights, American Economic Association, vol. 4(3), pages 289-304, September.
    15. Braakmann Nils, 2009. "The Impact of September 11th, 2001 on the Employment Prospects of Arabs and Muslims in the German Labor Market," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 229(1), pages 2-21, February.
    16. Markowsky, Eva, 2022. "Culture, Female Labour Force Participation, and Selective Migrationː New Meta-Analytic Evidence," WiSo-HH Working Paper Series 65, University of Hamburg, Faculty of Business, Economics and Social Sciences, WISO Research Laboratory.
    17. Robert Inklaar & Marcel P. Timmer, 2012. "Productivity Convergence Across Industries and Countries: The Importance of Theory-based Measurement," Chapters, in: Matilde Mas & Robert Stehrer (ed.), Industrial Productivity in Europe, chapter 11, Edward Elgar Publishing.
    18. Kaplan, Greg & Mitman, Kurt & Violante, Giovanni L., 2020. "Non-durable consumption and housing net worth in the Great Recession: Evidence from easily accessible data," Journal of Public Economics, Elsevier, vol. 189(C).
    19. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    20. Amberg, Niklas & Jansson, Thomas & Klein, Mathias & Rogantini Picco, Anna, 2021. "Five Facts about the Distributional Income Effects of Monetary Policy," Working Paper Series 403, Sveriges Riksbank (Central Bank of Sweden).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:joares:v:56:y:2018:i:2:p:313-362. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0021-8456 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.