IDEAS home Printed from https://ideas.repec.org/p/zbw/daredp/2302.html
   My bibliography  Save this paper

Lessons from the p-value debate and the replication crisis for "open Q science" – the editor's perspective or: will the revolution devour its children?

Author

Listed:
  • Hüttel, Silke
  • Hess, Sebastian

Abstract

The scientific production system is of ultimate importance for the way how humans address global challenges. Recently, scholars have begun to voice concerns about structural inefficiencies within this system, as e.g. the replication crisis, the p-value debate or the identification of various forms of publication bias have brought up. Most suggested remedies tend to address only partial aspects of the system's inefficiencies, while no unifying agenda towards an overall transformation of the system has yet emerged. We argue that a unifying agenda is even more urgently needed in light of Artificial intelligence (AI) that is arising as a tool for scientific writing services. Without appropriate reactions from the Q science community, this trend may even exponentiate present credibility problems due to limited replicability and ritual-based statistical practice, while amplifying all forms of already existing biases. Our review of these developments suggests that näive openness in the science system alone will unlikely lead to major efficiency gains. We contribute by identifying key elements for the definition of transformation pathways towards open, democratic and conscious learning, teaching, reviewing and publishing that will be supported by openly maintained AI tools. As part of this transition, roles and incentives for reviewers will have to gain in relation to authors: Future Q scientists will have to write less, learn differently and review more.

Suggested Citation

  • Hüttel, Silke & Hess, Sebastian, 2023. "Lessons from the p-value debate and the replication crisis for "open Q science" – the editor's perspective or: will the revolution devour its children?," DARE Discussion Papers 2302, Georg-August University of Göttingen, Department of Agricultural Economics and Rural Development (DARE).
  • Handle: RePEc:zbw:daredp:2302
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/277753/1/186016286X.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Dietmar Wolfram & Peiling Wang & Adam Hembree & Hyoungjoo Park, 2020. "Open peer review: promoting transparency in open science," Scientometrics, Springer;Akadémiai Kiadó, vol. 125(2), pages 1033-1051, November.
    2. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    3. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    4. T. D. Stanley, 2005. "Beyond Publication Bias," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 309-345, July.
    5. Heß, Sebastian & Bergmann, Holger & Sudmann, Lüder, 2006. "Die Förderung alternativer Energien - eine kritische Bestandsaufnahme," Department of Agricultural and Rural Development (DARE) Discussion Papers 187446, Georg-August-Universitaet Goettingen, Department of Agricultural Economics and Rural Development (DARE).
    6. Chris Doucouliagos, 2005. "Publication Bias in the Economic Freedom and Economic Growth Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 367-387, July.
    7. Heß, Sebastian & Bergmann, Holger & Sudmann, Lüder, 2006. "Die Förderung alternativer Energien - eine kritische Bestandsaufnahme," 54th Annual Conference, Goettingen, Germany, September 17-19, 2014 187446, German Association of Agricultural Economists (GEWISOLA).
    8. Hirschauer Norbert & Mußhoff Oliver & Grüner Sven & Frey Ulrich & Theesfeld Insa & Wagner Peter, 2016. "Die Interpretation des p-Wertes – Grundsätzliche Missverständnisse," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 236(5), pages 557-575, October.
    9. Bornmann, Lutz & Mutz, Rüdiger & Daniel, Hans-Dieter, 2007. "Gender differences in grant peer review: A meta-analysis," Journal of Informetrics, Elsevier, vol. 1(3), pages 226-238.
    10. Regina Nuzzo, 2015. "How scientists fool themselves – and how they can stop," Nature, Nature, vol. 526(7572), pages 182-185, October.
    11. Paul J. Ferraro & Pallavi Shukla, 2020. "Feature—Is a Replicability Crisis on the Horizon for Environmental and Resource Economics?," Review of Environmental Economics and Policy, University of Chicago Press, vol. 14(2), pages 339-351.
    12. Broder, Ivy E, 1993. "Review of NSF Economics Proposals: Gender and Institutional Patterns," American Economic Review, American Economic Association, vol. 83(4), pages 964-970, September.
    13. Stefan Wimmer & Robert Finger, 2023. "A note on synthetic data for replication purposes in agricultural economics," Journal of Agricultural Economics, Wiley Blackwell, vol. 74(1), pages 316-323, February.
    14. Antonio J. Herrera, 1999. "Language bias discredits the peer-review system," Nature, Nature, vol. 397(6719), pages 467-467, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    2. Alexander L. Brown & Taisuke Imai & Ferdinand M. Vieider & Colin F. Camerer, 2024. "Meta-analysis of Empirical Estimates of Loss Aversion," Journal of Economic Literature, American Economic Association, vol. 62(2), pages 485-516, June.
    3. Matteo Picchio & Michele Ubaldi, 2024. "Unemployment and health: A meta‐analysis," Journal of Economic Surveys, Wiley Blackwell, vol. 38(4), pages 1437-1472, September.
    4. Thibaut Arpinon & Marianne Lefebvre, 2024. "Registered Reports and Associated Benefits for Agricultural Economics," Post-Print hal-04635986, HAL.
    5. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2020. "Remittances and economic growth: A meta-analysis," World Development, Elsevier, vol. 134(C).
    6. Dominika Ehrenbergerova & Josef Bajzik & Tomas Havranek, 2023. "When Does Monetary Policy Sway House Prices? A Meta-Analysis," IMF Economic Review, Palgrave Macmillan;International Monetary Fund, vol. 71(2), pages 538-573, June.
    7. Snyder, Christopher & Zhuo, Ran, 2018. "Sniff Tests in Economics: Aggregate Distribution of Their Probability Values and Implications for Publication Bias," MetaArXiv 8vdrh, Center for Open Science.
    8. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    9. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    10. Cazachevici, Alina & Havranek, Tomas & Horvath, Roman, 2019. "Remittances and Economic Growth: A Quantitative Survey," MPRA Paper 96823, University Library of Munich, Germany.
    11. Dreber, Anna & Johannesson, Magnus, 2023. "A framework for evaluating reproducibility and replicability in economics," I4R Discussion Paper Series 38, The Institute for Replication (I4R).
    12. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    13. Heckelei, Thomas & Huettel, Silke & Odening, Martin & Rommel, Jens, 2021. "The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?," Discussion Papers 316369, University of Bonn, Institute for Food and Resource Economics.
    14. Römer, Ulf & Weber, Ron & Mußhoff, Oliver & Turvey, Calcum G., 2017. "Truth and consequences: Bogus pipeline experiment in informal small business lending," DARE Discussion Papers 1702, Georg-August University of Göttingen, Department of Agricultural Economics and Rural Development (DARE).
    15. Stefano DellaVigna & Elizabeth Linos, 2022. "RCTs to Scale: Comprehensive Evidence From Two Nudge Units," Econometrica, Econometric Society, vol. 90(1), pages 81-116, January.
    16. Klomp, Jeroen, 2023. "Political budget cycles in military expenditures: A meta-analysis," Economic Analysis and Policy, Elsevier, vol. 77(C), pages 1083-1102.
    17. Neves, Pedro Cunha & Afonso, Oscar & Silva, Diana & Sochirca, Elena, 2021. "The link between intellectual property rights, innovation, and growth: A meta-analysis," Economic Modelling, Elsevier, vol. 97(C), pages 196-209.
    18. Adam Gorajek & Joel Bank & Andrew Staib & Benjamin Malin & Hamish Fitchett, 2021. "Star Wars at Central Banks," RBA Research Discussion Papers rdp2021-02, Reserve Bank of Australia.
    19. Dimos, Christos & Pugh, Geoff & Hisarciklilar, Mehtap & Talam, Ema & Jackson, Ian, 2022. "The relative effectiveness of R&D tax credits and R&D subsidies: A comparative meta-regression analysis," Technovation, Elsevier, vol. 115(C).
    20. Jasper Brinkerink, 2023. "When Shooting for the Stars Becomes Aiming for Asterisks: P-Hacking in Family Business Research," Entrepreneurship Theory and Practice, , vol. 47(2), pages 304-343, March.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:daredp:2302. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/iagoede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.