IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/h29xn_v1.html
   My bibliography  Save this paper

Estimating the extent of inflated significance in economics

Author

Listed:
  • Bruns, Stephan
  • Deressa, Teshome Kebede
  • Stanley, T D
  • Doucouliagos, Chris
  • Ioannidis, John P.A.

Abstract

Using a sample of 70,399 published p-values from 192 meta-analyses, we empirically estimate the counterfactual distribution of p-values in the absence of any biases. Comparing observed p-values with counterfactually expected p-values allows us to estimate how many p-values are published as being statistically significant when they should have been published as being non-significant. We estimate the extent of inflated significance to range between 56.2% and 71.3% of the significant p-values. Subsample analysis suggests that the extent of inflated significance is reduced in research fields that use experimental designs, analyze microeconomics research questions and have at least some adequately powered studies.

Suggested Citation

  • Bruns, Stephan & Deressa, Teshome Kebede & Stanley, T D & Doucouliagos, Chris & Ioannidis, John P.A., 2022. "Estimating the extent of inflated significance in economics," MetaArXiv h29xn_v1, Center for Open Science.
    • Handle: RePEc:osf:metaar:h29xn_v1
    DOI: 10.31219/osf.io/h29xn_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/622b744153a4e80277516456/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/h29xn_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    2. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Burlig, Fiona & Preonas, Louis & Woerman, Matt, 2020. "Panel data and experimental design," Journal of Development Economics, Elsevier, vol. 144(C).
    2. Sergey Alexeev, 2023. "Technical change and wage premiums amongst skilled labour: Evidence from the economic transition," Economics of Transition and Institutional Change, John Wiley & Sons, vol. 31(1), pages 189-216, January.
    3. Ashesh Rambachan & Neil Shephard, 2019. "When do common time series estimands have nonparametric causal meaning?," Papers 1903.01637, arXiv.org, revised Jan 2025.
    4. Mayoral, Laura & Ray, Debraj, 2022. "Groups in conflict: Private and public prizes," Journal of Development Economics, Elsevier, vol. 154(C).
    5. Alexis H. Villacis & Jeffrey R. Alwang & Victor Barrera & Juan Dominguez, 2022. "Prices, specialty varieties, and postharvest practices: Insights from cacao value chains in Ecuador," Agribusiness, John Wiley & Sons, Ltd., vol. 38(2), pages 426-458, April.
    6. Millimet, Daniel L. & Bellemare, Marc, 2023. "Fixed Effects and Causal Inference," IZA Discussion Papers 16202, Institute of Labor Economics (IZA).
    7. Aguilar-Gomez, Sandra & Benshaul-Tolonen, Anja, 2023. "The evolution and persistence of women's roles: Evidence from the Gold Rush," Journal of Economic Behavior & Organization, Elsevier, vol. 215(C), pages 364-381.
    8. Nepal, Rabindra & Best, Rohan & Taylor, Madeline, 2023. "Strategies for reducing ethnic inequality in energy outcomes: A Nepalese example," Energy Economics, Elsevier, vol. 126(C).
    9. Czajkowski, Mikołaj & Zagórska, Katarzyna & Letki, Natalia & Tryjanowski, Piotr & Wąs, Adam, 2021. "Drivers of farmers’ willingness to adopt extensive farming practices in a globally important bird area," Land Use Policy, Elsevier, vol. 107(C).
    10. Mikel Bedayo & Raquel Vegas & Gabriel Jiménez & José-Luis Peydró, 2020. "Screening and Loan Origination Time: Lending Standards, Loan Defaults and Bank Failures," Working Papers 1215, Barcelona School of Economics.
    11. KAMKOUM, Arnaud Cedric, 2023. "The Federal Reserve’s Response to the Global Financial Crisis and its Effects: An Interrupted Time-Series Analysis of the Impact of its Quantitative Easing Programs," Thesis Commons d7pvg, Center for Open Science.
    12. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    13. Tal Gross & Timothy J. Layton & Daniel Prinz, 2022. "The Liquidity Sensitivity of Healthcare Consumption: Evidence from Social Security Payments," American Economic Review: Insights, American Economic Association, vol. 4(2), pages 175-190, June.
    14. Timothy J. Bartik & Nathan Sotherland, 2019. "Local Job Multipliers in the United States: Variation with Local Characteristics and with High-Tech Shocks," Upjohn Working Papers 19-301, W.E. Upjohn Institute for Employment Research.
    15. Francis,David C. & Kubinec ,Robert, 2022. "Beyond Political Connections : A Measurement Model Approach to Estimating Firm-levelPolitical Influence in 41 Economies," Policy Research Working Paper Series 10119, The World Bank.
    16. Christopher J. Ruhm, 2019. "Shackling the Identification Police?," Southern Economic Journal, John Wiley & Sons, vol. 85(4), pages 1016-1026, April.
    17. Christos Genakos & Andreas Lamprinidis & James Walker, 2023. "Evaluating merger effects," CEP Discussion Papers dp1921, Centre for Economic Performance, LSE.
    18. Sauermann, Jan, 2015. "Worker Reciprocity and the Returns to Training: Evidence from a Field Experiment," IZA Discussion Papers 9179, Institute of Labor Economics (IZA).
    19. Katherine Sawyer & Kathleen Gallagher Cunningham & William Reed, 2017. "The Role of External Support in Civil War Termination," Journal of Conflict Resolution, Peace Science Society (International), vol. 61(6), pages 1174-1202, July.
    20. Joo, Hailey Hayeon & Lee, Jungmin, 2018. "Encountering female politicians," Journal of Economic Behavior & Organization, Elsevier, vol. 151(C), pages 88-122.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:h29xn_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.