IDEAS home Printed from https://ideas.repec.org/p/osf/metaar/s42ba_v1.html
   My bibliography  Save this paper

Achieving Statistical Significance with Covariates and without Transparency

Author

Listed:
  • Lenz, Gabriel

    (UC Berkeley)

  • Sahn, Alexander

    (University of North Carolina, Chapel Hill)

Abstract

How often do articles depend on suppression effects for their findings? How often do they disclose this fact? By suppression effects, we mean control-variable-induced increases in estimated effect sizes. Researchers generally scrutinize suppression effects as they want reassurance that researchers have a strong explanation for them, especially when the statistical significance of the key finding depends on them. In a re-analysis of observational studies from a leading journal, we find that over 30% of articles depend on suppression effects for statistical significance. Although increases in key effect estimates from including control variables are of course potentially justifiable, none of the articles justify or disclose them. These findings may point to a hole in the review process: journals are accepting articles that depend on suppression effects without readers, reviewers, or editors being made aware.

Suggested Citation

  • Lenz, Gabriel & Sahn, Alexander, 2017. "Achieving Statistical Significance with Covariates and without Transparency," MetaArXiv s42ba_v1, Center for Open Science.
    • Handle: RePEc:osf:metaar:s42ba_v1
    DOI: 10.31219/osf.io/s42ba_v1
    as

    Download full text from publisher

    File URL: https://osf.io/download/58e82561594d9002510ab37f/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/s42ba_v1?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Hainmueller, Jens, 2012. "Entropy Balancing for Causal Effects: A Multivariate Reweighting Method to Produce Balanced Samples in Observational Studies," Political Analysis, Cambridge University Press, vol. 20(1), pages 25-46, January.
    2. Jonah B. Gelbach, 2016. "When Do Covariates Matter? And Which Ones, and How Much?," Journal of Labor Economics, University of Chicago Press, vol. 34(2), pages 509-543.
    3. Katherine Casey & Rachel Glennerster & Edward Miguel, 2012. "Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(4), pages 1755-1812.
    4. Wooldridge, Jeffrey M., 2016. "Should instrumental variables be used as matching variables?," Research in Economics, Elsevier, vol. 70(2), pages 232-237.
    5. Ho, Daniel E. & Imai, Kosuke & King, Gary & Stuart, Elizabeth A., 2007. "Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference," Political Analysis, Cambridge University Press, vol. 15(3), pages 199-236, July.
    6. Kosuke Imai & Marc Ratkovic, 2014. "Covariate balancing propensity score," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 76(1), pages 243-263, January.
    7. Jonas Peters & Peter Bühlmann & Nicolai Meinshausen, 2016. "Causal inference by using invariant prediction: identification and confidence intervals," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 78(5), pages 947-1012, November.
    8. Wyss Richard & Lunt Mark & Brookhart M. Alan & Glynn Robert J. & Stürmer Til, 2014. "Reducing Bias Amplification in the Presence of Unmeasured Confounding through Out-of-Sample Estimation Strategies for the Disease Risk Score," Journal of Causal Inference, De Gruyter, vol. 2(2), pages 131-146, September.
    9. Sekhon, Jasjeet S., 2011. "Multivariate and Propensity Score Matching Software with Automated Balance Optimization: The Matching package for R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 42(i07).
    10. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2005. "An Evaluation of Instrumental Variable Strategies for Estimating the Effects of Catholic Schooling," Journal of Human Resources, University of Wisconsin Press, vol. 40(4), pages 791-821.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lenz, Gabriel & Sahn, Alexander, 2017. "Achieving Statistical Significance with Covariates and without Transparency," MetaArXiv s42ba, Center for Open Science.
    2. Adeola Oyenubi & Martin Wittenberg, 2021. "Does the choice of balance-measure matter under genetic matching?," Empirical Economics, Springer, vol. 61(1), pages 489-502, July.
    3. Baron, Opher & Callen, Jeffrey L. & Segal, Dan, 2023. "Does the bullwhip matter economically? A cross-sectional firm-level analysis," International Journal of Production Economics, Elsevier, vol. 259(C).
    4. Frölich, Markus & Huber, Martin & Wiesenfarth, Manuel, 2017. "The finite sample performance of semi- and non-parametric estimators for treatment effects and policy evaluation," Computational Statistics & Data Analysis, Elsevier, vol. 115(C), pages 91-102.
    5. Hugo Bodory & Lorenzo Camponovo & Martin Huber & Michael Lechner, 2020. "The Finite Sample Performance of Inference Methods for Propensity Score Matching and Weighting Estimators," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 38(1), pages 183-200, January.
    6. Jason J. Sauppe & Sheldon H. Jacobson, 2017. "The role of covariate balance in observational studies," Naval Research Logistics (NRL), John Wiley & Sons, vol. 64(4), pages 323-344, June.
    7. Fukui Hideki, 2023. "Evaluating Different Covariate Balancing Methods: A Monte Carlo Simulation," Statistics, Politics and Policy, De Gruyter, vol. 14(2), pages 205-326, June.
    8. Bensch, Gunther & Kluve, Jochen & Stöterau, Jonathan, 2021. "The market-based dissemination of energy-access technologies as a business model for rural entrepreneurs: Evidence from Kenya," Resource and Energy Economics, Elsevier, vol. 66(C).
    9. Cousineau, Martin & Verter, Vedat & Murphy, Susan A. & Pineau, Joelle, 2023. "Estimating causal effects with optimization-based methods: A review and empirical comparison," European Journal of Operational Research, Elsevier, vol. 304(2), pages 367-380.
    10. Martin Cousineau & Vedat Verter & Susan A. Murphy & Joelle Pineau, 2022. "Estimating causal effects with optimization-based methods: A review and empirical comparison," Papers 2203.00097, arXiv.org.
    11. Zichen Deng & Maarten Lindeboom, 2021. "Early-life Famine Exposure, Hunger Recall and Later-life Health," Tinbergen Institute Discussion Papers 21-054/V, Tinbergen Institute.
    12. Pedro H. C. Sant'Anna & Xiaojun Song & Qi Xu, 2022. "Covariate distribution balance via propensity scores," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(6), pages 1093-1120, September.
    13. Renata Baborska & Emilio Hernandez & Emiliano Magrini & Cristian Morales-Opazo, 2020. "The impact of financial inclusion on rural food security experience: A perspective from low-and middle-income countries," Review of Development Finance Journal, Chartered Institute of Development Finance, vol. 10(2), pages 1-18.
    14. Jasjeet Singh Sekhon & Richard D. Grieve, 2012. "A matching method for improving covariate balance in cost‐effectiveness analyses," Health Economics, John Wiley & Sons, Ltd., vol. 21(6), pages 695-714, June.
    15. Macchioni Giaquinto, Annarita & Jones, Andrew M. & Rice, Nigel & Zantomio, Francesca, 2021. "Labour supply and informal care responses to health shocks within couples: evidence from the UKHLS," GLO Discussion Paper Series 806, Global Labor Organization (GLO).
    16. Dasom Lee & Shu Yang & Lin Dong & Xiaofei Wang & Donglin Zeng & Jianwen Cai, 2023. "Improving trial generalizability using observational studies," Biometrics, The International Biometric Society, vol. 79(2), pages 1213-1225, June.
    17. Ubaldi, Michele & Picchio, Matteo, 2023. "Intergenerational scars: The impact of parental unemployment on individual health later in life," GLO Discussion Paper Series 1271, Global Labor Organization (GLO).
    18. Robert J. Johnston & Klaus Moeltner, 2019. "Special Flood Hazard Effects on Coastal and Interior Home Values: One Size Does Not Fit All," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 74(1), pages 181-210, September.
    19. Söhnke M Bartram & Jennifer Conrad & Jongsub Lee & Marti G Subrahmanyam, 2022. "Credit Default Swaps around the World," The Review of Financial Studies, Society for Financial Studies, vol. 35(5), pages 2464-2524.
    20. Neffke, Frank & Nedelkoska, Ljubica & Wiederhold, Simon, 2024. "Skill mismatch and the costs of job displacement," Research Policy, Elsevier, vol. 53(2).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:metaar:s42ba_v1. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/metaarxiv .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.