IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/cahyq.html
   My bibliography  Save this paper

Bayesian Specification Curve Analysis

Author

Listed:
  • Semken, Christoph
  • Rossell, David

Abstract

Science suffers from a reproducibility crisis. Specification Curve Analysis (SCA) helps address this crisis by preventing the selective reporting of results and arbitrary data analysis choices. SCA plots the variability (or heterogeneity) of treatment effects against all ‘reasonable specifications’ (ways to conduct analysis). However, SCA has also been used for formal statistical inference on a type of global average (median) treatment effect (ATE), leading a study by Orben & Przybylski to conclude that ‘the association of [adolescent mental] well-being with regularly eating potatoes was nearly as negative as the association with technology use.’ In contrast, we find relevant associations between certain technologies and well-being, and sharp discrepancies between parent and teenager assessments. These heterogeneous effects are masked by taking medians. In layman’s terms, an ATE may appear practically irrelevant due to averaging over apples and oranges. In addition, the SCA median can have large bias and variance, due to over-weighting statistically implausible control variable specifications. With the Bayesian Specification Curve Analysis (BSCA) we extend SCA to estimate both individual and, if desired, average treatment effects, with controls weighted via Bayesian Model Averaging. The strategy allows to test individual effects, a missing feature in SCA, while improving statistical properties and protecting against false positives. We provide R code that implements BSCA and reproduces our analyses.

Suggested Citation

  • Semken, Christoph & Rossell, David, 2020. "Bayesian Specification Curve Analysis," OSF Preprints cahyq, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:cahyq
    DOI: 10.31219/osf.io/cahyq
    as

    Download full text from publisher

    File URL: https://osf.io/download/5fa57ec323568c0210d653f8/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/cahyq?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    2. Jean M. Twenge & Jonathan Haidt & Thomas E. Joiner & W. Keith Campbell, 2020. "Underestimating digital media harm," Nature Human Behaviour, Nature, vol. 4(4), pages 346-348, April.
    3. Michael Muthukrishna & Joseph Henrich, 2019. "A problem in theory," Nature Human Behaviour, Nature, vol. 3(3), pages 221-229, March.
    4. Hunt Allcott & Luca Braghieri & Sarah Eichmeyer & Matthew Gentzkow, 2020. "The Welfare Effects of Social Media," American Economic Review, American Economic Association, vol. 110(3), pages 629-676, March.
    5. Amy Orben & Andrew K. Przybylski, 2019. "The association between adolescent well-being and digital technology use," Nature Human Behaviour, Nature, vol. 3(2), pages 173-182, February.
    6. Uri Simonsohn & Joseph P. Simmons & Leif D. Nelson, 2020. "Specification curve analysis," Nature Human Behaviour, Nature, vol. 4(11), pages 1208-1214, November.
    7. Adam Slez, 2019. "The Difference Between Instability and Uncertainty: Comment on Young and Holsteen (2017)," Sociological Methods & Research, , vol. 48(2), pages 400-430, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christoph Semken & David Rossell, 2022. "Specification analysis for technology use and teenager well‐being: Statistical validity and a Bayesian proposal," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(5), pages 1330-1355, November.
    2. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    3. Byrnes, Alex, 2024. "Reanalysis of Sanders et al. (2024): An Umbrella Review of the Benefits and Risks Associated with Youths' Interactions with Electronic Screens," I4R Discussion Paper Series 154, The Institute for Replication (I4R).
    4. Linda K. Kaye & Amy Orben & David A. Ellis & Simon C. Hunter & Stephen Houghton, 2020. "The Conceptual and Methodological Mayhem of “Screen Time”," IJERPH, MDPI, vol. 17(10), pages 1-10, May.
    5. Hoong, Ruru, 2021. "Self control and smartphone use: An experimental study of soft commitment devices," European Economic Review, Elsevier, vol. 140(C).
    6. Jürges, Hendrik & Khanam, Rasheda, 2021. "Adolescents’ time allocation and skill production," Economics of Education Review, Elsevier, vol. 85(C).
    7. Cantone, Giulio Giacomo & Tomaselli, Venera, 2024. "On the Coherence of Composite Indexes: Multiversal Model and Specification Analysis for an Index of Well-Being," MetaArXiv d5y26, Center for Open Science.
    8. Ebersberger, Bernd & Kuckertz, Andreas, 2021. "Hop to it! The impact of organization type on innovation response time to the COVID-19 crisis," Journal of Business Research, Elsevier, vol. 124(C), pages 126-135.
    9. Jyotirmoy Sarkar, 2018. "Will P†Value Triumph over Abuses and Attacks?," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 7(4), pages 66-71, July.
    10. Patricia A. Norberg & Daniel R. Horne, 2024. "When the bridge is not human: Algorithmic interference in forming social relationships through the manipulation of weak ties," Journal of Consumer Affairs, Wiley Blackwell, vol. 58(2), pages 606-629, June.
    11. Omar Hegazi & Samer Alalalmeh & Ahmad Alfaresi & Soheil Dashtinezhad & Ahmed Bahada & Moyad Shahwan & Ammar Abdulrahman Jairoun & Tesleem K. Babalola & Haya Yasin, 2022. "Development, Validation, and Utilization of a Social Media Use and Mental Health Questionnaire among Middle Eastern and Western Adults: A Pilot Study from the UAE," IJERPH, MDPI, vol. 19(23), pages 1-18, November.
    12. Nicolás Ajzenman & Bruno Ferman & Pedro C. Sant’Anna, 2023. "Rooting for the Same Team: On the Interplay between Political and Social Identities in the Formation of Social Ties," Working Papers 231, Red Nacional de Investigadores en Economía (RedNIE).
    13. Kevin J. Boyle & Mark Morrison & Darla Hatton MacDonald & Roderick Duncan & John Rose, 2016. "Investigating Internet and Mail Implementation of Stated-Preference Surveys While Controlling for Differences in Sample Frames," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 64(3), pages 401-419, July.
    14. Jelte M Wicherts & Marjan Bakker & Dylan Molenaar, 2011. "Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-7, November.
    15. Larsen, Vegard H. & Thorsrud, Leif Anders & Zhulanova, Julia, 2021. "News-driven inflation expectations and information rigidities," Journal of Monetary Economics, Elsevier, vol. 117(C), pages 507-520.
    16. Eibich, Peter & Goldzahl, Léontine, 2021. "Does retirement affect secondary preventive care use? Evidence from breast cancer screening," Economics & Human Biology, Elsevier, vol. 43(C).
    17. Frederique Bordignon, 2020. "Self-correction of science: a comparative study of negative citations and post-publication peer review," Scientometrics, Springer;Akadémiai Kiadó, vol. 124(2), pages 1225-1239, August.
    18. Omar Al-Ubaydli & John A. List, 2015. "Do Natural Field Experiments Afford Researchers More or Less Control than Laboratory Experiments? A Simple Model," NBER Working Papers 20877, National Bureau of Economic Research, Inc.
    19. Kovacic, Matija & Orso, Cristina Elisa, 2024. "Adverse childhood experiences and social media use in adulthood. Evidence from a novel EU survey," GLO Discussion Paper Series 1531, Global Labor Organization (GLO).
    20. Cook, Joseph & Kabubo-Mariara, Jane & Kimuyu, Peter, 2021. "The Short-Run Impacts of Reducing Water Collection Times on Time Use, Well-Being and Education in Rural Kenya," EfD Discussion Paper 21-9, Environment for Development, University of Gothenburg.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:cahyq. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.