IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/cahyq.html
   My bibliography  Save this paper

Bayesian Specification Curve Analysis

Author

Listed:
  • Semken, Christoph
  • Rossell, David

Abstract

Science suffers from a reproducibility crisis. Specification Curve Analysis (SCA) helps address this crisis by preventing the selective reporting of results and arbitrary data analysis choices. SCA plots the variability (or heterogeneity) of treatment effects against all ‘reasonable specifications’ (ways to conduct analysis). However, SCA has also been used for formal statistical inference on a type of global average (median) treatment effect (ATE), leading a study by Orben & Przybylski to conclude that ‘the association of [adolescent mental] well-being with regularly eating potatoes was nearly as negative as the association with technology use.’ In contrast, we find relevant associations between certain technologies and well-being, and sharp discrepancies between parent and teenager assessments. These heterogeneous effects are masked by taking medians. In layman’s terms, an ATE may appear practically irrelevant due to averaging over apples and oranges. In addition, the SCA median can have large bias and variance, due to over-weighting statistically implausible control variable specifications. With the Bayesian Specification Curve Analysis (BSCA) we extend SCA to estimate both individual and, if desired, average treatment effects, with controls weighted via Bayesian Model Averaging. The strategy allows to test individual effects, a missing feature in SCA, while improving statistical properties and protecting against false positives. We provide R code that implements BSCA and reproduces our analyses.

Suggested Citation

  • Semken, Christoph & Rossell, David, 2020. "Bayesian Specification Curve Analysis," OSF Preprints cahyq, Center for Open Science.
    • Handle: RePEc:osf:osfxxx:cahyq
    DOI: 10.31219/osf.io/cahyq
    as

    Download full text from publisher

    File URL: https://osf.io/download/5fa57ec323568c0210d653f8/
    Download Restriction: no
    File URL: https://libkey.io/10.31219/osf.io/cahyq?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    2. Jean M. Twenge & Jonathan Haidt & Thomas E. Joiner & W. Keith Campbell, 2020. "Underestimating digital media harm," Nature Human Behaviour, Nature, vol. 4(4), pages 346-348, April.
    3. Michael Muthukrishna & Joseph Henrich, 2019. "A problem in theory," Nature Human Behaviour, Nature, vol. 3(3), pages 221-229, March.
    4. Hunt Allcott & Luca Braghieri & Sarah Eichmeyer & Matthew Gentzkow, 2020. "The Welfare Effects of Social Media," American Economic Review, American Economic Association, vol. 110(3), pages 629-676, March.
    5. Amy Orben & Andrew K. Przybylski, 2019. "The association between adolescent well-being and digital technology use," Nature Human Behaviour, Nature, vol. 3(2), pages 173-182, February.
    6. Uri Simonsohn & Joseph P. Simmons & Leif D. Nelson, 2020. "Specification curve analysis," Nature Human Behaviour, Nature, vol. 4(11), pages 1208-1214, November.
    7. Adam Slez, 2019. "The Difference Between Instability and Uncertainty: Comment on Young and Holsteen (2017)," Sociological Methods & Research, , vol. 48(2), pages 400-430, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christoph Semken & David Rossell, 2022. "Specification analysis for technology use and teenager well‐being: Statistical validity and a Bayesian proposal," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 71(5), pages 1330-1355, November.
    2. Cantone, Giulio Giacomo & Tomaselli, Venera, 2024. "On the Coherence of Composite Indexes: Multiversal Model and Specification Analysis for an Index of Well-Being," MetaArXiv d5y26, Center for Open Science.
    3. Schweinsberg, Martin & Feldman, Michael & Staub, Nicola & van den Akker, Olmo R. & van Aert, Robbie C.M. & van Assen, Marcel A.L.M. & Liu, Yang & Althoff, Tim & Heer, Jeffrey & Kale, Alex & Mohamed, Z, 2021. "Same data, different conclusions: Radical dispersion in empirical results when independent analysts operationalize and test the same hypothesis," Organizational Behavior and Human Decision Processes, Elsevier, vol. 165(C), pages 228-249.
    4. Byrnes, Alex, 2024. "Reanalysis of Sanders et al. (2024): An Umbrella Review of the Benefits and Risks Associated with Youths' Interactions with Electronic Screens," I4R Discussion Paper Series 154, The Institute for Replication (I4R).
    5. Linda K. Kaye & Amy Orben & David A. Ellis & Simon C. Hunter & Stephen Houghton, 2020. "The Conceptual and Methodological Mayhem of “Screen Time”," IJERPH, MDPI, vol. 17(10), pages 1-10, May.
    6. Hoong, Ruru, 2021. "Self control and smartphone use: An experimental study of soft commitment devices," European Economic Review, Elsevier, vol. 140(C).
    7. Ebersberger, Bernd & Kuckertz, Andreas, 2021. "Hop to it! The impact of organization type on innovation response time to the COVID-19 crisis," Journal of Business Research, Elsevier, vol. 124(C), pages 126-135.
    8. Jürges, Hendrik & Khanam, Rasheda, 2021. "Adolescents’ time allocation and skill production," Economics of Education Review, Elsevier, vol. 85(C).
    9. Alexander Frankel & Maximilian Kasy, 2022. "Which Findings Should Be Published?," American Economic Journal: Microeconomics, American Economic Association, vol. 14(1), pages 1-38, February.
    10. Jyotirmoy Sarkar, 2018. "Will P†Value Triumph over Abuses and Attacks?," Biostatistics and Biometrics Open Access Journal, Juniper Publishers Inc., vol. 7(4), pages 66-71, July.
    11. Wu, Haitao & Ba, Ning & Ren, Siyu & Xu, Lu & Chai, Jingxia & Irfan, Muhammad & Hao, Yu & Lu, Zhi-Nan, 2022. "The impact of internet development on the health of Chinese residents: Transmission mechanisms and empirical tests," Socio-Economic Planning Sciences, Elsevier, vol. 81(C).
    12. Patricia A. Norberg & Daniel R. Horne, 2024. "When the bridge is not human: Algorithmic interference in forming social relationships through the manipulation of weak ties," Journal of Consumer Affairs, Wiley Blackwell, vol. 58(2), pages 606-629, June.
    13. Omar Hegazi & Samer Alalalmeh & Ahmad Alfaresi & Soheil Dashtinezhad & Ahmed Bahada & Moyad Shahwan & Ammar Abdulrahman Jairoun & Tesleem K. Babalola & Haya Yasin, 2022. "Development, Validation, and Utilization of a Social Media Use and Mental Health Questionnaire among Middle Eastern and Western Adults: A Pilot Study from the UAE," IJERPH, MDPI, vol. 19(23), pages 1-18, November.
    14. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    15. Maurizio Pugno, 2024. "Social media effects on well‐being: The hypothesis of addiction of a new variety," Kyklos, Wiley Blackwell, vol. 77(3), pages 690-704, August.
    16. Karin Langenkamp & Bodo Rödel & Kerstin Taufenbach & Meike Weiland, 2018. "Open Access in Vocational Education and Training Research," Publications, MDPI, vol. 6(3), pages 1-12, July.
    17. Nicolás Ajzenman & Bruno Ferman & Pedro C. Sant’Anna, 2023. "Rooting for the Same Team: On the Interplay between Political and Social Identities in the Formation of Social Ties," Working Papers 231, Red Nacional de Investigadores en Economía (RedNIE).
    18. Kevin J. Boyle & Mark Morrison & Darla Hatton MacDonald & Roderick Duncan & John Rose, 2016. "Investigating Internet and Mail Implementation of Stated-Preference Surveys While Controlling for Differences in Sample Frames," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 64(3), pages 401-419, July.
    19. McNamara, Trent & Mosquera, Roberto, 2024. "The political divide: The case of expectations and preferences," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 110(C).
    20. Jelte M Wicherts & Marjan Bakker & Dylan Molenaar, 2011. "Willingness to Share Research Data Is Related to the Strength of the Evidence and the Quality of Reporting of Statistical Results," PLOS ONE, Public Library of Science, vol. 6(11), pages 1-7, November.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:cahyq. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.