IDEAS home Printed from https://ideas.repec.org/p/arx/papers/1603.07532.html
   My bibliography  Save this paper

A Short Note on P-Value Hacking

Author

Listed:
  • Nassim Nicholas Taleb

Abstract

We present the expected values from p-value hacking as a choice of the minimum p-value among $m$ independents tests, which can be considerably lower than the "true" p-value, even with a single trial, owing to the extreme skewness of the meta-distribution. We first present an exact probability distribution (meta-distribution) for p-values across ensembles of statistically identical phenomena. We derive the distribution for small samples $2

Suggested Citation

  • Nassim Nicholas Taleb, 2016. "A Short Note on P-Value Hacking," Papers 1603.07532, arXiv.org, revised Jan 2018.
    • Handle: RePEc:arx:papers:1603.07532
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/1603.07532
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Gelman, Andrew & Stern, Hal, 2006. "The Difference Between," The American Statistician, American Statistical Association, vol. 60, pages 328-331, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Howley, P.; & Moro, M.; & Waqas, M.; & Delaney, L.; & Heron, T.;, 2018. "Immigration and self-reported well-being in the UK," Health, Econometrics and Data Group (HEDG) Working Papers 18/12, HEDG, c/o Department of Economics, University of York.
    2. Diana M. Hechavarría & Steven A. Brieger, 2022. "Practice rather than preach: cultural practices and female social entrepreneurship," Small Business Economics, Springer, vol. 58(2), pages 1131-1151, February.
    3. Thomas Neise & Franziska Sohns & Moritz Breul & Javier Revilla Diez, 2022. "The effect of natural disasters on FDI attraction: a sector-based analysis over time and space," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 110(2), pages 999-1023, January.
    4. Colin F. Camerer & Anna Dreber & Felix Holzmeister & Teck-Hua Ho & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Gideon Nave & Brian A. Nosek & Thomas Pfeiffer & Adam Altmejd & Nick Buttrick , 2018. "Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015," Nature Human Behaviour, Nature, vol. 2(9), pages 637-644, September.
    5. Asongu, Simplice & Odhiambo, Nicholas, 2020. "The role of governance in quality education in sub-Saharan Africa," MPRA Paper 107497, University Library of Munich, Germany.
    6. Ruomeng Cui & Jun Li & Dennis J. Zhang, 2020. "Reducing Discrimination with Reviews in the Sharing Economy: Evidence from Field Experiments on Airbnb," Management Science, INFORMS, vol. 66(3), pages 1071-1094, March.
    7. Lukas Haffert, 2019. "War mobilization or war destruction? The unequal rise of progressive taxation revisited," The Review of International Organizations, Springer, vol. 14(1), pages 59-82, March.
    8. Michael A. Allen & Michael E. Flynn & Julie VanDusky-Allen, 2017. "Regions of Hierarchy and Security: US Troop Deployments, Spatial Relations, and Defense Burdens," International Interactions, Taylor & Francis Journals, vol. 43(3), pages 397-423, May.
    9. Duxbury, Scott W, 2019. "Mediation and Moderation in Statistical Network Models," SocArXiv 9bs4u, Center for Open Science.
    10. Roberta Rutigliano & Gøsta Esping-Andersen, 2018. "Partnership Choice and Childbearing in Norway and Spain," European Journal of Population, Springer;European Association for Population Studies, vol. 34(3), pages 367-386, August.
    11. David Spiegelhalter, 2017. "Trust in numbers," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(4), pages 948-965, October.
    12. Andrew P. Jaciw & Li Lin & Boya Ma, 2016. "An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials," Evaluation Review, , vol. 40(5), pages 410-443, October.
    13. Peter Howley & Muhammad Waqas & Mirko Moro & Liam Delaney & Tony Heron, 2020. "It’s Not All about the Economy Stupid! Immigration and Subjective Well-Being in England," Work, Employment & Society, British Sociological Association, vol. 34(5), pages 919-936, October.
    14. Melisa Bubonya & Deborah A. Cobb-Clark & Daniel Christensen & Sarah E. Johnson & Stephen R. Zubrick, 2019. "The Great Recession and Children’s Mental Health in Australia," IJERPH, MDPI, vol. 16(4), pages 1-19, February.
    15. Anderson, Brian S. & Wennberg, Karl & McMullen, Jeffery S., 2019. "Editorial: Enhancing quantitative theory-testing entrepreneurship research," Journal of Business Venturing, Elsevier, vol. 34(5), pages 1-1.
    16. Wennberg, Karl & Anderson, Brian S. & McMullen, Jeffrey, 2019. "2 Editorial: Enhancing Quantitative Theory-Testing Entrepreneurship Research," Ratio Working Papers 323, The Ratio Institute.
    17. Shilpi Agrawala & Lindsey A Sjaarda & Ukpebo R Omosigho & Neil J Perkins & Robert M Silver & Sunni L Mumford & Matthew T Connell & Ashley I Naimi & Lisa M Halvorson & Enrique F Schisterman, 2019. "Effect of preconception low dose aspirin on pregnancy and live birth according to socioeconomic status: A secondary analysis of a randomized clinical trial," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-14, April.
    18. Byzalov, Dmitri & Basu, Sudipta, 2019. "Modeling the determinants of meet-or-just-beat behavior in distribution discontinuity tests," Journal of Accounting and Economics, Elsevier, vol. 68(2).
    19. Ignacio, Escañuela Romana, 2019. "The elasticities of passenger transport demand in the Northeast Corridor," Research in Transportation Economics, Elsevier, vol. 78(C).
    20. Gezahegn, Tafesse & Van Passel, Steven & Berhanu, Tekeste & D'Haese, Marijke & Maertens, Miet, 2020. "Structural and Institutional Heterogeneity among Agricultural Cooperatives in Ethiopia: Does it Matter for Farmers’ Welfare?," Journal of Agricultural and Resource Economics, Western Agricultural Economics Association, vol. 46(2), August.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:1603.07532. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.