IDEAS home Printed from https://ideas.repec.org/a/zbw/espost/215709.html
   My bibliography  Save this article

Can p-values be meaningfully interpreted without random sampling?

Author

Listed:
  • Hirschauer, Norbert
  • Grüner, Sven
  • Mußhoff, Oliver
  • Becker, Claudia
  • Jantsch, Antje

Abstract

Besides the inferential errors that abound in the interpretation of p-values, the probabilistic pre-conditions (i.e. random sampling or equivalent) for using them at all are not often met by observational studies in the social sciences. This paper systematizes different sampling designs and discusses the restrictive requirements of data collection that are the indispensable prerequisite for using p-values.

Suggested Citation

  • Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia & Jantsch, Antje, 2020. "Can p-values be meaningfully interpreted without random sampling?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 14, pages 71-91.
    • Handle: RePEc:zbw:espost:215709
    DOI: 10.1214/20-SS129
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/215709/1/Hirschauer_2020_p_values.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.1214/20-SS129?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. James J. Heckman, 1976. "The Common Structure of Statistical Models of Truncation, Sample Selection and Limited Dependent Variables and a Simple Estimator for Such Models," NBER Chapters, in: Annals of Economic and Social Measurement, Volume 5, number 4, pages 475-492, National Bureau of Economic Research, Inc.
    2. repec:bla:obuest:v:62:y:2000:i:2:p:305-22 is not listed on IDEAS
    3. Hirschauer, Norbert & Grüner, Sven & Mußhoff, Oliver & Becker, Claudia, 2020. "Inference in economic experiments," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 14, pages 1-14.
    4. Alberto Abadie & Susan Athey & Guido W Imbens & Jeffrey M Wooldridge, 2023. "When Should You Adjust Standard Errors for Clustering?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(1), pages 1-35.
    5. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 318, University of California, Davis, Department of Economics.
    6. Walter Krämer, 2011. "The Cult of Statistical Significance – What Economists Should and Should Not Do to Make their Data Talk," Schmollers Jahrbuch : Journal of Applied Social Science Studies / Zeitschrift für Wirtschafts- und Sozialwissenschaften, Duncker & Humblot, Berlin, vol. 131(3), pages 455-468.
    7. Rosenbaum, Paul R., 2010. "Design Sensitivity and Efficiency in Observational Studies," Journal of the American Statistical Association, American Statistical Association, vol. 105(490), pages 692-702.
    8. James G. MacKinnon, 2019. "How cluster-robust inference is changing applied econometrics," Canadian Journal of Economics, Canadian Economics Association, vol. 52(3), pages 851-881, August.
    9. J. B. Copas & H. G. Li, 1997. "Inference for Non‐random Samples," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 59(1), pages 55-95.
    10. A. Colin Cameron & Douglas L. Miller, 2015. "A Practitioner’s Guide to Cluster-Robust Inference," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 317-372.
    11. Byung‐Joo Lee & L. C. Marsh, 2000. "Sample Selection Bias Correction for Missing Response Observations," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 62(2), pages 305-322, May.
    12. Gary Solon & Steven J. Haider & Jeffrey M. Wooldridge, 2015. "What Are We Weighting For?," Journal of Human Resources, University of Wisconsin Press, vol. 50(2), pages 301-316.
    13. Vella, Francis & Verbeek, Marno, 1999. "Two-step estimation of panel data models with censored endogenous variables and selection bias," Journal of Econometrics, Elsevier, vol. 90(2), pages 239-263, June.
    14. Kevin E. Levay & Jeremy Freese & James N. Druckman, 2016. "The Demographic and Political Composition of Mechanical Turk Samples," SAGE Open, , vol. 6(1), pages 21582440166, March.
    15. Ronald L. Wasserstein & Nicole A. Lazar, 2016. "The ASA's Statement on p -Values: Context, Process, and Purpose," The American Statistician, Taylor & Francis Journals, vol. 70(2), pages 129-133, May.
    16. Greene, William H, 1981. "Sample Selection Bias as a Specification Error: Comment," Econometrica, Econometric Society, vol. 49(3), pages 795-798, May.
    17. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    18. Deirdre N. McCloskey & Stephen T. Ziliak, 1996. "The Standard Error of Regressions," Journal of Economic Literature, American Economic Association, vol. 34(1), pages 97-114, March.
    19. James J. Heckman & Hidehiko Ichimura & Petra E. Todd, 1997. "Matching As An Econometric Evaluation Estimator: Evidence from Evaluating a Job Training Programme," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 605-654.
    20. Joyce J Chen & Daniel Crown, 2019. "The Gender Pay Gap in Academia: Evidence from the Ohio State University," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 101(5), pages 1337-1352.
    21. Massenot, Baptiste & Pettinicchi, Yuri, 2018. "Can firms see into the future? Survey evidence from Germany," Journal of Economic Behavior & Organization, Elsevier, vol. 145(C), pages 66-79.
    22. A. Colin Cameron & Douglas L. Miller, 2010. "Robust Inference with Clustered Data," Working Papers 107, University of California, Davis, Department of Economics.
    23. Joyce J. Chen & Daniel Crown, 2019. "The Gender Pay Gap in Academia: Evidence from the Ohio State University," American Journal of Agricultural Economics, John Wiley & Sons, vol. 101(5), pages 1337-1352, October.
    24. David Trafimow, 2019. "Five Nonobvious Changes in Editorial Practice for Editors and Reviewers to Consider When Evaluating Submissions in a Post p," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 340-345, March.
    25. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    26. Andrew Gelman & John Carlin, 2017. "Some Natural Solutions to the -Value Communication Problem—and Why They Won’t Work," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 899-901, July.
    27. Donald Berry, 2017. "A -Value to Die For," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 112(519), pages 895-897, July.
    28. Alberto Abadie & Susan Athey & Guido W. Imbens & Jeffrey M. Wooldridge, 2014. "Finite Population Causal Standard Errors," NBER Working Papers 20325, National Bureau of Economic Research, Inc.
    29. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881, September.
    30. Hirschauer, Norbert & Gruener, Sven & Mußhoff, Oliver & Becker, Claudia, 2019. "Economic experiments and inference," SocArXiv 67mws, Center for Open Science.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Schröter, Iris & Mergenthaler, Marcus, 2021. "Applying the HEXACO Model of Personality to German Livestock Farmers: Item Scale Validation, Personality Structure and Influence on Participation in Livestock Certification Schemes," International Journal on Food System Dynamics, International Center for Management, Communication, and Research, vol. 12(03), September.
    2. Heckelei, Thomas & Huettel, Silke & Odening, Martin & Rommel, Jens, 2021. "The replicability crisis and the p-value debate – what are the consequences for the agricultural and food economics community?," Discussion Papers 316369, University of Bonn, Institute for Food and Resource Economics.
    3. Thomas Dufhues & Judith Möllers & Antje Jantsch & Gertrud Buchenrieder & Laura Camfield, 2023. "Don’t Look Up! Individual Income Comparisons and Subjective Well-Being of Students in Thailand," Journal of Happiness Studies, Springer, vol. 24(2), pages 477-503, February.
    4. Gruener, Sven, 2019. "An empirical study on Internet-based false news stories: experiences, problem awareness, and responsibilities," SocArXiv xbez9, Center for Open Science.
    5. Antje Jantsch & Gertrud Buchenrieder & Thomas Dufhues & Judith Möllers, 2024. "Social Comparisons Under Pandemic Stress: Income Reference Groups, Comparison Patterns, and the Subjective Well-Being of German Students," Journal of Happiness Studies, Springer, vol. 25(7), pages 1-24, October.
    6. , Hirschauer, 2022. "Some Thoughts About Statistical Inference In The 21st Century," SocArXiv exdfg, Center for Open Science.
    7. Peter Backus & Thien Nguyen, 2021. "The Effect of the Sex Buyer Law on the Market for Sex, Sexual Health and Sexual Violence," Economics Discussion Paper Series 2106, Economics, The University of Manchester.
    8. J. M. Santos & H. Horta & H. Luna, 2022. "The relationship between academics’ strategic research agendas and their preferences for basic research, applied research, or experimental development," Scientometrics, Springer;Akadémiai Kiadó, vol. 127(7), pages 4191-4225, July.
    9. Hirschauer, Norbert & Grüner , Sven, 2021. "A Primer on p-Value Thresholds and α-Levels – Two Different Kettles of Fish," German Journal of Agricultural Economics, Humboldt-Universitaet zu Berlin, Department for Agricultural Economics, vol. 70(02), January.
    10. Filiptseva, Anna & Filler, Günther & Odening, Martin, 2023. "Compensation schemes for plant quarantine pest costs: A case study for Germany," Economic Analysis and Policy, Elsevier, vol. 80(C), pages 1381-1395.
    11. Krzysztof Jajuga & Józef Pociecha & Mirosław Szreder, 2024. "Statistical inference and statistical learning in economic research – selected challenges," Ekonomista, Polskie Towarzystwo Ekonomiczne, issue 2, pages 138-154.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Miguel Santolino & Magnus Söderberg, 2014. "Modelling appellate courts’ responses in motor injury disputes," European Journal of Law and Economics, Springer, vol. 38(3), pages 393-407, December.
    2. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    3. Morrissey, Karyn & Kinderman, Peter & Pontin, Eleanor & Tai, Sara & Schwannauer, Mathias, 2016. "Web based health surveys: Using a Two Step Heckman model to examine their potential for population health analysis," Social Science & Medicine, Elsevier, vol. 163(C), pages 45-53.
    4. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    5. James G. MacKinnon & Matthew D. Webb, 2020. "When and How to Deal with Clustered Errors in Regression Models," Working Paper 1421, Economics Department, Queen's University.
    6. Embaye, Weldensie T. & Bergtold, Jason S. & Archer, David & Flora, Cornelia & Andrango, Graciela C. & Odening, Marting & Buysse, Jeroen, 2018. "Examining farmers' willingness to grow and allocate land for oilseed crops for biofuel production," Energy Economics, Elsevier, vol. 71(C), pages 311-320.
    7. Paul Hunermund & Elias Bareinboim, 2019. "Causal Inference and Data Fusion in Econometrics," Papers 1912.09104, arXiv.org, revised Mar 2023.
    8. Matthew D. Webb, 2023. "Reworking wild bootstrap‐based inference for clustered errors," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 56(3), pages 839-858, August.
    9. Clément de Chaisemartin & Jaime Ramirez-Cuellar, 2024. "At What Level Should One Cluster Standard Errors in Paired and Small-Strata Experiments?," American Economic Journal: Applied Economics, American Economic Association, vol. 16(1), pages 193-212, January.
    10. P.W. Miller & S. Rummery, 1989. "Gender Wage Discrimination in Australia: A reassessment," Economics Discussion / Working Papers 89-21, The University of Western Australia, Department of Economics.
    11. Takashi Yamagata & Chris Orme, 2005. "On Testing Sample Selection Bias Under the Multicollinearity Problem," Econometric Reviews, Taylor & Francis Journals, vol. 24(4), pages 467-481.
    12. Benjamin L. Collier & Andrew F. Haughwout & Howard C. Kunreuther & Erwann O. Michel‐Kerjan, 2020. "Firms’ Management of Infrequent Shocks," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 52(6), pages 1329-1359, September.
    13. Sloczynski, Tymon, 2018. "A General Weighted Average Representation of the Ordinary and Two-Stage Least Squares Estimands," IZA Discussion Papers 11866, Institute of Labor Economics (IZA).
    14. Turati, Riccardo, 2024. "Network Abroad and Culture: Global Individual-Level Evidence," GLO Discussion Paper Series 1488, Global Labor Organization (GLO).
    15. Ian Gazeley & Rose Holmes & Andrew Newell & Kevin Reynolds & Hector Gutierrez Rufrancos, 2023. "Escaping from hunger before WW1: the nutritional transition and living standards in Western Europe and USA in the late nineteenth century," Cliometrica, Springer;Cliometric Society (Association Francaise de Cliométrie), vol. 17(3), pages 533-565, September.
    16. Gregory S. Crawford & Nicola Pavanini & Fabiano Schivardi, 2018. "Asymmetric Information and Imperfect Competition in Lending Markets," American Economic Review, American Economic Association, vol. 108(7), pages 1659-1701, July.
    17. Jeffrey Smith & Arthur Sweetman, 2016. "Viewpoint: Estimating the causal effects of policies and programs," Canadian Journal of Economics, Canadian Economics Association, vol. 49(3), pages 871-905, August.
    18. Jan Stede, 2019. "Do Energy Efficiency Networks Save Energy? Evidence from German Plant-Level Data," Discussion Papers of DIW Berlin 1813, DIW Berlin, German Institute for Economic Research.
    19. Richard Blundell & Monica Costa Dias, 2009. "Alternative Approaches to Evaluation in Empirical Microeconomics," Journal of Human Resources, University of Wisconsin Press, vol. 44(3).
    20. Bruno Ferman, 2023. "Inference in difference‐in‐differences: How much should we trust in independent clusters?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(3), pages 358-369, April.

    More about this item

    Keywords

    sampling design; propensity scores; p-values; random sampling; selection bias; inference;
    All these keywords.

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:espost:215709. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/zbwkide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.