IDEAS home Printed from https://ideas.repec.org/a/bla/istatr/v88y2020i3p642-657.html
   My bibliography  Save this article

Is there a 'safe area' where the nonresponse rate has only a modest effect on bias despite non‐ignorable nonresponse?

Author

Listed:
  • Dan Hedlin

Abstract

Rising nonresponse rates in social surveys make the issue of nonresponse bias contentious. There are conflicting messages about the importance of high response rates and the hazards of low rates. Some articles (e.g. Groves and Peytcheva, 2008) suggest that the response rate is in general not a good predictor of survey quality. Equally, it is well known that nonresponse may induce bias and increase data collection costs. We go back in the history of the literature of nonresponse and suggest a possible reason to the notion that even a rather small nonresponse rate makes the quality of a survey debatable. We also explore the relationship between nonresponse rate and bias, assuming non‐ignorable nonresponse and focusing on estimates of totals or means. We show that there is a ‘safe area’ enclosed by the response rate on the one hand and the correlation between the response propensity and the study variable on the other hand; in this area, (1) the response rate does not greatly affect the nonresponse bias, and (2) the nonresponse bias is small.

Suggested Citation

  • Dan Hedlin, 2020. "Is there a 'safe area' where the nonresponse rate has only a modest effect on bias despite non‐ignorable nonresponse?," International Statistical Review, International Statistical Institute, vol. 88(3), pages 642-657, December.
    • Handle: RePEc:bla:istatr:v:88:y:2020:i:3:p:642-657
    DOI: 10.1111/insr.12359
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/insr.12359
    Download Restriction: no
    File URL: https://libkey.io/10.1111/insr.12359?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Shlomo, Natalie & Skinner, Chris J. & Schouten, Barry, 2012. "Estimation of an indicator of the representativeness of survey response," LSE Research Online Documents on Economics 39124, London School of Economics and Political Science, LSE Library.
    2. John Micklewright & Sylke V. Schnepf & Chris Skinner, 2012. "Non-response biases in surveys of schoolchildren: the case of the English Programme for International Student Assessment (PISA) samples," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 175(4), pages 915-938, October.
    3. J. Michael Brick & Michael E. Jones, 2008. "Propensity to respond and nonresponse bias," Metron - International Journal of Statistics, Dipartimento di Statistica, Probabilità e Statistiche Applicate - University of Rome, vol. 0(1), pages 51-73.
    4. John L. Czajka & Amy Beyler, "undated". "Declining Response Rates in Federal Surveys: Trends and Implications (Background Paper)," Mathematica Policy Research Reports a714f76e878f4a74a6ad9f15d, Mathematica Policy Research.
    5. F. Kreuter & K. Olson & J. Wagner & T. Yan & T. M. Ezzati‐Rice & C. Casas‐Cordero & M. Lemay & A. Peytchev & R. M. Groves & T. E. Raghunathan, 2010. "Using proxy measures and other correlates of survey outcomes to adjust for non‐response: examples from multiple surveys," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(2), pages 389-407, April.
    6. Jamie C. Moore & Gabriele B. Durrant & Peter W. F. Smith, 2018. "Data set representativeness during data collection in three UK social surveys: generalizability and the effects of auxiliary covariate choice," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(1), pages 229-248, January.
    7. Houshmand Shirani-Mehr & David Rothschild & Sharad Goel & Andrew Gelman, 2018. "Disentangling Bias and Variance in Election Polls," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 607-614, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Brick J. Michael, 2013. "Unit Nonresponse and Weighting Adjustments: A Critical Review," Journal of Official Statistics, Sciendo, vol. 29(3), pages 329-353, June.
    2. Roberts Caroline & Vandenplas Caroline & Herzing Jessica M.E., 2020. "A Validation of R-Indicators as a Measure of the Risk of Bias using Data from a Nonresponse Follow-Up Survey," Journal of Official Statistics, Sciendo, vol. 36(3), pages 675-701, September.
    3. Gabriele B. Durrant & Sylke V. Schnepf, 2018. "Which schools and pupils respond to educational achievement surveys?: a focus on the English Programme for International Student Assessment sample," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 181(4), pages 1057-1075, October.
    4. Olga Maslovskaya & Peter Lugtig, 2022. "Representativeness in six waves of CROss‐National Online Survey (CRONOS) panel," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(3), pages 851-871, July.
    5. Barry Schouten & Fannie Cobben & Peter Lundquist & James Wagner, 2016. "Does more balanced survey response imply less non-response bias?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(3), pages 727-748, June.
    6. David Cutler & Kaushik Ghosh & Irina Bondarenko & Kassandra Messer & Trivellore Raghunathan & Susan Stewart & Allison B. Rosen, 2018. "Attributing Medical Spending to Conditions: A Comparison of Methods," NBER Working Papers 25233, National Bureau of Economic Research, Inc.
    7. Fetzer, Thiemo & Yotzov, Ivan, 2023. "(How) Do electoral surprises drive business cycles? Evidence from a new dataset," CAGE Online Working Paper Series 672, Competitive Advantage in the Global Economy (CAGE).
    8. Jakubowski, Maciej & Pokropek, Artur, 2015. "Reading achievement progress across countries," International Journal of Educational Development, Elsevier, vol. 45(C), pages 77-88.
    9. Patrick Gleiser & Joseph W. Sakshaug & Marieke Volkert & Peter Ellguth & Susanne Kohaut & Iris Möller, 2022. "Introducing Web in a mixed‐mode establishment survey: Effects on nonresponse," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(3), pages 891-915, July.
    10. Durrant Gabriele B. & Maslovskaya Olga & Smith Peter W. F., 2017. "Using Prior Wave Information and Paradata: Can They Help to Predict Response Outcomes and Call Sequence Length in a Longitudinal Study?," Journal of Official Statistics, Sciendo, vol. 33(3), pages 801-833, September.
    11. Hisler, Garrett C. & Twenge, Jean M., 2021. "Sleep characteristics of U.S. adults before and during the COVID-19 pandemic," Social Science & Medicine, Elsevier, vol. 276(C).
    12. Laura Zieger & John Jerrim & Jake Anders & Nikki Shure, 2020. "Conditioning: How background variables can influence PISA scores," CEPEO Working Paper Series 20-09, UCL Centre for Education Policy and Equalising Opportunities, revised Apr 2020.
    13. Lisa Meehan & Gail Pacheco & Thomas Schober, 2023. "Basic Reading and Mathematics Skills and the Labour Market Outcomes of Young People: Evidence from PISA and Linked Administrative Data," The Economic Record, The Economic Society of Australia, vol. 99(327), pages 473-491, December.
    14. Felderer Barbara & Kirchner Antje & Kreuter Frauke, 2019. "The Effect of Survey Mode on Data Quality: Disentangling Nonresponse and Measurement Error Bias," Journal of Official Statistics, Sciendo, vol. 35(1), pages 93-115, March.
    15. Steele, Fiona & Durrant, Gabriele B., 2011. "Alternative approaches to multilevel modelling of survey non-contact and refusal," LSE Research Online Documents on Economics 50113, London School of Economics and Political Science, LSE Library.
    16. Schnepf, Sylke V. & Durrant, Gabriele B. & Micklewright, John, 2014. "Which Schools and Pupils Respond to Educational Achievement Surveys? A Focus on the English PISA Sample," IZA Discussion Papers 8411, Institute of Labor Economics (IZA).
    17. Särndal Carl-Erik & Traat Imbi & Lumiste Kaur, 2018. "Interaction Between Data Collection And Estimation Phases In Surveys With Nonresponse," Statistics in Transition New Series, Statistics Poland, vol. 19(2), pages 183-200, June.
    18. Earp Morgan & Toth Daniell & Phipps Polly & Oslund Charlotte, 2018. "Assessing Nonresponse in a Longitudinal Establishment Survey Using Regression Trees," Journal of Official Statistics, Sciendo, vol. 34(2), pages 463-481, June.
    19. Wu Shiya & Schouten Barry & Meijers Ralph & Moerbeek Mirjam, 2022. "Data Collection Expert Prior Elicitation in Survey Design: Two Case Studies," Journal of Official Statistics, Sciendo, vol. 38(2), pages 637-662, June.
    20. Tobias Gummer, 2019. "Assessing Trends and Decomposing Change in Nonresponse Bias: The Case of Bias in Cohort Distributions," Sociological Methods & Research, , vol. 48(1), pages 92-115, February.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:istatr:v:88:y:2020:i:3:p:642-657. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/isiiinl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.