IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v43y2019i5p307-330.html
   My bibliography  Save this article

Rethinking Response Rates: New Evidence of Little Relationship Between Survey Response Rates and Nonresponse Bias

Author

Listed:
  • Richard Hendra
  • Aaron Hill

Abstract

Background: Federally funded evaluation research projects typically strive for an 80% survey response rate, but the increasing difficulty and expense in reaching survey respondents raises the question of whether such a threshold is necessary for reducing bias and increasing the accuracy of survey estimates. Objectives: This analysis focuses on a particular component of survey methodology: the survey response rate and its relationship to nonresponse bias. Following a review of the literature, new analysis of data from a large, multisite random assignment experiment explores the relationship between survey response rates and measured nonresponse bias. Research Design: With detailed survey disposition data, the analysis simulates nonresponse bias at lower response rates. The subjects included 12,000 individuals who were fielded for 16 identical surveys as part of the Employment Retention and Advancement evaluation. Results: The results suggest scant relationship between survey nonresponse bias and response rates. The results also indicate that the pursuit of high response rates lengthens the fielding period, which can create other measurement problems. Conclusions: The costly pursuit of a high response rate may offer little or no reduction of nonresponse bias. Achieving such a high rate of response requires considerable financial resources that might be better applied to methods and techniques shown to have a greater effect on the reduction of nonresponse bias.

Suggested Citation

  • Richard Hendra & Aaron Hill, 2019. "Rethinking Response Rates: New Evidence of Little Relationship Between Survey Response Rates and Nonresponse Bias," Evaluation Review, , vol. 43(5), pages 307-330, October.
    • Handle: RePEc:sae:evarev:v:43:y:2019:i:5:p:307-330
    DOI: 10.1177/0193841X18807719
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X18807719
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X18807719?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. John Deke & Hanley Chiang, 2017. "The WWC Attrition Standard," Evaluation Review, , vol. 41(2), pages 130-154, April.
    2. Daniel Hoechle, 2007. "Robust standard errors for panel regressions with cross-sectional dependence," Stata Journal, StataCorp LP, vol. 7(3), pages 281-312, September.
    3. David J. Hand & Heikki Mannila & Padhraic Smyth, 2001. "Principles of Data Mining," MIT Press Books, The MIT Press, edition 1, volume 1, number 026208290x, April.
    4. Abhijit Banerjee & Esther Duflo & Rachel Glennerster & Cynthia Kinnan, 2015. "The Miracle of Microfinance? Evidence from a Randomized Evaluation," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 22-53, January.
    5. Newey, Whitney & West, Kenneth, 2014. "A simple, positive semi-definite, heteroscedasticity and autocorrelation consistent covariance matrix," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 33(1), pages 125-132.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rana Orhan & John Middleton & Thomas Krafft & Katarzyna Czabanowska, 2021. "Climate Action at Public Health Schools in the European Region," IJERPH, MDPI, vol. 18(4), pages 1-16, February.
    2. Gross, Deborah & Bettencourt, Amie F. & Holmes Finch, W. & Plesko, Corinne & Paulson, Rachael & Singleton, Demetria L., 2022. "Developing an equitable measure of parent engagement in early childhood education for urban schools," Children and Youth Services Review, Elsevier, vol. 141(C).
    3. Toma, Mattie & Bell, Elizabeth, 2024. "Understanding and increasing policymakers’ sensitivity to program impact," Journal of Public Economics, Elsevier, vol. 234(C).
    4. Burt S. Barnow & David H. Greenberg, 2019. "Special Issue Editors’ Essay," Evaluation Review, , vol. 43(5), pages 231-265, October.
    5. Sian Harrison & Fiona Alderdice & Maria A Quigley, 2020. "External validity of prevalence estimates from the national maternity surveys in England: The impact of response rate," PLOS ONE, Public Library of Science, vol. 15(11), pages 1-18, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kai Daniel Schmid & Michael Schmidt, 2012. "EMU and the Renaissance of Sovereign Credit Risk Perception," IAW Discussion Papers 87, Institut für Angewandte Wirtschaftsforschung (IAW).
    2. Singh, Amarendra Pratap & Narayanan, Krishnan, 2016. "How can weather affect crop area diversity? Panel data evidence from Andhra Pradesh, a rice growing state of India," Studies in Agricultural Economics, Research Institute for Agricultural Economics, vol. 118(2), pages 1-10, August.
    3. Petreski, Aleksandar & Schäfer, Dorothea & Stephan, Andreas, 2023. "The reputation effect of green bond issuance and its impact on the cost of capital," Working Paper Series in Economics and Institutions of Innovation 493, Royal Institute of Technology, CESIS - Centre of Excellence for Science and Innovation Studies.
    4. Tsani, Stella, 2013. "Natural resources, governance and institutional quality: The role of resource funds," Resources Policy, Elsevier, vol. 38(2), pages 181-195.
    5. Hinsche, Isabelle Cathérine, 2021. "A greenium for the next generation EU green bonds: Analysis of a potential green bond premium and its drivers," CFS Working Paper Series 663, Center for Financial Studies (CFS).
    6. Atif Khan Jadoon & Sania Akhtar & Ambreen Sarwar & Syeda Azra Batool & Sarvjeet Kaur Chatrath & Saima Liaqat, 2021. "Is Economic Growth And Industrial Growth The Reason For Environmental Degradation In Saarc Countries," International Journal of Energy Economics and Policy, Econjournals, vol. 11(6), pages 418-426.
    7. Ouoba, Youmanli, 2016. "Natural resources: Funds and economic performance of resource-rich countries," Resources Policy, Elsevier, vol. 50(C), pages 108-116.
    8. Hoechle, Daniel & Schmid, Markus & Zimmermann, Heinz, 2012. "Decomposing Performance," Working Papers on Finance 1216, University of St. Gallen, School of Finance, revised Nov 2015.
    9. Adeel Ahmad DAR & Taj MUHAMMAD & M. Wasif SIDDIQI, 2020. "Bureaucratic Quality and FDI Inflows Nexus: A South Asian Perspective," Journal for Economic Forecasting, Institute for Economic Forecasting, vol. 0(3), pages 149-168, September.
    10. Phetkeo Poumanyvong & Shinji Kaneko & Shobhakar Dhakal, 2012. "Impacts of urbanization on national residential energy use and CO2 emissions: Evidence from low-, middle- and high-income countries," IDEC DP2 Series 2-5, Hiroshima University, Graduate School for International Development and Cooperation (IDEC).
    11. Jan Kluge & Robert Lehmann, 2013. "Marshall or Jacobs? New insights from an interaction model," Review of Regional Research: Jahrbuch für Regionalwissenschaft, Springer;Gesellschaft für Regionalforschung (GfR), vol. 33(2), pages 107-133, October.
    12. Zhang, Chuanguo & Nian, Jiang, 2013. "Panel estimation for transport sector CO2 emissions and its affecting factors: A regional analysis in China," Energy Policy, Elsevier, vol. 63(C), pages 918-926.
    13. Wei Li & Tao Zhao & Yanan Wang & Fang Guo, 2017. "Investigating the learning effects of technological advancement on CO2 emissions: a regional analysis in China," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 88(2), pages 1211-1227, September.
    14. Vogelsang, Timothy J., 2012. "Heteroskedasticity, autocorrelation, and spatial correlation robust inference in linear panel models with fixed-effects," Journal of Econometrics, Elsevier, vol. 166(2), pages 303-319.
    15. Auer, Benjamin R. & Rottmann, Horst, 2019. "Have capital market anomalies worldwide attenuated in the recent era of high liquidity and trading activity?," Journal of Economics and Business, Elsevier, vol. 103(C), pages 61-79.
    16. de Soysa Indra, 2020. "Does Foreign Direct Investment Encourage State Militarization and Reduce Societal Security? An Empirical Test, 1980–2017," Peace Economics, Peace Science, and Public Policy, De Gruyter, vol. 26(1), pages 1-15, February.
    17. Zhang, Chuanguo & Zhou, Xiangxue, 2016. "Does foreign direct investment lead to lower CO2 emissions? Evidence from a regional analysis in China," Renewable and Sustainable Energy Reviews, Elsevier, vol. 58(C), pages 943-951.
    18. Aldashev, Alisher & Dietz, Barbara, 2011. "Determinants of internal migration in Kazakhstan," MPRA Paper 34922, University Library of Munich, Germany.
    19. Fratianni, Michele & Marchionne, Francesco, 2013. "The fading stock market response to announcements of bank bailouts," Journal of Financial Stability, Elsevier, vol. 9(1), pages 69-89.
    20. Elif Guneren Genc & Ozlem Deniz Basar, 2017. "The Impact of OECD Countries¡¯ Macroeconomic Factors on Turkey¡¯s Foreign Trade," Research in World Economy, Research in World Economy, Sciedu Press, vol. 8(1), pages 24-36, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:43:y:2019:i:5:p:307-330. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.