IDEAS home Printed from https://ideas.repec.org/a/vrs/offsta/v35y2019i1p93-115n5.html
   My bibliography  Save this article

The Effect of Survey Mode on Data Quality: Disentangling Nonresponse and Measurement Error Bias

Author

Listed:
  • Felderer Barbara

    (University of Mannheim, Mannheim, 68131, Germany.)

  • Kirchner Antje

    (RTI International, 3040 E. Cornwallis Road, RTP, NC 27709, U.S.A.)

  • Kreuter Frauke

    (University of Maryland, College Park, Maryland20742, U.S.A.)

Abstract

More and more surveys are conducted online. While web surveys are generally cheaper and tend to have lower measurement error in comparison to other survey modes, especially for sensitive questions, potential advantages might be offset by larger nonresponse bias. This article compares the data quality in a web survey administration to another common mode of survey administration, the telephone.The unique feature of this study is the availability of administrative records for all sampled individuals in combination with a random assignment of survey mode. This specific design allows us to investigate and compare potential bias in survey statistics due to 1) nonresponse error, 2) measurement error, and 3) combined bias of these two error sources and hence, an overall assessment of data quality for two common modes of survey administration, telephone and web.Our results show that overall mean estimates on the web are more biased compared to the telephone mode. Nonresponse and measurement bias tend to reinforce each other in both modes, with nonresponse bias being somewhat more pronounced in the web mode. While measurement error bias tends to be smaller in the web survey implementation, interestingly, our results also show that the web does not consistently outperform the telephone mode for sensitive questions.

Suggested Citation

  • Felderer Barbara & Kirchner Antje & Kreuter Frauke, 2019. "The Effect of Survey Mode on Data Quality: Disentangling Nonresponse and Measurement Error Bias," Journal of Official Statistics, Sciendo, vol. 35(1), pages 93-115, March.
    • Handle: RePEc:vrs:offsta:v:35:y:2019:i:1:p:93-115:n:5
    DOI: 10.2478/jos-2019-0005
    as

    Download full text from publisher

    File URL: https://doi.org/10.2478/jos-2019-0005
    Download Restriction: no
    File URL: https://libkey.io/10.2478/jos-2019-0005?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Katharine G. Abraham & Aaron Maitland & Suzanne M. Bianchi, 2006. "Non-response in the American Time Use Survey: Who Is Missing from the Data and How Much Does It Matter?," NBER Technical Working Papers 0328, National Bureau of Economic Research, Inc.
    2. Bound, John & Krueger, Alan B, 1991. "The Extent of Measurement Error in Longitudinal Earnings Data: Do Two Wrongs Make a Right?," Journal of Labor Economics, University of Chicago Press, vol. 9(1), pages 1-24, January.
    3. F. Kreuter & K. Olson & J. Wagner & T. Yan & T. M. Ezzati‐Rice & C. Casas‐Cordero & M. Lemay & A. Peytchev & R. M. Groves & T. E. Raghunathan, 2010. "Using proxy measures and other correlates of survey outcomes to adjust for non‐response: examples from multiple surveys," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 173(2), pages 389-407, April.
    4. Jelke Bethlehem, 2010. "Selection Bias in Web Surveys," International Statistical Review, International Statistical Institute, vol. 78(2), pages 161-188, August.
    5. Kristen Olson, 2013. "Do non-response follow-ups improve or reduce data quality?: a review of the existing literature," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 176(1), pages 129-145, January.
    6. Lynn, Peter & Jäckle, Annette & Hope, Steven & C. Campanelli, Pamela & Nicolaas, Gerry, 2014. "The role of the interviewer in producing mode effects: results from a mixed modes experiment comparing face-to-face, telephone and web administration," ISER Working Paper Series 2014-20, Institute for Social and Economic Research.
    7. Duncan, Greg J & Hill, Daniel H, 1985. "An Investigation of the Extent and Consequences of Measurement Error in Labor-Economic Survey Data," Journal of Labor Economics, University of Chicago Press, vol. 3(4), pages 508-532, October.
    8. Barry Schouten & Fannie Cobben & Peter Lundquist & James Wagner, 2016. "Does more balanced survey response imply less non-response bias?," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(3), pages 727-748, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Laisney, François & Pohlmeier, Winfried & Staat, Matthias, 1991. "Estimation of labour supply functions using panel data: a survey," ZEW Discussion Papers 91-05, ZEW - Leibniz Centre for European Economic Research.
    2. John Abowd & Martha Stinson, 2011. "Estimating Measurement Error in SIPP Annual Job Earnings: A Comparison of Census Bureau Survey and SSA Administrative Data," Working Papers 11-20, Center for Economic Studies, U.S. Census Bureau.
    3. Peter Gottschalk & Minh Huynh, 2010. "Are Earnings Inequality and Mobility Overstated? The Impact of Nonclassical Measurement Error," The Review of Economics and Statistics, MIT Press, vol. 92(2), pages 302-315, May.
    4. de Nicola, Francesca & Giné, Xavier, 2014. "How accurate are recall data? Evidence from coastal India," Journal of Development Economics, Elsevier, vol. 106(C), pages 52-65.
    5. Roberts Caroline & Vandenplas Caroline & Herzing Jessica M.E., 2020. "A Validation of R-Indicators as a Measure of the Risk of Bias using Data from a Nonresponse Follow-Up Survey," Journal of Official Statistics, Sciendo, vol. 36(3), pages 675-701, September.
    6. Arnaud Lefranc, 2003. "On the sensitivity of returns to seniority to the measurement of earnings," Post-Print hal-01651776, HAL.
    7. Lisa M. Dragoset & Gary S. Fields, 2006. "U.S. Earnings Mobility: Comparing Survey-Based and Administrative-Based Estimates," Working Papers 55, ECINEQ, Society for the Study of Economic Inequality.
    8. Paulus, Alari, 2015. "Tax evasion and measurement error: An econometric analysis of survey data linked with tax records," ISER Working Paper Series 2015-10, Institute for Social and Economic Research.
    9. Brownstone, David & Valletta, Robert G, 1996. "Modeling Earnings Measurement Error: A Multiple Imputation Approach," The Review of Economics and Statistics, MIT Press, vol. 78(4), pages 705-717, November.
    10. Michele Lalla & Maddalena Cavicchioli, 2020. "Nonresponse and measurement errors in income: matching individual survey data with administrative tax data," Department of Economics 0170, University of Modena and Reggio E., Faculty of Economics "Marco Biagi".
    11. Cyrille Hagneré & Arnaud Lefranc, 2006. "Étendue et conséquences des erreurs de mesure dans les données individuelles d'enquête : une évaluation à partir des données appariées des enquêtes emploi et revenus fiscaux," Economie & Prévision, La Documentation Française, vol. 0(3), pages 131-154.
    12. Jenkins, Stephen P. & Rios-Avila, Fernando, 2021. "Reconciling Reports: Modelling Employment Earnings and Measurement Errors Using Linked Survey and Administrative Data," IZA Discussion Papers 14405, Institute of Labor Economics (IZA).
    13. Akee, Randall K. Q., 2007. "Errors in Self-Reported Earnings: The Role of Previous Earnings Volatility," IZA Discussion Papers 3263, Institute of Labor Economics (IZA).
    14. Denyse L. Dagenais & Marcel Dagenais, 1995. "Higher Moment Estimators for Linear Regression Models With Errors in the Variables," CIRANO Working Papers 95s-13, CIRANO.
    15. Pischke, Jorn-Steffen, 1995. "Individual Income, Incomplete Information, and Aggregate Consumption," Econometrica, Econometric Society, vol. 63(4), pages 805-840, July.
    16. Julian Cristia & Jonathan A. Schwabish, 2007. "Measurement Error in the SIPP: Evidence from Matched Administrative Records: Working Paper 2007-03," Working Papers 18322, Congressional Budget Office.
    17. James M. Poterba & Lawrence H. Summers, 1993. "Unemployment Benefits, Labor Market Transitions, and Spurious Flows: A Multinational Logit Model with Errors in Classification," NBER Working Papers 4434, National Bureau of Economic Research, Inc.
    18. Frauke Kreuter & Gerrit Müller & Mark Trappmann, 2014. "A Note on Mechanisms Leading to Lower Data Quality of Late or Reluctant Respondents," Sociological Methods & Research, , vol. 43(3), pages 452-464, August.
    19. Joshua D. Angrist, 1990. "Does Labor Supply Explain Fluctuations in Average Hours Worked?," NBER Working Papers 3312, National Bureau of Economic Research, Inc.
    20. Jungmin Lee & Sokbae Lee, 2012. "Does it Matter WHO Responded to the Survey? Trends in the U.S. Gender Earnings Gap Revisited," ILR Review, Cornell University, ILR School, vol. 65(1), pages 148-160, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:vrs:offsta:v:35:y:2019:i:1:p:93-115:n:5. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.sciendo.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.