IDEAS home Printed from https://ideas.repec.org/p/fip/fedgfe/2015-08.html
   My bibliography  Save this paper

Mode effects in mixed-mode economic surveys: Insights from a randomized experiment

Author

Listed:

Abstract

Web-based surveys have become increasingly common in economic, marketing, and other social science research. However, questions exist about the comparability of data gathered using a web interview and data gathered using more traditional survey modes, particularly for surveys on household economic behavior. Differences between data from different survey modes may arise through two different mechanisms: sample selectivity due to (lack of) web access and mode effects. This study leverages the randomized experimental design of the mixed-mode Cognitive Economics Study to examine mode effects separately from sample selectivity issues. In particular, we examine differences in survey response rates, item nonresponse, and data quality due to mode effects. Our results indicate that, in contrast to mail mode, web mode surveys (1) attain higher response rates among web users, (2) display lower item nonresponse, and (3) elicit more precise values for financial measures. We conclude that, for web-using populations, web mode surveys appear to result in more usable data than mail mode surveys, and these data appear to be of high quality. However, we also find no systematic mode differences in the categorical distributions of responses to items, providing no evidence that pooling data from the two modes is inadvisable.

Suggested Citation

  • Joanne W. Hsu & Brooke H. McFall, 2015. "Mode effects in mixed-mode economic surveys: Insights from a randomized experiment," Finance and Economics Discussion Series 2015-8, Board of Governors of the Federal Reserve System (U.S.).
    • Handle: RePEc:fip:fedgfe:2015-08
    DOI: 10.17016/FEDS.2015.008
    as

    Download full text from publisher

    File URL: http://www.federalreserve.gov/econresdata/feds/2015/files/2015008pap.pdf
    File Function: Full text
    Download Restriction: no
    File URL: http://dx.doi.org/10.17016/FEDS.2015.008
    File Function: http://dx.doi.org/10.17016/FEDS.2015.008
    Download Restriction: no
    File URL: https://libkey.io/10.17016/FEDS.2015.008?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. de Leeuw, E.D. & Hox, J.J.C.M. & Scherpenzeel, A.C., 2011. "Mode effect or question wording? Measurement error in mixed mode surveys," Other publications TiSEM 4218c762-6d80-4dfc-97ee-8, Tilburg University, School of Economics and Management.
    2. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    3. Couper, Mick P. & Kapteyn, Arie & Schonlau, Matthias & Winter, Joachim, 2007. "Noncoverage and nonresponse in an Internet survey," Munich Reprints in Economics 20093, University of Munich, Department of Economics.
    4. repec:cai:poeine:pope_1002_0285 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Crossley, Thomas F. & Fisher, Paul & Low, Hamish, 2021. "The heterogeneous and regressive consequences of COVID-19: Evidence from high quality panel data," Journal of Public Economics, Elsevier, vol. 193(C).
    2. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," IZA Discussion Papers 11799, Institute of Labor Economics (IZA).
    3. Arthur van Soest & Arie Kapteyn, 2009. "Mode and Context Effects in Measuring Household Assets," Working Papers 200949, Geary Institute, University College Dublin.
    4. repec:aia:aiaswp:wp76 is not listed on IDEAS
    5. Giovanni Franco, 2024. "The return of non-probability sample: the electoral polls at the time of internet and social media," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(4), pages 3811-3830, August.
    6. Arthur van Soest & Arie Kapteyn, 2009. "Mode and Context Effects in Measuring Household Assets," Working Papers 200949, Geary Institute, University College Dublin.
    7. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2023. "Can internet surveys represent the entire population? A practitioners’ analysis," European Journal of Political Economy, Elsevier, vol. 78(C).
    8. Guzi, Martin & de Pedraza, Pablo, 2013. "A Web Survey Analysis of the Subjective Well-being of Spanish Workers," IZA Discussion Papers 7618, Institute of Labor Economics (IZA).
    9. Joachim Winter & Amelie Wuppermann, 2014. "Do They Know What Is At Risk? Health Risk Perception Among The Obese," Health Economics, John Wiley & Sons, Ltd., vol. 23(5), pages 564-585, May.
    10. Axsen, Jonn & Mountain, Dean C. & Jaccard, Mark, 2009. "Combining stated and revealed choice research to simulate the neighbor effect: The case of hybrid-electric vehicles," Resource and Energy Economics, Elsevier, vol. 31(3), pages 221-238, August.
    11. Maciej Berȩsewicz & Dagmara Nikulin, 2021. "Estimation of the size of informal employment based on administrative records with non‐ignorable selection mechanism," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 70(3), pages 667-690, June.
    12. Stéphane Legleye & Géraldine Charrance & Nicolas Razafindratsima & Nathalie Bajos & Aline Bohet & Caroline Moreau, 2018. "The Use of a Nonprobability Internet Panel to Monitor Sexual and Reproductive Health in the General Population," Sociological Methods & Research, , vol. 47(2), pages 314-348, March.
    13. Heng Chen & Geoffrey Dunbar & Q. Rallye Shen, 2020. "The Mode is the Message: Using Predata as Exclusion Restrictions to Evaluate Survey Design," Advances in Econometrics, in: Essays in Honor of Cheng Hsiao, volume 41, pages 341-357, Emerald Group Publishing Limited.
    14. Hildebrand Sean, 2015. "Coerced Confusion? Local Emergency Policy Implementation After September 11," Journal of Homeland Security and Emergency Management, De Gruyter, vol. 12(2), pages 273-298, June.
    15. Knox, Melissa A. & Oddo, Vanessa M. & Walkinshaw, Lina Pinero & Jones-Smith, Jessica, 2020. "Is the public sweet on sugary beverages? Social desirability bias and sweetened beverage taxes," Economics & Human Biology, Elsevier, vol. 38(C).
    16. Mohorko Anja & Leeuw Edith de & Hox Joop, 2013. "Internet Coverage and Coverage Bias in Europe: Developments Across Countries and Over Time," Journal of Official Statistics, Sciendo, vol. 29(4), pages 609-622, December.
    17. Mick P. Couper & Eleanor Singer & Carrie A. Levin & Floyd J. Fowler Jr. & Angela Fagerlin & Brian J. Zikmund-Fisher, 2010. "Use of the Internet and Ratings of Information Sources for Medical Decisions: Results from the DECISIONS Survey," Medical Decision Making, , vol. 30(5_suppl), pages 106-114, September.
    18. Richard, James E. & Purnell, Fruen, 2017. "Rethinking Catalogue and Online B2B Buyer Channel Preferences in the Education Supplies Market," Journal of Interactive Marketing, Elsevier, vol. 37(C), pages 1-15.
    19. Lang, Megan & Ligon, Ethan, 2022. "SMS Surveys of Selected Expenditures," Department of Agricultural & Resource Economics, UC Berkeley, Working Paper Series qt7p7336h5, Department of Agricultural & Resource Economics, UC Berkeley.
    20. Magdalena Smyk & Joanna Tyrowicz & Lucas van der Velde, 2021. "A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator," Sociological Methods & Research, , vol. 50(1), pages 429-464, February.
    21. Bruine de Bruin, Wändi & van der Klaauw, Wilbert & van Rooij, Maarten & Teppa, Federica & de Vos, Klaas, 2017. "Measuring expectations of inflation: Effects of survey mode, wording, and opportunities to revise," Journal of Economic Psychology, Elsevier, vol. 59(C), pages 45-58.

    More about this item

    Keywords

    Data quality; household surveys; mode effects; response rates;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedgfe:2015-08. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Ryan Wolfslayer ; Keisha Fournillier (email available below). General contact details of provider: https://edirc.repec.org/data/frbgvus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.