IDEAS home Printed from https://ideas.repec.org/p/fip/fedgfe/2015-08.html
   My bibliography  Save this paper

Mode effects in mixed-mode economic surveys: Insights from a randomized experiment

Author

Listed:

Abstract

Web-based surveys have become increasingly common in economic, marketing, and other social science research. However, questions exist about the comparability of data gathered using a web interview and data gathered using more traditional survey modes, particularly for surveys on household economic behavior. Differences between data from different survey modes may arise through two different mechanisms: sample selectivity due to (lack of) web access and mode effects. This study leverages the randomized experimental design of the mixed-mode Cognitive Economics Study to examine mode effects separately from sample selectivity issues. In particular, we examine differences in survey response rates, item nonresponse, and data quality due to mode effects. Our results indicate that, in contrast to mail mode, web mode surveys (1) attain higher response rates among web users, (2) display lower item nonresponse, and (3) elicit more precise values for financial measures. We conclude that, for web-using populations, web mode surveys appear to result in more usable data than mail mode surveys, and these data appear to be of high quality. However, we also find no systematic mode differences in the categorical distributions of responses to items, providing no evidence that pooling data from the two modes is inadvisable.

Suggested Citation

  • Joanne W. Hsu & Brooke H. McFall, 2015. "Mode effects in mixed-mode economic surveys: Insights from a randomized experiment," Finance and Economics Discussion Series 2015-8, Board of Governors of the Federal Reserve System (U.S.).
    • Handle: RePEc:fip:fedgfe:2015-08
    DOI: 10.17016/FEDS.2015.008
    as

    Download full text from publisher

    File URL: http://www.federalreserve.gov/econresdata/feds/2015/files/2015008pap.pdf
    File Function: Full text
    Download Restriction: no
    File URL: http://dx.doi.org/10.17016/FEDS.2015.008
    File Function: http://dx.doi.org/10.17016/FEDS.2015.008
    Download Restriction: no
    File URL: https://libkey.io/10.17016/FEDS.2015.008?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. de Leeuw, E.D. & Hox, J.J.C.M. & Scherpenzeel, A.C., 2011. "Mode effect or question wording? Measurement error in mixed mode surveys," Other publications TiSEM 4218c762-6d80-4dfc-97ee-8, Tilburg University, School of Economics and Management.
    2. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    3. Couper, Mick P. & Kapteyn, Arie & Schonlau, Matthias & Winter, Joachim, 2007. "Noncoverage and nonresponse in an Internet survey," Munich Reprints in Economics 20093, University of Munich, Department of Economics.
    4. repec:cai:poeine:pope_1002_0285 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Crossley, Thomas F. & Fisher, Paul & Low, Hamish, 2021. "The heterogeneous and regressive consequences of COVID-19: Evidence from high quality panel data," Journal of Public Economics, Elsevier, vol. 193(C).
    2. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2018. "Can Online Surveys Represent the Entire Population?," IZA Discussion Papers 11799, Institute of Labor Economics (IZA).
    3. Arthur van Soest & Arie Kapteyn, 2009. "Mode and Context Effects in Measuring Household Assets," Working Papers 200949, Geary Institute, University College Dublin.
    4. Giovanni Franco, 2024. "The return of non-probability sample: the electoral polls at the time of internet and social media," Quality & Quantity: International Journal of Methodology, Springer, vol. 58(4), pages 3811-3830, August.
    5. Grewenig, Elisabeth & Lergetporer, Philipp & Simon, Lisa & Werner, Katharina & Woessmann, Ludger, 2023. "Can internet surveys represent the entire population? A practitioners’ analysis," European Journal of Political Economy, Elsevier, vol. 78(C).
    6. repec:aia:aiaswp:wp76 is not listed on IDEAS
    7. Arthur van Soest & Arie Kapteyn, 2009. "Mode and Context Effects in Measuring Household Assets," Working Papers 200949, Geary Institute, University College Dublin.
    8. Guzi, Martin & de Pedraza, Pablo, 2013. "A Web Survey Analysis of the Subjective Well-being of Spanish Workers," IZA Discussion Papers 7618, Institute of Labor Economics (IZA).
    9. Stéphane Legleye & Géraldine Charrance & Nicolas Razafindratsima & Nathalie Bajos & Aline Bohet & Caroline Moreau, 2018. "The Use of a Nonprobability Internet Panel to Monitor Sexual and Reproductive Health in the General Population," Sociological Methods & Research, , vol. 47(2), pages 314-348, March.
    10. Heng Chen & Geoffrey Dunbar & Q. Rallye Shen, 2020. "The Mode is the Message: Using Predata as Exclusion Restrictions to Evaluate Survey Design," Advances in Econometrics, in: Essays in Honor of Cheng Hsiao, volume 41, pages 341-357, Emerald Group Publishing Limited.
    11. Hildebrand Sean, 2015. "Coerced Confusion? Local Emergency Policy Implementation After September 11," Journal of Homeland Security and Emergency Management, De Gruyter, vol. 12(2), pages 273-298, June.
    12. Knox, Melissa A. & Oddo, Vanessa M. & Walkinshaw, Lina Pinero & Jones-Smith, Jessica, 2020. "Is the public sweet on sugary beverages? Social desirability bias and sweetened beverage taxes," Economics & Human Biology, Elsevier, vol. 38(C).
    13. Mohorko Anja & Leeuw Edith de & Hox Joop, 2013. "Internet Coverage and Coverage Bias in Europe: Developments Across Countries and Over Time," Journal of Official Statistics, Sciendo, vol. 29(4), pages 609-622, December.
    14. Richard, James E. & Purnell, Fruen, 2017. "Rethinking Catalogue and Online B2B Buyer Channel Preferences in the Education Supplies Market," Journal of Interactive Marketing, Elsevier, vol. 37(C), pages 1-15.
    15. Magdalena Smyk & Joanna Tyrowicz & Lucas van der Velde, 2021. "A Cautionary Note on the Reliability of the Online Survey Data: The Case of Wage Indicator," Sociological Methods & Research, , vol. 50(1), pages 429-464, February.
    16. Hung, Kam & Law, Rob, 2011. "An overview of Internet-based surveys in hospitality and tourism journals," Tourism Management, Elsevier, vol. 32(4), pages 717-724.
    17. Fiore, M. & Gaviglio, A. & Demartini, E. & La Sala, P., 2018. "Sugarcoating Food Technologies and consumers’ acceptance of long-life fish," 2018 Conference, July 28-August 2, 2018, Vancouver, British Columbia 275971, International Association of Agricultural Economists.
    18. Richard Valliant & Jill A. Dever, 2011. "Estimating Propensity Adjustments for Volunteer Web Surveys," Sociological Methods & Research, , vol. 40(1), pages 105-137, February.
    19. Joel Martin & Michael Toczko & Emily Locke & Ryan McCarthy & Italia Milani & Nathalie Barrios & Samer Koutoubi & Jatin Ambegaonkar & Niyati Dhokai & Ali Boolani, 2022. "Influence of Grit on Physical Activity, Sitting Time and Dietary Behaviors: A Multi-Study Analysis," Sustainability, MDPI, vol. 15(1), pages 1-14, December.
    20. Dewaele Alexis & Caen Maya & Buysse Ann, 2014. "Comparing Survey and Sampling Methods for Reaching Sexual Minority Individuals in Flanders," Journal of Official Statistics, Sciendo, vol. 30(2), pages 251-251, June.
    21. Almut Schumann & Detlev Lück, 2023. "Better to ask online when it concerns intimate relationships? Survey mode differences in the assessment of relationship quality," Demographic Research, Max Planck Institute for Demographic Research, Rostock, Germany, vol. 48(22), pages 609-640.

    More about this item

    Keywords

    Data quality; household surveys; mode effects; response rates;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:fedgfe:2015-08. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Ryan Wolfslayer ; Keisha Fournillier (email available below). General contact details of provider: https://edirc.repec.org/data/frbgvus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.