Are Factorial Survey Experiments Prone to Survey Mode Effects?

Drawing on data from a nation-wide survey on the fairness of earnings, this piece of work examines the susceptibility of factorial survey experiments to mode effects. It uses multilevel models to compare the results from a completely self-administered questionnaire to those of an experimental condition in which interviewers were present. From a theoretical point of view, interviewers might affect measurement error in different directions. On the one hand, they can provide explanations and assistance if respondents feel unsure about answering the rather complex vignette questions, or even motivate them to put more effort into the response process. As a result, respondents should be less inclined to use satisficing as a response strategy. On the other hand, respondents might feel pressured to provide answers quickly and in line with what is socially desirable. We compare the two survey modes with regard to a set of indicators for data quality: nonresponse, common heuristics (ticking the middle and straight-lining), response consistency throughout the vignette set and social desirability bias. The latter is examined by analysing substantive results from the vignette module, which allows for quantifying the gender wage gap that respondents perceive as just. The study discusses to what extent interviewers can relieve the cognitive burden associated with factorial surveys and positively affect data quality. Do the favourable effects of interviewers outweigh the negative ones? The study can help practitioners to decide if the gains in data quality are great enough to justify investment in a more expensive personal survey mode.