Are we doing more harm than good? Hypothetical bias reduction techniques in potentially consequential survey settings

Researchers deploying stated preference surveys to elicit monetary valuations for public goods commonly use techniques devised to reduce bias in hypothetical choice settings. This practice is conceptually at odds with accumulated evidence that most survey respondents instead perceive that their decisions have economic consequences (i.e., affect their future welfare). We examine three bias reduction procedures in both hypothetical choice and incentive compatible, real payment settings: cheap talk, solemn oath, and certainty adjustment. While we find that the oath reduces willingness to pay (WTP) in a hypothetical setting, the oath instead increases WTP by over 30% in a consequential setting. Cheap talk does not alter mean WTP in a consequential setting but leads to a stark difference in WTP across sexes. Applying the common rules for ex post adjustment of choices based on stated response certainty leads to significant and large decreases in WTP estimates for both hypothetical and consequential cases. Our results suggest that survey researchers should make use of screening questions to better target hypothetical bias reduction techniques to only those prone to bias.