IDEAS home Printed from https://ideas.repec.org/a/sae/medema/v39y2019i7p827-841.html
   My bibliography  Save this article

Mode and Frame Matter: Assessing the Impact of Survey Mode and Sample Frame in Choice Experiments

Author

Listed:
  • Verity Watson

    (Health Economics Research Unit, University of Aberdeen, Foresterhill, Aberdeen, Scotland)

  • Terry Porteous

    (Health Economics Research Unit, University of Aberdeen, Foresterhill, Aberdeen, Scotland)

  • Tim Bolt

    (Faculty of Economics, Saitama University, Sakura-ku, Saitama, Japan)

  • Mandy Ryan

    (Health Economics Research Unit, University of Aberdeen, Foresterhill, Aberdeen, Scotland)

Abstract

Background. Choice experiments (CE) are applied in health economics to elicit public preferences and willingness to pay (WTP). CEs are frequently administered as Internet-based surveys. Internet surveys have recognized advantages, but concerns exist about the representativeness of Internet samples, data quality, and the impact on elicited values. Aim. We conducted the first study in health comparing an Internet-based CE survey with the more traditional general population mail survey. We also compared the Internet-based and mail CE surveys with computer-assisted personal interviews (CAPIs), which are commonly used to elicit health state valuations. Methods. Two separate samples were drawn from 2 United Kingdom (UK) volunteer Internet panels (IPs), CAPIs were undertaken with respondents sampled from UK Census Output Areas, and mail surveys were sent to UK households drawn from the postcode address file (PAF). Each mode received more than 1000 respondents. We compared modes and frames using objective measures (response rate, sample representativeness of the UK population, elicited values, theoretical validity, and cost per response) and subjective/self-reported measures (time taken to complete the study, perceived study consequentiality, and stated attribute nonattendance). This study intentionally confounded the survey modes and sample frame by choosing sample frames that are typically used by researchers for each mode. Results. Estimated WTP differs across mode-frame pairs. On most measures, CAPIs dominated. They are more expensive, however. On all measures, except response rates, Internet surveys dominated the mail survey. They were also cheaper. Conclusion. Researchers using IPs should pay attention to response rates and be aware that the quality of IPs differs. Given the importance of perceived consequentiality and attribute attendance in CEs, future research should address their impact across modes and frames.

Suggested Citation

  • Verity Watson & Terry Porteous & Tim Bolt & Mandy Ryan, 2019. "Mode and Frame Matter: Assessing the Impact of Survey Mode and Sample Frame in Choice Experiments," Medical Decision Making, , vol. 39(7), pages 827-841, October.
    • Handle: RePEc:sae:medema:v:39:y:2019:i:7:p:827-841
    DOI: 10.1177/0272989X19871035
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0272989X19871035
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0272989X19871035?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Donna Rowen & John Brazier & Anju Keetharuth & Aki Tsuchiya & Clara Mukuria, 2016. "Comparison of Modes of Administration and Alternative Formats for Eliciting Societal Preferences for Burden of Illness," Applied Health Economics and Health Policy, Springer, vol. 14(1), pages 89-104, February.
    2. Arne Risa Hole, 2007. "A comparison of approaches to estimating confidence intervals for willingness to pay measures," Health Economics, John Wiley & Sons, Ltd., vol. 16(8), pages 827-840, August.
    3. Richard Carson & Theodore Groves, 2007. "Incentive and informational properties of preference questions," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 37(1), pages 181-210, May.
    4. Verity Watson & Frauke Becker & Esther de Bekker‐Grob, 2017. "Discrete Choice Experiment Response Rates: A Meta‐analysis," Health Economics, John Wiley & Sons, Ltd., vol. 26(6), pages 810-817, June.
    5. Madeleine T. King & Jane Hall & Emily Lancsar & Denzil Fiebig & Ishrat Hossain & Jordan Louviere & Helen K. Reddel & Christine R. Jenkins, 2007. "Patient preferences for managing asthma: results from a discrete choice experiment," Health Economics, John Wiley & Sons, Ltd., vol. 16(7), pages 703-717, July.
    6. J. D. Snowball & K. G. Willis, 2011. "Interview versus self-completion questionnaires in discrete choice experiments," Applied Economics Letters, Taylor & Francis Journals, vol. 18(16), pages 1521-1525.
    7. Vossler, Christian A. & Watson, Sharon B., 2013. "Understanding the consequences of consequentiality: Testing the validity of stated preferences in the field," Journal of Economic Behavior & Organization, Elsevier, vol. 86(C), pages 137-147.
    8. Michael Keane & Nada Wasi, 2013. "Comparing Alternative Models Of Heterogeneity In Consumer Choice Behavior," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(6), pages 1018-1045, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ruixuan Jiang & Eleanor Pullenayegum & James W. Shaw & Axel Mühlbacher & Todd A. Lee & Surrey Walton & Thomas Kohlmann & Richard Norman & A. Simon Pickard, 2023. "Comparison of Preferences and Data Quality between Discrete Choice Experiments Conducted in Online and Face-to-Face Respondents," Medical Decision Making, , vol. 43(6), pages 667-679, August.
    2. Mandy Ryan & Emmanouil Mentzakis & Catriona Matheson & Christine Bond, 2020. "Survey modes comparison in contingent valuation: Internet panels and mail surveys," Health Economics, John Wiley & Sons, Ltd., vol. 29(2), pages 234-242, February.
    3. Zawojska, Ewa & Gastineau, Pascal & Mahieu, Pierre-Alexandre & Cheze, Benoit & Paris, Anthony, 2021. "Measuring policy consequentiality perceptions in stated preference surveys," 2021 Annual Meeting, August 1-3, Austin, Texas 313977, Agricultural and Applied Economics Association.
    4. MuLong Mao & Paul James, 2020. "Evaluating Customer Satisfaction Using the ACSI Model in a Themed Restaurant in Bangkok," Business Management and Strategy, Macrothink Institute, vol. 11(1), pages 55-71, June.
    5. Ishant Sharma & Sabyasachee Mishra, 2023. "Ranking preferences towards adopting autonomous vehicles based on peer inputs and advertisements," Transportation, Springer, vol. 50(6), pages 2139-2192, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Gordillo, Fernando & Elsasser, Peter & Günter, Sven, 2019. "Willingness to pay for forest conservation in Ecuador: Results from a nationwide contingent valuation survey in a combined “referendum” – “Consequential open-ended” design," Forest Policy and Economics, Elsevier, vol. 105(C), pages 28-39.
    2. Robert J. Johnston & Kevin J. Boyle & Wiktor (Vic) Adamowicz & Jeff Bennett & Roy Brouwer & Trudy Ann Cameron & W. Michael Hanemann & Nick Hanley & Mandy Ryan & Riccardo Scarpa & Roger Tourangeau & Ch, 2017. "Contemporary Guidance for Stated Preference Studies," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 4(2), pages 319-405.
    3. Shr, Yau-Huo (Jimmy) & Zhang, Wendong, 2024. "Omitted downstream attributes and the benefits of nutrient reductions: Implications for choice experiments," Ecological Economics, Elsevier, vol. 222(C).
    4. Frondel Manuel & Kutzschbauch Ole & Sommer Stephan & Traub Stefan, 2017. "Die Gerechtigkeitslücke in der Verteilung der Kosten der Energiewende auf die privaten Haushalte," Perspektiven der Wirtschaftspolitik, De Gruyter, vol. 18(4), pages 335-347, November.
    5. Thapa, Samir & Morrison, Mark & Parton, Kevin A, 2021. "Willingness to pay for domestic biogas plants and distributing carbon revenues to influence their purchase: A case study in Nepal," Energy Policy, Elsevier, vol. 158(C).
    6. Pengfei Liu & Lingling Hou & Dongqing Li & Shi Min & Yueying Mu, 2021. "Determinants of Livestock Insurance Demand: Experimental Evidence from Chinese Herders," Journal of Agricultural Economics, Wiley Blackwell, vol. 72(2), pages 430-451, June.
    7. Andor, Mark A. & Lange, Andreas & Sommer, Stephan, 2022. "Fairness and the support of redistributive environmental policies," Journal of Environmental Economics and Management, Elsevier, vol. 114(C).
    8. Malte Welling & Ewa Zawojska & Julian Sagebiel, 2022. "Information, Consequentiality and Credibility in Stated Preference Surveys: A Choice Experiment on Climate Adaptation," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 82(1), pages 257-283, May.
    9. Meles, Tensay Hadush & Lokina, Razack & Mtenga, Erica Louis & Tibanywana, Julieth Julius, 2023. "Stated preferences with survey consequentiality and outcome uncertainty: A split sample discrete choice experiment," EfD Discussion Paper 23-16, Environment for Development, University of Gothenburg.
    10. Zawojska, Ewa & Bartczak, Anna & Czajkowski, Mikołaj, 2019. "Disentangling the effects of policy and payment consequentiality and risk attitudes on stated preferences," Journal of Environmental Economics and Management, Elsevier, vol. 93(C), pages 63-84.
    11. Schaafsma, M. & van Beukering, P.J.H. & Oskolokaite, I., 2017. "Combining focus group discussions and choice experiments for economic valuation of peatland restoration: A case study in Central Kalimantan, Indonesia," Ecosystem Services, Elsevier, vol. 27(PA), pages 150-160.
    12. Wei, Xuan & Khachatryan, Hayk, 2023. "How consequential is policy consequentiality? Evidence from online discrete choice experiment with ornamental plants," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 107(C).
    13. Marco Lerro & Riccardo Vecchio & Francesco Caracciolo & Stefano Pascucci & Luigi Cembalo, 2018. "Consumers' heterogeneous preferences for corporate social responsibility in the food industry," Corporate Social Responsibility and Environmental Management, John Wiley & Sons, vol. 25(6), pages 1050-1061, November.
    14. Jie He & Jérôme Dupras & Franck Ndefo & Thomas Poder, 2020. "Payment and provision consequentiality in voluntary contribution mechanism: separate or joint effects?," Letters in Spatial and Resource Sciences, Springer, vol. 13(1), pages 11-36, April.
    15. Christian A. Vossler, 2016. "Chamberlin Meets Ciriacy-Wantrup: Using Insights from Experimental Economics to Inform Stated Preference Research," Canadian Journal of Agricultural Economics/Revue canadienne d'agroeconomie, Canadian Agricultural Economics Society/Societe canadienne d'agroeconomie, vol. 64(1), pages 33-48, March.
    16. Zawojska, Ewa & Gastineau, Pascal & Mahieu, Pierre-Alexandre & Cheze, Benoit & Paris, Anthony, 2021. "Measuring policy consequentiality perceptions in stated preference surveys," 2021 Annual Meeting, August 1-3, Austin, Texas 313977, Agricultural and Applied Economics Association.
    17. Christian A. Vossler & Stéphane Bergeron & Maurice Doyon & Daniel Rondeau, 2023. "Revisiting the Gap between the Willingness to Pay and Willingness to Accept for Public Goods," Journal of the Association of Environmental and Resource Economists, University of Chicago Press, vol. 10(2), pages 413-445.
    18. Manuel Frondel & Stephan Sommer & Lukas Tomberg, 2021. "WTA-WTP Disparity: The Role of Perceived Realism of the Valuation Setting," Land Economics, University of Wisconsin Press, vol. 97(1), pages 196-206.
    19. Lopez-Becerra, E.I. & Alcon, F., 2021. "Social desirability bias in the environmental economic valuation: An inferred valuation approach," Ecological Economics, Elsevier, vol. 184(C).
    20. Dardanoni, Valentino & Guerriero, Carla, 2021. "Young people' s willingness to pay for environmental protection," Ecological Economics, Elsevier, vol. 179(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:medema:v:39:y:2019:i:7:p:827-841. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.