IDEAS home Printed from https://ideas.repec.org/p/osf/osfxxx/qrwg4.html
   My bibliography  Save this paper

Social Media, Web, and Panel Surveys: Using Non- Probability Samples in Social and Policy Research

Author

Listed:
  • Lehdonvirta, Vili
  • Oksanen, Atte

    (University of Tampere)

  • Räsänen, Pekka
  • Blank, Grant

Abstract

The use of online surveys has grown rapidly in social science and policy research, surpassing more established methods. We argue that a better understanding is needed, especially of the strengths and weaknesses of non-probability online surveys that can be conducted relatively quickly and cheaply. We describe two common approaches to non-probability online surveys – river and panel sampling – and theorize their inherent selection biases: topical self-selection and economic self-selection. We conduct an empirical comparison of two river samples (Facebook and web-based) and one panel sample (from a major survey research company) with benchmark data grounded in a comprehensive population registry. We examine (1) how closely the online samples correspond with the benchmark, and (2) their usefulness in studying a non-demographic subpopulation. The river samples diverge from the benchmark on demographic variables and yield much higher means on non-demographic variables, even after weighting; we attribute this to topical self-selection. The panel is closer to the benchmark. When examining the characteristics of a non-demographic subpopulation, we detect no differences between the river and panel samples. We conclude that non-probability online surveys don’t replace probability surveys, but augment the researcher’s toolkit with new digital practices, such as exploratory studies of small and emerging non-demographic subpopulations.

Suggested Citation

  • Lehdonvirta, Vili & Oksanen, Atte & Räsänen, Pekka & Blank, Grant, 2020. "Social Media, Web, and Panel Surveys: Using Non- Probability Samples in Social and Policy Research," OSF Preprints qrwg4, Center for Open Science.
  • Handle: RePEc:osf:osfxxx:qrwg4
    DOI: 10.31219/osf.io/qrwg4
    as

    Download full text from publisher

    File URL: https://osf.io/download/5e8aff5b430166009ca0c239/
    Download Restriction: no

    File URL: https://libkey.io/10.31219/osf.io/qrwg4?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Malhotra, Neil & Krosnick, Jon A., 2007. "The Effect of Survey Mode and Sampling on Inferences about Political Attitudes and Behavior: Comparing the 2000 and 2004 ANES to Internet Surveys with Nonprobability Samples," Political Analysis, Cambridge University Press, vol. 15(3), pages 286-323, July.
    2. Lehdonvirta, Vili, 2018. "Flexibility in the Gig Economy: Managing Time on Three Online Piecework Platforms," SocArXiv k3hy4, Center for Open Science.
    3. Jelke Bethlehem, 2010. "Selection Bias in Web Surveys," International Statistical Review, International Statistical Institute, vol. 78(2), pages 161-188, August.
    4. Alexander Coppock & Thomas J. Leeper & Kevin J. Mullinix, 2018. "Generalizability of heterogeneous treatment effect estimates across samples," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 115(49), pages 12441-12446, December.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Stefano Visintin & Kea Tijdens & Stephanie Steinmetz & Pablo de Pedraza, 2015. "Task implementation heterogeneity and wage dispersion," IZA Journal of Labor Economics, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 4(1), pages 1-24, December.
    2. Gary Bolton & Eugen Dimant & Ulrich Schmidt, 2018. "When a Nudge Backfires. Using Observation with Social and Economic Incentives to Promote Pro-Social Behavior," PPE Working Papers 0017, Philosophy, Politics and Economics, University of Pennsylvania.
    3. Herr, Benjamin & Schörpf, Philip & Flecker, Jörg, 2021. "Virtuelle Kommunikationsräume als Ausgangspunkt für Interessenartikulation in der Onlineplattformarbeit [Virtual communication rooms as a launching area for the articulation of interests in remot," Industrielle Beziehungen. Zeitschrift für Arbeit, Organisation und Management, Verlag Barbara Budrich, vol. 28(2), pages 172-193.
    4. Lamberova, Natalia, 2021. "The puzzling politics of R&D: Signaling competence through risky projects," Journal of Comparative Economics, Elsevier, vol. 49(3), pages 801-818.
    5. Lasse J. Jessen & Sebastian Koehne & Patrick Nüß & Jens Ruhose, 2024. "Socioeconomic Inequality in Life Expectancy: Perception and Policy Demand," CESifo Working Paper Series 10940, CESifo.
    6. Ramón Ferri-García & María del Mar Rueda, 2022. "Variable selection in Propensity Score Adjustment to mitigate selection bias in online surveys," Statistical Papers, Springer, vol. 63(6), pages 1829-1881, December.
    7. Lehmann, Nico & Sloot, Daniel & Schüle, Christopher & Ardone, Armin & Fichtner, Wolf, 2023. "The motivational drivers behind consumer preferences for regional electricity – Results of a choice experiment in Southern Germany," Energy Economics, Elsevier, vol. 120(C).
    8. Giulia Casu & Marco Giovanni Mariani & Rita Chiesa & Dina Guglielmi & Paola Gremigni, 2021. "The Role of Organizational Citizenship Behavior and Gender between Job Satisfaction and Task Performance," IJERPH, MDPI, vol. 18(18), pages 1-15, September.
    9. Gilles Paché, 2020. "Inside Delivery Platforms: The Covid-19 Pandemic And After," Post-Print hal-03041080, HAL.
    10. Kevin J. Boyle & Mark Morrison & Darla Hatton MacDonald & Roderick Duncan & John Rose, 2016. "Investigating Internet and Mail Implementation of Stated-Preference Surveys While Controlling for Differences in Sample Frames," Environmental & Resource Economics, Springer;European Association of Environmental and Resource Economists, vol. 64(3), pages 401-419, July.
    11. Bertram, Christine & Rehdanz, Katrin, 2015. "The role of urban green space for human well-being," Ecological Economics, Elsevier, vol. 120(C), pages 139-152.
    12. Karytsas, Spyridon & Theodoropoulou, Helen, 2014. "Socioeconomic and demographic factors that influence publics' awareness on the different forms of renewable energy sources," Renewable Energy, Elsevier, vol. 71(C), pages 480-485.
    13. Galperin, Hernan & Arcidiacono, Malena, 2021. "Employment and the gender digital divide in Latin America: A decomposition analysis," Telecommunications Policy, Elsevier, vol. 45(7).
    14. Alex J. Wood, 2021. "Algorithmic Management: Consequences for Work Organisation and Working Conditions," JRC Working Papers on Labour, Education and Technology 2021-07, Joint Research Centre.
    15. Luis Castro-Martín & María del Mar Rueda & Ramón Ferri-García, 2020. "Estimating General Parameters from Non-Probability Surveys Using Propensity Score Adjustment," Mathematics, MDPI, vol. 8(11), pages 1-14, November.
    16. Bicchieri, Cristina & Dimant, Eugen & Xiao, Erte, 2021. "Deviant or wrong? The effects of norm information on the efficacy of punishment," Journal of Economic Behavior & Organization, Elsevier, vol. 188(C), pages 209-235.
    17. Aaron C. Sparks & Heather Hodges & Sarah Oliver & Eric R. A. N. Smith, 2020. "Confidence in Local, National, and International Scientists on Climate Change," Sustainability, MDPI, vol. 13(1), pages 1-13, December.
    18. Sakshaug Joseph W. & Wiśniowski Arkadiusz & Ruiz Diego Andres Perez & Blom Annelies G., 2019. "Supplementing Small Probability Samples with Nonprobability Samples: A Bayesian Approach," Journal of Official Statistics, Sciendo, vol. 35(3), pages 653-681, September.
    19. Lindhjem, Henrik & Navrud, Ståle, 2011. "Using Internet in Stated Preference Surveys: A Review and Comparison of Survey Modes," International Review of Environmental and Resource Economics, now publishers, vol. 5(4), pages 309-351, September.
    20. Grilli, Gianluca & Curtis, John, 2021. "An evaluation of public initiatives to change behaviours that affect water quality," Papers WP696, Economic and Social Research Institute (ESRI).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:osf:osfxxx:qrwg4. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: OSF (email available below). General contact details of provider: https://osf.io/preprints/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.