IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/122120.html
   My bibliography  Save this paper

The efficacy of propensity score matching for separating selection and measurement effects across different survey modes

Author

Listed:
  • Kibuchi, Eliud
  • Sturgis, Patrick
  • Durrant, Gabriele B.
  • Maslovskaya, Olga

Abstract

Effective evaluation of data quality between data collected in different modes is complicated by the confounding of selection and measurement effects. This study evaluates the utility of propensity score matching (PSM) as a method that has been proposed as a means of removing selection effects across surveys conducted in different modes. Our results show large differences in estimates for the same variables between parallel face-to-face and online surveys, even after matching on standard demographic variables. Moreover, discrepancies in estimates are still present after matching between surveys conducted in the same (online) mode, where differences in measurement properties can be ruled out a priori. Our findings suggest that PSM has substantial limitations as a method for separating measurement and selection differences across modes and should be used only with caution.

Suggested Citation

  • Kibuchi, Eliud & Sturgis, Patrick & Durrant, Gabriele B. & Maslovskaya, Olga, 2024. "The efficacy of propensity score matching for separating selection and measurement effects across different survey modes," LSE Research Online Documents on Economics 122120, London School of Economics and Political Science, LSE Library.
    • Handle: RePEc:ehl:lserod:122120
    as

    Download full text from publisher

    File URL: http://eprints.lse.ac.uk/122120/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    2. Olga Maslovskaya & Bella Struminskaya & Gabriele Durrant, 2022. "The future of online data collection in social surveys: Challenges, developments and applications," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(3), pages 768-772, July.
    3. Alberto Abadie & David Drukker & Jane Leber Herr & Guido W. Imbens, 2004. "Implementing matching estimators for average treatment effects in Stata," Stata Journal, StataCorp LP, vol. 4(3), pages 290-311, September.
    4. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    5. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sascha O. Becker & Marco Caliendo, 2007. "Sensitivity analysis for average treatment effects," Stata Journal, StataCorp LP, vol. 7(1), pages 71-83, February.
    2. Fatema, Naureen, 2019. "Can land title reduce low-intensity interhousehold conflict incidences and associated damages in eastern DRC?," World Development, Elsevier, vol. 123(C), pages 1-1.
    3. Loris Guery & Anne Stevenot & Geoffrey T. Wood & Chris Brewster, 2017. "The Impact of Private Equity on Employment: The Consequences of Fund Country of Origin—New Evidence from France," Industrial Relations: A Journal of Economy and Society, Wiley Blackwell, vol. 56(4), pages 723-750, October.
    4. Szulc, Adam, 2009. "A matching estimator of household equivalence scales," Economics Letters, Elsevier, vol. 103(2), pages 81-83, May.
    5. Florian Gunsilius & Yuliang Xu, 2021. "Matching for causal effects via multimarginal unbalanced optimal transport," Papers 2112.04398, arXiv.org, revised Jul 2022.
    6. Cauê Carrilho & Gabriela Demarchi & Amy Duchelle & Sven Wunder & Carla Morsello, 2022. "Permanence of avoided deforestation in a Transamazon REDD+ initiative (Pará, Brazil)," CEE-M Working Papers hal-03614704, CEE-M, Universtiy of Montpellier, CNRS, INRA, Montpellier SupAgro.
    7. Carrilho, Cauê D. & Demarchi, Gabriela & Duchelle, Amy E. & Wunder, Sven & Morsello, Carla, 2022. "Permanence of avoided deforestation in a Transamazon REDD+ project (Pará, Brazil)," Ecological Economics, Elsevier, vol. 201(C).
    8. David McKenzie & John Gibson & Steven Stillman, 2010. "How Important Is Selection? Experimental vs. Non-Experimental Measures of the Income Gains from Migration," Journal of the European Economic Association, MIT Press, vol. 8(4), pages 913-945, June.
    9. Metin Çakır & Joseph V. Balagtas & Abigail M. Okrent & Mariana Urbina‐Ramirez, 2021. "Effects of Package Size on Household Food Purchases," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(2), pages 781-801, June.
    10. Shibayama, Sotaro, 2019. "Sustainable development of science and scientists: Academic training in life science labs," Research Policy, Elsevier, vol. 48(3), pages 676-692.
    11. McKenzie, David & Gibson, John & Stillman, Steven, 2006. "How important is selection ? Experimental versus non-experimental measures of the income gains from migration," Policy Research Working Paper Series 3906, The World Bank.
    12. Aassve, Arnstein & Arpino, Bruno, 2008. "Estimation of causal effects of fertility on economic wellbeing: evidence from rural Vietnam," ISER Working Paper Series 2007-27, Institute for Social and Economic Research.
    13. Steckenleiter, Carina & Lechner, Michael & Pawlowski, Tim & Schüttoff, Ute, 2019. "Do local public expenditures on sports facilities affect sports participation in Germany?," Economics Working Paper Series 1905, University of St. Gallen, School of Economics and Political Science.
    14. Heinrich, Carolyn J. & Lopez, Yeri, 2009. "Does Community Participation Produce Dividends in Social Investment Fund Projects?," World Development, Elsevier, vol. 37(9), pages 1554-1568, September.
    15. Martin Huber & Eva-Maria Oe{ss}, 2024. "A joint test of unconfoundedness and common trends," Papers 2404.16961, arXiv.org, revised Jun 2024.
    16. Greg J. Duncan & Christina M. Gibson-Davis, 2006. "Connecting Child Care Quality to Child Outcomes," Evaluation Review, , vol. 30(5), pages 611-630, October.
    17. Crespo Cuaresma, Jesus & Oberhofer, Harald & Vincelette, Gallina Andronova, 2014. "Firm growth and productivity in Belarus: New empirical evidence from the machine building industry," Journal of Comparative Economics, Elsevier, vol. 42(3), pages 726-738.
    18. Laure Kuhfuss & Julie Subervie, 2015. "Do agri-environmental schemes help reduce herbicide use? Evidence from a natural experiment in France," Post-Print hal-01199067, HAL.
    19. Rodrigo Dorantes-Gilardi & Aurora A. Ramírez-Álvarez & Diana Terrazas-Santamaría, 2023. "Is there a differentiated gender effect of collaboration with super-cited authors? Evidence from junior researchers in economics," Scientometrics, Springer;Akadémiai Kiadó, vol. 128(4), pages 2317-2336, April.
    20. González, Xulia & Pazó, Consuelo, 2008. "Do public subsidies stimulate private R&D spending?," Research Policy, Elsevier, vol. 37(3), pages 371-389, April.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:122120. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.