IDEAS home Printed from https://ideas.repec.org/p/ehl/lserod/122120.html
   My bibliography  Save this paper

The efficacy of propensity score matching for separating selection and measurement effects across different survey modes

Author

Listed:
  • Kibuchi, Eliud
  • Sturgis, Patrick
  • Durrant, Gabriele B.
  • Maslovskaya, Olga

Abstract

Effective evaluation of data quality between data collected in different modes is complicated by the confounding of selection and measurement effects. This study evaluates the utility of propensity score matching (PSM) as a method that has been proposed as a means of removing selection effects across surveys conducted in different modes. Our results show large differences in estimates for the same variables between parallel face-to-face and online surveys, even after matching on standard demographic variables. Moreover, discrepancies in estimates are still present after matching between surveys conducted in the same (online) mode, where differences in measurement properties can be ruled out a priori. Our findings suggest that PSM has substantial limitations as a method for separating measurement and selection differences across modes and should be used only with caution.

Suggested Citation

  • Kibuchi, Eliud & Sturgis, Patrick & Durrant, Gabriele B. & Maslovskaya, Olga, 2024. "The efficacy of propensity score matching for separating selection and measurement effects across different survey modes," LSE Research Online Documents on Economics 122120, London School of Economics and Political Science, LSE Library.
    • Handle: RePEc:ehl:lserod:122120
    as

    Download full text from publisher

    File URL: http://eprints.lse.ac.uk/122120/
    File Function: Open access version.
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    2. Olga Maslovskaya & Bella Struminskaya & Gabriele Durrant, 2022. "The future of online data collection in social surveys: Challenges, developments and applications," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 185(3), pages 768-772, July.
    3. Alberto Abadie & David Drukker & Jane Leber Herr & Guido W. Imbens, 2004. "Implementing matching estimators for average treatment effects in Stata," Stata Journal, StataCorp LP, vol. 4(3), pages 290-311, September.
    4. Guido W. Imbens, 2004. "Nonparametric Estimation of Average Treatment Effects Under Exogeneity: A Review," The Review of Economics and Statistics, MIT Press, vol. 86(1), pages 4-29, February.
    5. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. G. Miller & Yuriy Pylypchuk, 2014. "Marital Status, Spousal Characteristics, and the Use of Preventive Care," Journal of Family and Economic Issues, Springer, vol. 35(3), pages 323-338, September.
    2. Grau, Nicolas & Hojman, Daniel & Mizala, Alejandra, 2018. "School closure and educational attainment: Evidence from a market-based system," Economics of Education Review, Elsevier, vol. 65(C), pages 1-17.
    3. Gillespie, Jeffrey & Nehring, Richard, 2014. "Pasture-Based versus Conventional Milk Production: Where Is the Profit?," Journal of Agricultural and Applied Economics, Cambridge University Press, vol. 46(4), pages 543-558, November.
    4. Sascha O. Becker & Marco Caliendo, 2007. "Sensitivity analysis for average treatment effects," Stata Journal, StataCorp LP, vol. 7(1), pages 71-83, February.
    5. Fatema, Naureen, 2019. "Can land title reduce low-intensity interhousehold conflict incidences and associated damages in eastern DRC?," World Development, Elsevier, vol. 123(C), pages 1-1.
    6. Loris Guery & Anne Stevenot & Geoffrey T. Wood & Chris Brewster, 2017. "The Impact of Private Equity on Employment: The Consequences of Fund Country of Origin—New Evidence from France," Industrial Relations: A Journal of Economy and Society, Wiley Blackwell, vol. 56(4), pages 723-750, October.
    7. Card, David E. & De la Rica Goiricelaya, Sara, 2004. "The effect of firm-level contracts on the structure of wages: evidence from matched employer-employee data," DFAEII Working Papers 1988-088X, University of the Basque Country - Department of Foundations of Economic Analysis II.
    8. Wagstaff, Adam & Yu, Shengchao, 2007. "Do health sector reforms have their intended impacts?: The World Bank's Health VIII project in Gansu province, China," Journal of Health Economics, Elsevier, vol. 26(3), pages 505-535, May.
    9. Anne Stevenot & Loris Guery & Geoffrey Wood & Chris Brewster, 2018. "Country of Origin Effects and New Financial Actors: Private Equity Investment and Work and Employment Practices of French Firms," British Journal of Industrial Relations, London School of Economics, vol. 56(4), pages 859-881, December.
    10. Szulc, Adam, 2009. "A matching estimator of household equivalence scales," Economics Letters, Elsevier, vol. 103(2), pages 81-83, May.
    11. Gabriela Simonet & Julie Subervie & Driss Ezzine-De-Blas & Marina Cromberg & Amy Duchelle, 2015. "Paying smallholders not to cut down the amazon forest: impact evaluation of a REDD+ pilot project," Working Papers 1514, Chaire Economie du climat.
    12. Neelsen, Sven & Limwattananon, Supon & O'Donnell, Owen & van Doorslaer, Eddy, 2019. "Universal health coverage: A (social insurance) job half done?," World Development, Elsevier, vol. 113(C), pages 246-258.
    13. Florian Gunsilius & Yuliang Xu, 2021. "Matching for causal effects via multimarginal unbalanced optimal transport," Papers 2112.04398, arXiv.org, revised Jul 2022.
    14. Cauê Carrilho & Gabriela Demarchi & Amy Duchelle & Sven Wunder & Carla Morsello, 2022. "Permanence of avoided deforestation in a Transamazon REDD+ initiative (Pará, Brazil)," CEE-M Working Papers hal-03614704, CEE-M, Universtiy of Montpellier, CNRS, INRA, Montpellier SupAgro.
    15. Carrilho, Cauê D. & Demarchi, Gabriela & Duchelle, Amy E. & Wunder, Sven & Morsello, Carla, 2022. "Permanence of avoided deforestation in a Transamazon REDD+ project (Pará, Brazil)," Ecological Economics, Elsevier, vol. 201(C).
    16. David McKenzie & John Gibson & Steven Stillman, 2010. "How Important Is Selection? Experimental vs. Non-Experimental Measures of the Income Gains from Migration," Journal of the European Economic Association, MIT Press, vol. 8(4), pages 913-945, June.
    17. Tommaso Nannicini, 2007. "Simulation-based sensitivity analysis for matching estimators," Stata Journal, StataCorp LP, vol. 7(3), pages 334-350, September.
    18. González, Xulia & Pazó, Consuelo, 2008. "Do public subsidies stimulate private R&D spending?," Research Policy, Elsevier, vol. 37(3), pages 371-389, April.
    19. Metin Çakır & Joseph V. Balagtas & Abigail M. Okrent & Mariana Urbina‐Ramirez, 2021. "Effects of Package Size on Household Food Purchases," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(2), pages 781-801, June.
    20. Plamen Nikolov & Hongjian Wang & Kevin Acker, 2020. "Wage premium of Communist Party membership: Evidence from China," Pacific Economic Review, Wiley Blackwell, vol. 25(3), pages 309-338, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ehl:lserod:122120. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: LSERO Manager (email available below). General contact details of provider: https://edirc.repec.org/data/lsepsuk.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.