IDEAS home Printed from https://ideas.repec.org/a/cup/polals/v26y2018i03p275-291_00.html
   My bibliography  Save this article

Worth Weighting? How to Think About and Use Weights in Survey Experiments

Author

Listed:
  • Miratrix, Luke W.
  • Sekhon, Jasjeet S.
  • Theodoridis, Alexander G.
  • Campos, Luis F.

Abstract

The popularity of online surveys has increased the prominence of using sampling weights to enhance claims of representativeness. Yet, much uncertainty remains regarding how these weights should be employed in survey experiment analysis: should they be used? If so, which estimators are preferred? We offer practical advice, rooted in the Neyman–Rubin model, for researchers working with survey experimental data. We examine simple, efficient estimators, and give formulas for their biases and variances. We provide simulations that examine these estimators as well as real examples from experiments administered online through YouGov. We find that for examining the existence of population treatment effects using high-quality, broadly representative samples recruited by top online survey firms, sample quantities, which do not rely on weights, are often sufficient. We found that sample average treatment effect (SATE) estimates did not appear to differ substantially from their weighted counterparts, and they avoided the substantial loss of statistical power that accompanies weighting. When precise estimates of population average treatment effects (PATE) are essential, we analytically show poststratifying on survey weights and/or covariates highly correlated with outcomes to be a conservative choice. While we show substantial gains in simulations, we find limited evidence of them in practice.

Suggested Citation

  • Miratrix, Luke W. & Sekhon, Jasjeet S. & Theodoridis, Alexander G. & Campos, Luis F., 2018. "Worth Weighting? How to Think About and Use Weights in Survey Experiments," Political Analysis, Cambridge University Press, vol. 26(3), pages 275-291, July.
    • Handle: RePEc:cup:polals:v:26:y:2018:i:03:p:275-291_00
    as

    Download full text from publisher

    File URL: https://www.cambridge.org/core/product/identifier/S1047198718000013/type/journal_article
    File Function: link to article abstract page
    Download Restriction: no
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Shannan K. Sweet & Jonathon P. Schuldt & Johannes Lehmann & Deborah A. Bossio & Dominic Woolf, 2021. "Perceptions of naturalness predict US public support for Soil Carbon Storage as a climate solution," Climatic Change, Springer, vol. 166(1), pages 1-15, May.
    2. Benjamin Lu & Jia Wan & Derek Ouyang & Jacob Goldin & Daniel E. Ho, 2024. "Quantifying the Uncertainty of Imputed Demographic Disparity Estimates: The Dual Bootstrap," NBER Chapters, in: Race, Ethnicity, and Economic Statistics for the 21st Century, National Bureau of Economic Research, Inc.
    3. Nicole E. Pashley & Luke W. Miratrix, 2021. "Insights on Variance Estimation for Blocked and Matched Pairs Designs," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 271-296, June.
    4. Jens Eger & Sebastian H. Schneider & Martin Bruder & Solveig H. Gleser, 2023. "Does Evidence Matter? The Impact of Evidence Regarding Aid Effectiveness on Attitudes Towards Aid," The European Journal of Development Research, Palgrave Macmillan;European Association of Development Research and Training Institutes (EADI), vol. 35(5), pages 1149-1172, October.
    5. Young, Linda J & Rater, Barbara R, 2021. "The Farm Producer Survey: Unit and Item Nonresponse," NASS Research Reports 327249, United States Department of Agriculture, National Agricultural Statistics Service.
    6. Chris Hanretty & Benjamin E. Lauderdale & Nick Vivyan, 2020. "A Choice‐Based Measure of Issue Importance in the Electorate," American Journal of Political Science, John Wiley & Sons, vol. 64(3), pages 519-535, July.
    7. Di Shu & Jessica G. Young & Sengwee Toh & Rui Wang, 2021. "Variance estimation in inverse probability weighted Cox models," Biometrics, The International Biometric Society, vol. 77(3), pages 1101-1117, September.
    8. Hege H. Bye & Hui Yu & Jennie Sofia Portice & Charles A. Ogunbode, 2023. "Interactions between migrant race and social status in predicting acceptance of climate migrants in Norway," Climatic Change, Springer, vol. 176(4), pages 1-16, April.
    9. Ranjbar, Setareh & Salvati, Nicola & Pacini, Barbara, 2023. "Estimating heterogeneous causal effects in observational studies using small area predictors," Computational Statistics & Data Analysis, Elsevier, vol. 184(C).
    10. Melody Y Huang & Harsh Parikh, 2024. "Towards Generalizing Inferences from Trials to Target Populations," Papers 2402.17042, arXiv.org, revised May 2024.
    11. Nicole E. Pashley & Luke W. Miratrix, 2022. "Block What You Can, Except When You Shouldn’t," Journal of Educational and Behavioral Statistics, , vol. 47(1), pages 69-100, February.
    12. Robert Johns & Ann‐Kristin Kölln, 2020. "Moderation and Competence: How a Party's Ideological Position Shapes Its Valence Reputation," American Journal of Political Science, John Wiley & Sons, vol. 64(3), pages 649-663, July.
    13. Naoki Egami & Erin Hartman, 2021. "Covariate selection for generalizing experimental results: Application to a large‐scale development program in Uganda," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(4), pages 1524-1548, October.
    14. Zach Branson & Tirthankar Dasgupta, 2020. "Sampling‐based Randomised Designs for Causal Inference under the Potential Outcomes Framework," International Statistical Review, International Statistical Institute, vol. 88(1), pages 101-121, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cup:polals:v:26:y:2018:i:03:p:275-291_00. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Kirk Stebbing (email available below). General contact details of provider: https://www.cambridge.org/pan .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.