IDEAS home Printed from https://ideas.repec.org/p/bca/bocawp/17-43.html
   My bibliography  Save this paper

The Mode is the Message: Using Predata as Exclusion Restrictions to Evaluate Survey Design

Author

Listed:
  • Heng Chen
  • Geoffrey R. Dunbar
  • Rallye Shen

Abstract

Changes in survey mode (e.g., online, offline) may influence the values of survey responses, and may be particularly problematic when comparing repeated cross-sectional surveys. This paper identifies mode effects by correcting for both unit non-response and sampling selection using the sample profile data (predata) — the individual’s number of previous survey invitations, the number of completed past surveys, and the reward points balance. The findings show that there is statistically significant evidence of mode effects in recall and subjective questions, but not for factual ones.

Suggested Citation

  • Heng Chen & Geoffrey R. Dunbar & Rallye Shen, 2017. "The Mode is the Message: Using Predata as Exclusion Restrictions to Evaluate Survey Design," Staff Working Papers 17-43, Bank of Canada.
    • Handle: RePEc:bca:bocawp:17-43
    as

    Download full text from publisher

    File URL: https://www.bankofcanada.ca/wp-content/uploads/2017/10/swp2017-43.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jeffrey E. Zabel, 1998. "An Analysis of Attrition in the Panel Study of Income Dynamics and the Survey of Income and Program Participation with an Application to a Model of Labor Market Behavior," Journal of Human Resources, University of Wisconsin Press, vol. 33(2), pages 479-506.
    2. Hans Fricke & Markus Frölich & Martin Huber & Michael Lechner, 2020. "Endogeneity and non‐response bias in treatment evaluation – nonparametric identification of causal effects by instruments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(5), pages 481-504, August.
    3. Sokbae Lee & Bernard Salanié, 2018. "Identifying Effects of Multivalued Treatments," Econometrica, Econometric Society, vol. 86(6), pages 1939-1963, November.
    4. Whitney K. Newey, 2007. "NONPARAMETRIC CONTINUOUS/DISCRETE CHOICE MODELS," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 48(4), pages 1429-1439, November.
    5. Jae Kwang Kim & Jongho Im, 2014. "Propensity score adjustment with several follow-ups," Biometrika, Biometrika Trust, vol. 101(2), pages 439-448.
    6. Martin Huber, 2012. "Identification of Average Treatment Effects in Social Experiments Under Alternative Forms of Attrition," Journal of Educational and Behavioral Statistics, , vol. 37(3), pages 443-474, June.
    7. Robert M. Groves & Steven G. Heeringa, 2006. "Responsive design for household surveys: tools for actively controlling survey errors and costs," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 169(3), pages 439-457, July.
    8. Martin Huber, 2014. "Treatment Evaluation in the Presence of Sample Selection," Econometric Reviews, Taylor & Francis Journals, vol. 33(8), pages 869-905, November.
    9. Jing Qin & Dean A. Follmann, 2014. "Semiparametric maximum likelihood inference by using failed contact attempts to adjust for nonignorable nonresponse," Biometrika, Biometrika Trust, vol. 101(4), pages 985-991.
    10. Matthias Schonlau & Arthur van Soest & Arie Kapteyn & Mick Couper, 2009. "Selection Bias in Web Surveys and the Use of Propensity Scores," Sociological Methods & Research, , vol. 37(3), pages 291-318, February.
    11. Alberto Abadie & Guido W. Imbens, 2016. "Matching on the Estimated Propensity Score," Econometrica, Econometric Society, vol. 84, pages 781-807, March.
    12. Heng Chen & Marie-Hélène Felt & Kim P. Huynh, 2017. "Retail payment innovations and cash usage: accounting for attrition by using refreshment samples," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(2), pages 503-530, February.
    13. Christopher Henry & Kim Huynh & Rallye Shen, 2015. "2013 Methods-of-Payment Survey Results," Discussion Papers 15-4, Bank of Canada.
    14. Luc Behaghel & Bruno Crépon & Marc Gurgand & Thomas Le Barbanchon, 2015. "Please Call Again: Correcting Nonresponse Bias in Treatment Effect Models," The Review of Economics and Statistics, MIT Press, vol. 97(5), pages 1070-1080, December.
    15. David S. Lee, 2009. "Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(3), pages 1071-1102.
    16. repec:hal:pseose:halshs-01245539 is not listed on IDEAS
    17. Joshua Angrist & Ivan Fernandez-Val, 2010. "ExtrapoLATE-ing: External Validity and Overidentification in the LATE Framework," NBER Working Papers 16566, National Bureau of Economic Research, Inc.
    18. Poirier, Dale J., 1980. "Partial observability in bivariate probit models," Journal of Econometrics, Elsevier, vol. 12(2), pages 209-217, February.
    19. Hamermesh, Daniel S. & Donald, Stephen G., 2008. "The effect of college curriculum on earnings: An affinity identifier for non-ignorable non-response bias," Journal of Econometrics, Elsevier, vol. 144(2), pages 479-491, June.
    20. Damião Nóbrega Da Silva & Chris Skinner & Jae Kwang Kim, 2016. "Using Binary Paradata to Correct for Measurement Error in Survey Data Analysis," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(514), pages 526-537, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Christopher Henry & Kim Huynh & Gradon Nicholls, 2018. "Bitcoin Awareness and Usage in Canada: An Update," Staff Analytical Notes 2018-23, Bank of Canada.
    2. Heng Chen & Marie-Hélène Felt & Christopher Henry, 2018. "2017 Methods-of-Payment Survey: Sample Calibration and Variance Estimation," Technical Reports 114, Bank of Canada.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hans Fricke & Markus Frölich & Martin Huber & Michael Lechner, 2020. "Endogeneity and non‐response bias in treatment evaluation – nonparametric identification of causal effects by instruments," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 35(5), pages 481-504, August.
    2. Possebom, Vitor, 2018. "Sharp bounds on the MTE with sample selection," MPRA Paper 89785, University Library of Munich, Germany.
    3. Teresa Molina Millán & Karen Macours, 2017. "Attrition in randomized control trials: Using tracking information to correct bias," FEUNL Working Paper Series novaf:wp1702, Universidade Nova de Lisboa, Faculdade de Economia.
    4. Martin Huber & Anna Solovyeva, 2020. "Direct and Indirect Effects under Sample Selection and Outcome Attrition," Econometrics, MDPI, vol. 8(4), pages 1-25, December.
    5. Ghanem, Dalia & Hirshleifer, Sarojini & Kédagni, Désiré & Ortiz-Becerra, Karen, 2024. "Correcting attrition bias using changes-in-changes," Journal of Econometrics, Elsevier, vol. 241(2).
    6. Xiaolin Sun & Xueyan Zhao & D. S. Poskitt, 2024. "Partially Identified Heterogeneous Treatment Effect with Selection: An Application to Gender Gaps," Papers 2410.01159, arXiv.org, revised Oct 2024.
    7. Martin Huber, 2019. "An introduction to flexible methods for policy evaluation," Papers 1910.00641, arXiv.org.
    8. Patricio S. Dalton & Haki Pamuk & Ravindra Ramrattan & Burak Uras & Daan van Soest, 2024. "Electronic Payment Technology and Business Finance: A Randomized, Controlled Trial with Mobile Money," Management Science, INFORMS, vol. 70(4), pages 2590-2625, April.
    9. Sungjin Cho & Jihye Kam & Soohyung Lee, 2018. "Efficient Supply Of Human Capital: Role Of College Major," The Singapore Economic Review (SER), World Scientific Publishing Co. Pte. Ltd., vol. 63(05), pages 1319-1343, December.
    10. Delius, Antonia & Sterck, Olivier, 2024. "Cash transfers and micro-enterprise performance: Theory and quasi-experimental evidence from Kenya," Journal of Development Economics, Elsevier, vol. 167(C).
    11. Huber Martin & Wüthrich Kaspar, 2019. "Local Average and Quantile Treatment Effects Under Endogeneity: A Review," Journal of Econometric Methods, De Gruyter, vol. 8(1), pages 1-27, January.
    12. Clément de Chaisemartin, 2017. "Tolerating defiance? Local average treatment effects without monotonicity," Quantitative Economics, Econometric Society, vol. 8(2), pages 367-396, July.
    13. McGovern, Mark E. & Canning, David & Bärnighausen, Till, 2018. "Accounting for non-response bias using participation incentives and survey design: An application using gift vouchers," Economics Letters, Elsevier, vol. 171(C), pages 239-244.
    14. Sokbae Lee & Bernard Salanié, 2018. "Identifying Effects of Multivalued Treatments," Econometrica, Econometric Society, vol. 86(6), pages 1939-1963, November.
    15. Huber, Martin & Meier, Jonas & Wallimann, Hannes, 2022. "Business analytics meets artificial intelligence: Assessing the demand effects of discounts on Swiss train tickets," Transportation Research Part B: Methodological, Elsevier, vol. 163(C), pages 22-39.
    16. David McKenzie, 2017. "Identifying and Spurring High-Growth Entrepreneurship: Experimental Evidence from a Business Plan Competition," American Economic Review, American Economic Association, vol. 107(8), pages 2278-2307, August.
    17. Martin Huber, 2010. "Identification of average treatment effects in social experiments under different forms of attrition," University of St. Gallen Department of Economics working paper series 2010 2010-22, Department of Economics, University of St. Gallen.
    18. Henry, Christopher S. & Huynh, Kim P. & Nicholls, Gradon, 2018. "Bitcoin awareness and usage in Canada," Journal of Digital Banking, Henry Stewart Publications, vol. 2(4), pages 311-337, May.
    19. Markus Frölich & Martin Huber, 2014. "Treatment Evaluation With Multiple Outcome Periods Under Endogeneity and Attrition," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(508), pages 1697-1711, December.
    20. Luc Behaghel & Karen Macours & Julie Subervie, 2019. "How can randomised controlled trials help improve the design of the common agricultural policy?," European Review of Agricultural Economics, Oxford University Press and the European Agricultural and Applied Economics Publications Foundation, vol. 46(3), pages 473-493.

    More about this item

    Keywords

    Econometric and statistical methods;

    JEL classification:

    • C8 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bca:bocawp:17-43. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/bocgvca.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.