IDEAS home Printed from https://ideas.repec.org/a/eee/deveco/v163y2023ics0304387823000123.html
   My bibliography  Save this article

Assessors influence results: Evidence on enumerator effects and educational impact evaluations

Author

Listed:
  • Rodriguez-Segura, Daniel
  • Schueler, Beth E.

Abstract

A significant share of education research uses data collected by “enumerators.” It is well-documented that “enumerator effects”—or inconsistent practices among the people who administer measurement tools—can be a key source of error in survey data collection. However, it is less understood whether this is a problem for academic assessments. We leverage a remote phone-based mathematics assessment of primary school students and survey of their parents in Kenya. Enumerators, who were teachers in our partner's network, were randomized to students to study the presence of enumerator effects. We find that both the academic assessment and survey were prone to enumerator effects and use simulation to show that these effects were large enough to lead to spurious results at a troubling rate in impact evaluations. We therefore recommend assessment administrators randomize enumerators at the student level, orthogonal to categories being compared (e.g., treatment/control groups), and train enumerators to minimize bias.

Suggested Citation

  • Rodriguez-Segura, Daniel & Schueler, Beth E., 2023. "Assessors influence results: Evidence on enumerator effects and educational impact evaluations," Journal of Development Economics, Elsevier, vol. 163(C).
    • Handle: RePEc:eee:deveco:v:163:y:2023:i:c:s0304387823000123
    DOI: 10.1016/j.jdeveco.2023.103057
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0304387823000123
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jdeveco.2023.103057?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ian Brunton-Smith & Patrick Sturgis & George Leckie, 2017. "Detecting and understanding interviewer effects on survey data by using a cross-classified mixed effects location–scale model," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 180(2), pages 551-568, February.
    2. Baird, Sarah & Özler, Berk, 2012. "Examining the reliability of self-reported data on school participation," Journal of Development Economics, Elsevier, vol. 98(1), pages 89-93.
    3. Blattman, Christopher & Jamison, Julian & Koroknay-Palicz, Tricia & Rodrigues, Katherine & Sheridan, Margaret, 2016. "Measuring the measurement error: A method to qualitatively validate survey data," Journal of Development Economics, Elsevier, vol. 120(C), pages 99-112.
    4. Maayan S. Malter & Morris B. Holbrook & Barbara E. Kahn & Jeffrey R. Parker & Donald R. Lehmann, 2020. "The past, present, and future of consumer research," Marketing Letters, Springer, vol. 31(2), pages 137-149, September.
    5. Dubeck, Margaret M. & Gove, Amber, 2015. "The early grade reading assessment (EGRA): Its theoretical foundation, purpose, and limitations," International Journal of Educational Development, Elsevier, vol. 40(C), pages 315-322.
    6. Rodriguez-Segura, Daniel & Schueler, Beth E., 2022. "Can learning be measured by phone? Evidence from Kenya," Economics of Education Review, Elsevier, vol. 90(C).
    7. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(3), pages 1627-1673.
    8. Randall, Sara & Coast, Ernestina & Compaore, Natacha & Antoine, Philippe, 2013. "The power of the interviewer: a qualitative perspective on African survey data collection," LSE Research Online Documents on Economics 48277, London School of Economics and Political Science, LSE Library.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Beegle, Kathleen & Dillon, Andrew & Karlan, Dean & Udry, Christopher, 2024. "Introduction to the journal of development economics special issue on methods and measurement," Journal of Development Economics, Elsevier, vol. 170(C).
    2. Rebecca Walcott & Isabelle Cohen & Denise Ferris, 2024. "When Who Matters: Interviewer Effects and Survey Modality," Evaluation Review, , vol. 48(6), pages 1024-1049, December.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rodriguez-Segura, Daniel & Campton, Cole & Crouch, Luis & Slade, Timothy S., 2021. "Looking beyond changes in averages in evaluating foundational learning: Some inequality measures," International Journal of Educational Development, Elsevier, vol. 84(C).
    2. Rebecca Walcott & Isabelle Cohen & Denise Ferris, 2024. "When Who Matters: Interviewer Effects and Survey Modality," Evaluation Review, , vol. 48(6), pages 1024-1049, December.
    3. Fazzio, Ila & Eble, Alex & Lumsdaine, Robin L. & Boone, Peter & Bouy, Baboucarr & Hsieh, Pei-Tseng Jenny & Jayanty, Chitra & Johnson, Simon & Silva, Ana Filipa, 2021. "Large learning gains in pockets of extreme poverty: Experimental evidence from Guinea Bissau," Journal of Public Economics, Elsevier, vol. 199(C).
    4. Eble, Alex & Frost, Chris & Camara, Alpha & Bouy, Baboucarr & Bah, Momodou & Sivaraman, Maitri & Hsieh, Pei-Tseng Jenny & Jayanty, Chitra & Brady, Tony & Gawron, Piotr & Vansteelandt, Stijn & Boone, P, 2021. "How much can we remedy very low learning levels in rural parts of low-income countries? Impact and generalizability of a multi-pronged para-teacher intervention from a cluster-randomized trial in the ," Journal of Development Economics, Elsevier, vol. 148(C).
    5. Madeline Duhon & Lia Fernald & Joan Hamory & Edward Miguel & Eric Ochieng & Michael W. Walker, 2024. "Intergenerational Human Capital Impacts and Complementarities in Kenya," NBER Working Papers 32617, National Bureau of Economic Research, Inc.
    6. Elisabeth Beckmann & Lukas Olbrich & Joseph Sakshaug, 2024. "Multivariate assessment of interviewer-related errors in a cross-national economic survey (Lukas Olbrich, Elisabeth Beckmann, Joseph W. Sakshaug)," Working Papers 253, Oesterreichische Nationalbank (Austrian Central Bank).
    7. Yu Ding & Wayne S. DeSarbo & Dominique M. Hanssens & Kamel Jedidi & John G. Lynch & Donald R. Lehmann, 2020. "The past, present, and future of measurement and methods in marketing analysis," Marketing Letters, Springer, vol. 31(2), pages 175-186, September.
    8. Salman Asim & Stefan Dercon & Ravinder Gera Casley & Donna Harris, 2024. "Does Effective School Leadership Improve Student Progression and Test Scores? Evidence from a Field Experiment in Malawi," CSAE Working Paper Series 2024-05, Centre for the Study of African Economies, University of Oxford.
    9. Diether W Beuermann & C Kirabo Jackson & Laia Navarro-Sola & Francisco Pardo, 2023. "What is a Good School, and Can Parents Tell? Evidence on the Multidimensionality of School Output," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 90(1), pages 65-101.
    10. Tarcia Camily Cavalcante Quezado & Nuno Fortes & William Quezado Figueiredo Cavalcante, 2022. "The Influence of Corporate Social Responsibility and Business Ethics on Brand Fidelity: The Importance of Brand Love and Brand Attitude," Sustainability, MDPI, vol. 14(5), pages 1-20, March.
    11. Blimpo, Moussa P. & Pugatch, Todd, 2021. "Entrepreneurship education and teacher training in Rwanda," Journal of Development Economics, Elsevier, vol. 149(C).
    12. Deshpande, Ashwini & Desrochers, Alain & Ksoll, Christopher & Shonchoy, Abu S., 2017. "The Impact of a Computer-based Adult Literacy Program on Literacy and Numeracy: Evidence from India," World Development, Elsevier, vol. 96(C), pages 451-473.
    13. Jacobus de Hoop & Jed Friedman & Eeshani Kandpal & Furio C. Rosati, 2019. "Child Schooling and Child Work in the Presence of a Partial Education Subsidy," Journal of Human Resources, University of Wisconsin Press, vol. 54(2), pages 503-531.
    14. Christopher Blattman & Julian C. Jamison & Margaret Sheridan, 2015. "Reducing crime and violence: Experimental evidence from cognitive behavioral therapy in Liberia," NBER Working Papers 21204, National Bureau of Economic Research, Inc.
    15. Ardington, Cally & Wills, Gabrielle & Pretorius, Elizabeth & Mohohlwane, Nompumelelo & Menendez, Alicia, 2021. "Benchmarking oral reading fluency in the early grades in Nguni languages," International Journal of Educational Development, Elsevier, vol. 84(C).
    16. Fahmi Fidan Mahmut & Öztaş Ayhan H. & Batmaz İnci, 2021. "Interviewer allocation through interview–reinterview nested design for response error estimation in sample surveys," Statistics in Transition New Series, Statistics Poland, vol. 22(2), pages 67-93, June.
    17. Mo, Di & Bai, Yu & Shi, Yaojiang & Abbey, Cody & Zhang, Linxiu & Rozelle, Scott & Loyalka, Prashant, 2020. "Institutions, implementation, and program effectiveness: Evidence from a randomized evaluation of computer-assisted learning in rural China," Journal of Development Economics, Elsevier, vol. 146(C).
    18. Yue-Yi Hwa & Clare Leaver, 2021. "Management in education systems," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 37(2), pages 367-391.
    19. Bonan, Jacopo & Battiston, Pietro & Bleck, Jaimie & LeMay-Boucher, Philippe & Pareglio, Stefano & Sarr, Bassirou & Tavoni, Massimo, 2021. "Social interaction and technology adoption: Experimental evidence from improved cookstoves in Mali," World Development, Elsevier, vol. 144(C).
    20. Agostinelli, Francesco & Avitabile, Ciro & Bobba, Matteo, 2021. "Enhancing Human Capital in Children: A Case Study on Scaling," TSE Working Papers 21-1196, Toulouse School of Economics (TSE), revised Oct 2023.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:deveco:v:163:y:2023:i:c:s0304387823000123. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/devec .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.