IDEAS home Printed from https://ideas.repec.org/p/tky/fseres/2022cf1194.html
   My bibliography  Save this paper

Still Biased? A Remaining Classical Selection Problem of RCTs in Education

Author

Listed:
  • Hikaru Kawarazaki

    (Graduate school in Economics at University College London)

  • Minhaj Mahmud

    (Asian Development Bank)

  • Yasuyuki Sawada

    (Faculty of Economics, The University of Tokyo)

  • Mai Seki

    (Ritsumeikan University)

  • Kazuma Takakura

    (Graduate School of Economics, The University of Tokyo)

Abstract

In the already very rich and crowded literature on education interventions, the use of test scores to capture students’cognitive abilities has been the norm when measuring the impact. We show that even in randomized controlled trials (RCTs), estimated treatment effects on the true latent abilities can still be biased towards zero, because test scores are often censored outside of zero and full marks. This paper employs sui generis data from a field experiment in Bangladesh as well as data sets from existing highly-cited studies in developing countries to illustrate theoretically and empirically that this remaining classical sample selection problem exists. We suggest three concrete ways to correct such bias: First, to employ the conventional sample selection correction methods; second, to use tests that are designed with an extensive set of questions from easy to challenging levels which allow students to answer the maximum they could; and third, to incorporate each student’s completion time in estimation.

Suggested Citation

  • Hikaru Kawarazaki & Minhaj Mahmud & Yasuyuki Sawada & Mai Seki & Kazuma Takakura, 2022. "Still Biased? A Remaining Classical Selection Problem of RCTs in Education," CIRJE F-Series CIRJE-F-1194, CIRJE, Faculty of Economics, University of Tokyo.
    • Handle: RePEc:tky:fseres:2022cf1194
    as

    Download full text from publisher

    File URL: http://www.cirje.e.u-tokyo.ac.jp/research/dp/2022/2022cf1194.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Duflo, Esther & Dupas, Pascaline & Kremer, Michael, 2015. "School governance, teacher incentives, and pupil–teacher ratios: Experimental evidence from Kenyan primary schools," Journal of Public Economics, Elsevier, vol. 123(C), pages 92-110.
    2. David K. Evans & Anna Popova, 2016. "What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews," The World Bank Research Observer, World Bank, vol. 31(2), pages 242-270.
    3. Esther Duflo & Pascaline Dupas & Michael Kremer, 2011. "Peer Effects, Teacher Incentives, and the Impact of Tracking: Evidence from a Randomized Evaluation in Kenya," American Economic Review, American Economic Association, vol. 101(5), pages 1739-1774, August.
    4. Menno Pradhan & Daniel Suryadarma & Amanda Beatty & Maisy Wong & Arya Gaduh & Armida Alisjahbana & Rima Prama Artha, 2014. "Improving Educational Quality through Enhancing Community Participation: Results from a Randomized Field Experiment in Indonesia," American Economic Journal: Applied Economics, American Economic Association, vol. 6(2), pages 105-126, April.
    5. Esther Duflo & Rema Hanna & Stephen P. Ryan, 2012. "Incentives Work: Getting Teachers to Come to School," American Economic Review, American Economic Association, vol. 102(4), pages 1241-1278, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    2. Francesca Marchetta & Tom Dilly, 2019. "Supporting Education in Africa: Opportunities and Challenges for an Impact Investor," Working Papers hal-02288103, HAL.
    3. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(3), pages 1627-1673.
    4. de Hoyos, Rafael, 2020. "Equal opportunities to enhance growth," World Development, Elsevier, vol. 127(C).
    5. Sabrin Beg & Waqas Halim & Adrienne M. Lucas & Umar Saif, 2022. "Engaging Teachers with Technology Increased Achievement, Bypassing Teachers Did Not," American Economic Journal: Economic Policy, American Economic Association, vol. 14(2), pages 61-90, May.
    6. Piper, Benjamin & Simmons Zuilkowski, Stephanie & Dubeck, Margaret & Jepkemei, Evelyn & King, Simon J., 2018. "Identifying the essential ingredients to literacy and numeracy improvement: Teacher professional development and coaching, student textbooks, and structured teachers’ guides," World Development, Elsevier, vol. 106(C), pages 324-336.
    7. Harounan Kazianga & Leigh Linden & Ali Protik & Matt Sloan, 2015. "Impact Evaluation of Burkina Faso's BRIGHT Program: Design Report," Mathematica Policy Research Reports c0250cd3f27d448ea70d909c3, Mathematica Policy Research.
    8. Miguel Urquiola, 2015. "Progress and challenges in achieving an evidence-based education policy in Latin America and the Caribbean," Latin American Economic Review, Springer;Centro de Investigaciòn y Docencia Económica (CIDE), vol. 24(1), pages 1-30, December.
    9. Committee, Nobel Prize, 2019. "Understanding development and poverty alleviation," Nobel Prize in Economics documents 2019-2, Nobel Prize Committee.
    10. Peters, Jörg & Langbein, Jörg & Roberts, Gareth, 2016. "Policy evaluation, randomized controlled trials, and external validity—A systematic review," Economics Letters, Elsevier, vol. 147(C), pages 51-54.
    11. Eduard Marinov, 2019. "The 2019 Nobel Prize in Economics," Economic Thought journal, Bulgarian Academy of Sciences - Economic Research Institute, issue 6, pages 78-116.
    12. Clair Null & Clemencia Cosentino & Swetha Sridharan & Laura Meyer, "undated". "Policies and Programs to Improve Secondary Education in Developing Countries: A Review of the Evidence," Mathematica Policy Research Reports 516e420e637c4851b15e6a3f6, Mathematica Policy Research.
    13. Tessa Bold & Deon Filmer & Gayle Martin & Ezequiel Molina & Brian Stacy & Christophe Rockmore & Jakob Svensson & Waly Wane, 2017. "Enrollment without Learning: Teacher Effort, Knowledge, and Skill in Primary Schools in Africa," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 185-204, Fall.
    14. Evans, David K. & Popova, Anna, 2016. "Cost-Effectiveness Analysis in Development: Accounting for Local Costs and Noisy Impacts," World Development, Elsevier, vol. 77(C), pages 262-276.
    15. Asim, Minahil & Dee, Thomas S., 2022. "Mobile Phones, Civic Engagement, and School Performance in Pakistan," Economics of Education Review, Elsevier, vol. 89(C).
    16. De Hoyos Navarro,Rafael E. & Holland,Peter Anthony & Troiano,Sara, 2015. "Understanding the trends in learning outcomes in Argentina, 2000 to 2012," Policy Research Working Paper Series 7518, The World Bank.
    17. Benjamin A. Olken, 2020. "Banerjee, Duflo, Kremer, and the Rise of Modern Development Economics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 122(3), pages 853-878, July.
    18. Bold, Tessa & Molina, Ezequiel & Filmer, Deon & Svensson, Jakob, 2018. "The Lost Human Capital: Teacher Knowledge and Student Learning in Africa," CEPR Discussion Papers 12956, C.E.P.R. Discussion Papers.
    19. Holla,Alaka & Bendini,Maria Magdalena & Dinarte Diaz,Lelys Ileana & Trako,Iva, 2021. "Is Investment in Preprimary Education Too Low ? Lessons from (Quasi) ExperimentalEvidence across Countries," Policy Research Working Paper Series 9723, The World Bank.
    20. Baylis, Kathy & Ham, Andres, 2015. "How important is spatial correlation in randomized controlled trials?," 2015 AAEA & WAEA Joint Annual Meeting, July 26-28, San Francisco, California 205586, Agricultural and Applied Economics Association.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tky:fseres:2022cf1194. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: CIRJE administrative office (email available below). General contact details of provider: https://edirc.repec.org/data/ritokjp.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.