IDEAS home Printed from https://ideas.repec.org/a/eee/pubeco/v239y2024ics0047272724001828.html
   My bibliography  Save this article

Myths of official measurement: Limits to test-based education reforms with weak governance

Author

Listed:
  • Singh, Abhijeet
  • Berg, Petter

Abstract

Assessment-led school reforms are a central pillar of policy packages recommended to address low student achievement in developing countries. We use direct audit evidence to assess the truthfulness of official assessments in a reform that has tested over 6 million students annually since 2011 in the large Indian state of Madhya Pradesh. Comparing responses to the same test questions by the same students shows a doubling of reported achievement in administrative data versus independent tests. This difference is lower, within schools, in grades with multiple test booklets and external grading. Overall, in contexts with weak governance, interventions relying on test-based accountability appear unlikely to succeed without complementary investments to assure data integrity.

Suggested Citation

  • Singh, Abhijeet & Berg, Petter, 2024. "Myths of official measurement: Limits to test-based education reforms with weak governance," Journal of Public Economics, Elsevier, vol. 239(C).
    • Handle: RePEc:eee:pubeco:v:239:y:2024:i:c:s0047272724001828
    DOI: 10.1016/j.jpubeco.2024.105246
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0047272724001828
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jpubeco.2024.105246?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Kenneth Y. Chay & Patrick J. McEwan & Miguel Urquiola, 2005. "The Central Role of Noise in Evaluating Interventions That Use Test Scores to Rank Schools," American Economic Review, American Economic Association, vol. 95(4), pages 1237-1258, September.
    2. Esther Duflo & Michael Greenstone & Nicholas Ryan, 2013. "Truth-telling by Third-party Auditors and the Response of Polluting Firms: Experimental Evidence from India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(4), pages 1499-1545.
    3. Tahir Andrabi & Jishnu Das & Asim Ijaz Khwaja, 2017. "Report Cards: The Impact of Providing School and Child Test Scores on Educational Markets," American Economic Review, American Economic Association, vol. 107(6), pages 1535-1563, June.
    4. Afridi, Farzana & Barooah, Bidisha & Somanathan, Rohini, 2020. "Improving learning outcomes through information provision: Experimental evidence from Indian villages," Journal of Development Economics, Elsevier, vol. 146(C).
    5. Clare Leaver & Owen Ozier & Pieter Serneels & Andrew Zeitlin, 2021. "Recruitment, Effort, and Retention Effects of Performance Contracts for Civil Servants: Experimental Evidence from Rwandan Primary Schools," American Economic Review, American Economic Association, vol. 111(7), pages 2213-2246, July.
    6. Oana Borcan & Mikael Lindahl & Andreea Mitrut, 2017. "Fighting Corruption in Education: What Works and Who Benefits?," American Economic Journal: Economic Policy, American Economic Association, vol. 9(1), pages 180-209, February.
    7. Figlio, D. & Karbownik, K. & Salvanes, K.G., 2016. "Education Research and Administrative Data," Handbook of the Economics of Education,, Elsevier.
    8. Niehaus, Paul & Sukhtankar, Sandip, 2013. "The marginal rate of corruption in public programs: Evidence from India," Journal of Public Economics, Elsevier, vol. 104(C), pages 52-64.
    9. Thomas S. Dee & Will Dobbie & Brian A. Jacob & Jonah Rockoff, 2019. "The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations," American Economic Journal: Applied Economics, American Economic Association, vol. 11(3), pages 382-423, July.
    10. Prashant Loyalka & Sean Sylvia & Chengfang Liu & James Chu & Yaojiang Shi, 2019. "Pay by Design: Teacher Performance Pay Design and the Distribution of Student Achievement," Journal of Labor Economics, University of Chicago Press, vol. 37(3), pages 621-662.
    11. Joshua D. Angrist & Erich Battistin & Daniela Vuri, 2017. "In a Small Moment: Class Size and Moral Hazard in the Italian Mezzogiorno," American Economic Journal: Applied Economics, American Economic Association, vol. 9(4), pages 216-249, October.
    12. Ritva Reinikka & Jakob Svensson, 2004. "Local Capture: Evidence from a Central Government Transfer Program in Uganda," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(2), pages 679-705.
    13. Karthik Muralidharan & Abhijeet Singh, 2020. "Improving Public Sector Management at Scale? Experimental Evidence on School Governance India," NBER Working Papers 28129, National Bureau of Economic Research, Inc.
    14. Randall Reback & Jonah Rockoff & Heather L. Schwartz, 2014. "Under Pressure: Job Security, Resource Allocation, and Productivity in Schools under No Child Left Behind," American Economic Journal: Economic Policy, American Economic Association, vol. 6(3), pages 207-241, August.
    15. Rebecca Diamond & Petra Persson, 2016. "The Long-term Consequences of Teacher Discretion in Grading of High-stakes Tests," NBER Working Papers 22207, National Bureau of Economic Research, Inc.
    16. Jonah Rockoff & Lesley J. Turner, 2010. "Short-Run Impacts of Accountability on School Quality," American Economic Journal: Economic Policy, American Economic Association, vol. 2(4), pages 119-147, November.
    17. Brian Jacob & Jesse Rothstein, 2016. "The Measurement of Student Ability in Modern Assessment Systems," Journal of Economic Perspectives, American Economic Association, vol. 30(3), pages 85-108, Summer.
    18. Muralidharan, Karthik & Das, Jishnu & Holla, Alaka & Mohpal, Aakash, 2017. "The fiscal cost of weak governance: Evidence from teacher absence in India," Journal of Public Economics, Elsevier, vol. 145(C), pages 116-135.
    19. César Martinelli & Susan W. Parker & Ana Cristina Pérez-Gea & Rodimiro Rodrigo, 2018. "Cheating and Incentives: Learning from a Policy Experiment," American Economic Journal: Economic Policy, American Economic Association, vol. 10(1), pages 298-325, February.
    20. Figlio, David & Loeb, Susanna, 2011. "School Accountability," Handbook of the Economics of Education, in: Erik Hanushek & Stephen Machin & Ludger Woessmann (ed.), Handbook of the Economics of Education, edition 1, volume 3, chapter 8, pages 383-421, Elsevier.
    21. Isaac M. Mbiti, 2016. "The Need for Accountability in Education in Developing Countries," Journal of Economic Perspectives, American Economic Association, vol. 30(3), pages 109-132, Summer.
    22. Isaac Mbiti & Karthik Muralidharan & Mauricio Romero & Youdi Schipper & Constantine Manda & Rakesh Rajani, 2019. "Inputs, Incentives, and Complementarities in Education: Experimental Evidence from Tanzania," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 134(3), pages 1627-1673.
    23. Berkhout, Emilie & Pradhan, Menno & Rahmawati, & Suryadarma, Daniel & Swarnata, Arya, 2024. "Using technology to prevent fraud in high stakes national school examinations: Evidence from Indonesia," Journal of Development Economics, Elsevier, vol. 170(C).
    24. Mizala, Alejandra & Urquiola, Miguel, 2013. "School markets: The impact of information approximating schools' effectiveness," Journal of Development Economics, Elsevier, vol. 103(C), pages 313-335.
    25. Clare Leaver & Owen Ozier & Pieter Serneels & Andrew Zeitlin, 2021. "Recruitment, effort, and retention effects of performance contracts for civil servants: Experimental evidence from Rwandan primary schools," Department of Economics Working Papers 2021-04, Department of Economics, Williams College.
    26. Mizala, Alejandra & Romaguera, Pilar & Urquiola, Miguel, 2007. "Socioeconomic status or noise? Tradeoffs in the generation of school quality information," Journal of Development Economics, Elsevier, vol. 84(1), pages 61-75, September.
    27. Karthik Muralidharan & Venkatesh Sundararaman, 2011. "Teacher Performance Pay: Experimental Evidence from India," Journal of Political Economy, University of Chicago Press, vol. 119(1), pages 39-77.
    28. Benjamin A. Olken, 2007. "Monitoring Corruption: Evidence from a Field Experiment in Indonesia," Journal of Political Economy, University of Chicago Press, vol. 115(2), pages 200-249.
    29. Campbell, Donald T., 1979. "Assessing the impact of planned social change," Evaluation and Program Planning, Elsevier, vol. 2(1), pages 67-90, January.
    30. Abhijeet Singh, 2024. "Improving Administrative Data at Scale: Experimental Evidence on Digital Testing in Indian Schools," The Economic Journal, Royal Economic Society, vol. 134(661), pages 2207-2223.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yue-Yi Hwa & Clare Leaver, 2021. "Management in education systems," Oxford Review of Economic Policy, Oxford University Press and Oxford Review of Economic Policy Limited, vol. 37(2), pages 367-391.
    2. Cuesta, José Ignacio & González, Felipe & Larroulet Philippi, Cristian, 2020. "Distorted quality signals in school markets," Journal of Development Economics, Elsevier, vol. 147(C).
    3. Barrera-Osorio, Felipe & Cilliers, Jacobus & Cloutier, Marie-Hélène & Filmer, Deon, 2022. "Heterogenous teacher effects of two incentive schemes: Evidence from a low-income country," Journal of Development Economics, Elsevier, vol. 156(C).
    4. de Hoyos, Rafael & Garcia-Moreno, Vicente A. & Patrinos, Harry Anthony, 2017. "The impact of an accountability intervention with diagnostic feedback: Evidence from Mexico," Economics of Education Review, Elsevier, vol. 58(C), pages 123-140.
    5. Karthik Muralidharan & Paul Niehaus & Sandip Sukhtankar, 2016. "Building State Capacity: Evidence from Biometric Smartcards in India," American Economic Review, American Economic Association, vol. 106(10), pages 2895-2929, October.
    6. Machin, Stephen & McNally, Sandra & Ruiz-Valenzuela, Jenifer, 2020. "Entry through the narrow door: The costs of just failing high stakes exams," Journal of Public Economics, Elsevier, vol. 190(C).
    7. Hao, Zhuoqun & Liu, Yu & Zhang, Jinfan & Zhao, Xiaoxue, 2020. "Political connection, corporate philanthropy and efficiency: Evidence from China’s anti-corruption campaign," Journal of Comparative Economics, Elsevier, vol. 48(3), pages 688-708.
    8. Dennis Epple & Richard E. Romano & Miguel Urquiola, 2017. "School Vouchers: A Survey of the Economics Literature," Journal of Economic Literature, American Economic Association, vol. 55(2), pages 441-492, June.
    9. Berkhout, Emilie & Pradhan, Menno & Rahmawati, & Suryadarma, Daniel & Swarnata, Arya, 2024. "Using technology to prevent fraud in high stakes national school examinations: Evidence from Indonesia," Journal of Development Economics, Elsevier, vol. 170(C).
    10. Berlinski, Samuel & Ramos, Alejandra, 2020. "Teacher mobility and merit pay: Evidence from a voluntary public award program," Journal of Public Economics, Elsevier, vol. 186(C).
    11. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    12. M. Shahe Emran & Asadul Islam & Forhad Shilpi, 2020. "Distributional Effects of Corruption When Enforcement is Biased: Theory and Evidence from Bribery in Schools in Bangladesh," Economica, London School of Economics and Political Science, vol. 87(348), pages 985-1015, October.
    13. Cilliers, Jacobus & Kasirye, Ibrahim & Leaver, Clare & Serneels, Pieter & Zeitlin, Andrew, 2018. "Pay for locally monitored performance? A welfare analysis for teacher attendance in Ugandan primary schools," Journal of Public Economics, Elsevier, vol. 167(C), pages 69-90.
    14. Evans, David K. & Yuan, Fei & Filmer, Deon, 2022. "Teacher pay in Africa: Evidence from 15 countries," World Development, Elsevier, vol. 155(C).
    15. Muralidharan, Karthik & Das, Jishnu & Holla, Alaka & Mohpal, Aakash, 2017. "The fiscal cost of weak governance: Evidence from teacher absence in India," Journal of Public Economics, Elsevier, vol. 145(C), pages 116-135.
    16. Claudio Ferraz & Frederico Finan & Diana Belo Moreira, 2009. "Corrupting Learning: Evidence from Missing Federal Education Funds in Brazil," Textos para discussão 562, Department of Economics PUC-Rio (Brazil).
    17. Dustan, Andrew & Hernandez-Agramonte, Juan Manuel & Maldonado, Stanislao, 2023. "Motivating bureaucrats with behavioral insights when state capacity is weak: Evidence from large-scale field experiments in Peru," Journal of Development Economics, Elsevier, vol. 160(C).
    18. Abhijeet Singh, 2024. "Improving Administrative Data at Scale: Experimental Evidence on Digital Testing in Indian Schools," The Economic Journal, Royal Economic Society, vol. 134(661), pages 2207-2223.
    19. Michael Coelli & Gigi Foster & Andrew Leigh, 2018. "Do School Principals Respond to Increased Public Scrutiny? New Survey Evidence from Australia," The Economic Record, The Economic Society of Australia, vol. 94(S1), pages 73-101, June.
    20. Morozumi, Atsuyoshi & Tanaka, Ryuichi, 2020. "Should School-Level Results of National Assessments Be Made Public?," IZA Discussion Papers 13450, Institute of Labor Economics (IZA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:pubeco:v:239:y:2024:i:c:s0047272724001828. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/inca/505578 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.