IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/22165.html
   My bibliography  Save this paper

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations

Author

Listed:
  • Thomas S. Dee
  • Will Dobbie
  • Brian A. Jacob
  • Jonah Rockoff

Abstract

In this paper, we show that the design and decentralized, school-based scoring of New York’s high school exit exams – the Regents Examinations – led to the systematic manipulation of test sores just below important proficiency cutoffs. Our estimates suggest that teachers inflate approximately 40 percent of test scores near the proficiency cutoffs. Teachers are more likely to inflate the scores of high-achieving students on the margin, but low-achieving students benefit more from manipulation in aggregate due to the greater density of these students near the proficiency cutoffs. Exploiting a series of reforms that eliminated score manipulation, we find that inflating a student’s score to fall just above a cutoff increases his or her probability of graduating from high school by 27 percent. These results have important implications for educational attainment of marginal high school graduates. For example, we estimate that the black-white graduation gap is about 5 percent larger in the absence of test score manipulation.

Suggested Citation

  • Thomas S. Dee & Will Dobbie & Brian A. Jacob & Jonah Rockoff, 2016. "The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations," NBER Working Papers 22165, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:22165
    Note: ED
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w22165.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Hinnerich, Björn Tyrefors & Höglin, Erik & Johannesson, Magnus, 2011. "Are boys discriminated in Swedish high schools?," Economics of Education Review, Elsevier, vol. 30(4), pages 682-690, August.
    2. Ou, Dongshu, 2010. "To leave or not to leave? A regression discontinuity analysis of the impact of failing the high school exit exam," Economics of Education Review, Elsevier, vol. 29(2), pages 171-186, April.
    3. Victor Lavy, 2009. "Performance Pay and Teachers' Effort, Productivity, and Grading Ethics," American Economic Review, American Economic Association, vol. 99(5), pages 1979-2011, December.
    4. Lavy, Victor & Sand, Edith, 2015. "On The Origins of Gender Human Capital Gaps: Short and Long Term Consequences of Teachers’ Stereotypical Biases," The Warwick Economics Research Paper Series (TWERPS) 1085, University of Warwick, Department of Economics.
    5. Serena Canaan & Pierre Mouganie, 2018. "Returns to Education Quality for Low-Skilled Students: Evidence from a Discontinuity," Journal of Labor Economics, University of Chicago Press, vol. 36(2), pages 395-436.
    6. Papay, John P. & Willett, John B. & Murnane, Richard J., 2011. "Extending the regression-discontinuity approach to multiple assignment variables," Journal of Econometrics, Elsevier, vol. 161(2), pages 203-207, April.
    7. Derek Neal & Diane Whitmore Schanzenbach, 2010. "Left Behind by Design: Proficiency Counts and Test-Based Accountability," The Review of Economics and Statistics, MIT Press, vol. 92(2), pages 263-283, May.
    8. Lavy, Victor, 2008. "Do gender stereotypes reduce girls' or boys' human capital outcomes? Evidence from a natural experiment," Journal of Public Economics, Elsevier, vol. 92(10-11), pages 2083-2105, October.
    9. Giorgio Brunello & Margherita Fort & Guglielmo Weber, 2009. "Changes in Compulsory Schooling, Education and the Distribution of Wages in Europe," Economic Journal, Royal Economic Society, vol. 119(536), pages 516-539, March.
    10. Brian A. Jacob & Steven D. Levitt, 2003. "Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(3), pages 843-877.
    11. Jörn-Steffen Pischke & Till von Wachter, 2008. "Zero Returns to Compulsory Schooling in Germany: Evidence and Interpretation," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 592-598, August.
    12. Simon Burgess & Ellen Greaves, 2013. "Test Scores, Subjective Assessment, and Stereotyping of Ethnic Minorities," Journal of Labor Economics, University of Chicago Press, vol. 31(3), pages 535-576.
    13. Julie Berry Cullen & Randall Reback, 2006. "Tinkering Toward Accolades: School Gaming Under a Performance Accountability System," NBER Working Papers 12286, National Bureau of Economic Research, Inc.
    14. Henrik J. Kleven & Mazhar Waseem, 2013. "Using Notches to Uncover Optimization Frictions and Structural Elasticities: Theory and Evidence from Pakistan," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(2), pages 669-723.
    15. Petra Persson, 2020. "Social Insurance and the Marriage Market," Journal of Political Economy, University of Chicago Press, vol. 128(1), pages 252-300.
    16. Joshua D. Angrist & Alan B. Keueger, 1991. "Does Compulsory School Attendance Affect Schooling and Earnings?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 106(4), pages 979-1014.
    17. Derek Neal, 2013. "The Consequences of Using one Assessment System to Pursue two Objectives," The Journal of Economic Education, Taylor & Francis Journals, vol. 44(4), pages 339-352, October.
    18. Emmanuel Saez, 2010. "Do Taxpayers Bunch at Kink Points?," American Economic Journal: Economic Policy, American Economic Association, vol. 2(3), pages 180-212, August.
    19. Oreopoulos, Philip, 2007. "Do dropouts drop out too soon? Wealth, health and happiness from compulsory schooling," Journal of Public Economics, Elsevier, vol. 91(11-12), pages 2213-2229, December.
    20. Roland G. Fryer, 2013. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," Journal of Labor Economics, University of Chicago Press, vol. 31(2), pages 373-407.
    21. Damon Clark & Paco Martorell, 2014. "The Signaling Value of a High School Diploma," Journal of Political Economy, University of Chicago Press, vol. 122(2), pages 282-318.
    22. Raj Chetty & John N. Friedman & Tore Olsen & Luigi Pistaferri, 2011. "Adjustment Costs, Firm Responses, and Micro vs. Macro Labor Supply Elasticities: Evidence from Danish Tax Records," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(2), pages 749-804.
    23. Derek Neal, 2013. "The Consequences of Using One Assessment System To Pursue Two Objectives," NBER Working Papers 19214, National Bureau of Economic Research, Inc.
    24. David N. Figlio & Lawrence S. Getzler, 2002. "Accountability , Ability and Disability: Gaming the System," NBER Working Papers 9307, National Bureau of Economic Research, Inc.
    25. Jaeger, David A & Page, Marianne E, 1996. "Degrees Matter: New Evidence on Sheepskin Effects in the Returns to Education," The Review of Economics and Statistics, MIT Press, vol. 78(4), pages 733-740, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Figlio, D. & Karbownik, K. & Salvanes, K.G., 2016. "Education Research and Administrative Data," Handbook of the Economics of Education,, Elsevier.
    2. Rebecca Diamond & Petra Persson, 2016. "The Long-term Consequences of Teacher Discretion in Grading of High-stakes Tests," NBER Working Papers 22207, National Bureau of Economic Research, Inc.
    3. Cuesta, José Ignacio & González, Felipe & Larroulet Philippi, Cristian, 2020. "Distorted quality signals in school markets," Journal of Development Economics, Elsevier, vol. 147(C).
    4. Terrier, Camille, 2020. "Boys lag behind: How teachers’ gender biases affect student achievement," Economics of Education Review, Elsevier, vol. 77(C).
    5. Brehm, Margaret & Imberman, Scott A. & Lovenheim, Michael F., 2017. "Achievement effects of individual performance incentives in a teacher merit pay tournament," Labour Economics, Elsevier, vol. 44(C), pages 133-150.
    6. Seth Gershenson & Stephen B. Holt & Nicholas Papageorge, 2015. "Who Believes in Me? The Effect of Student-Teacher Demographic Match on Teacher Expectations," Upjohn Working Papers 15-231, W.E. Upjohn Institute for Employment Research.
    7. Huong Thu Le & Ha Trong Nguyen, 2018. "The evolution of the gender test score gap through seventh grade: new insights from Australia using unconditional quantile regression and decomposition," IZA Journal of Labor Economics, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 7(1), pages 1-42, December.
    8. Cornelisz, Ilja & Meeter, Martijn & van Klaveren, Chris, 2019. "Educational equity and teacher discretion effects in high stake exams," Economics of Education Review, Elsevier, vol. 73(C).
    9. Fan, Elliott & Lien, Hsienming & Ma, Ching-to Albert, 2019. "Uterus at a price: Disability insurance and hysterectomy," Journal of Health Economics, Elsevier, vol. 66(C), pages 1-17.
    10. Camille Terrier, 2014. "Giving a Little Help to Girls? Evidence on Grade Discrimination and its Effect on Students Achievement," Working Papers hal-01080834, HAL.
    11. Ulrik Hvidman & Hans Henrik Sievertsen, 2021. "High-Stakes Grades and Student Behavior," Journal of Human Resources, University of Wisconsin Press, vol. 56(3), pages 821-849.
    12. Barrera-Osorio, Felipe & Cilliers, Jacobus & Cloutier, Marie-Hélène & Filmer, Deon, 2022. "Heterogenous teacher effects of two incentive schemes: Evidence from a low-income country," Journal of Development Economics, Elsevier, vol. 156(C).
    13. Esmée Zwiers, 2024. "Estimating the Lifecycle Fertility Consequences of WWII Using Bunching," Tinbergen Institute Discussion Papers 24-027/V, Tinbergen Institute.
    14. Murat G. Kırdar & Meltem Dayıoğlu & İsmet Koç, 2016. "Does Longer Compulsory Education Equalize Schooling by Gender and Rural/Urban Residence?," The World Bank Economic Review, World Bank, vol. 30(3), pages 549-579.
    15. Rapoport, Benoît & Thibout, Claire, 2018. "Why do boys and girls make different educational choices? The influence of expected earnings and test scores," Economics of Education Review, Elsevier, vol. 62(C), pages 205-229.
    16. Meng, Xin & Zhao, Guochang, 2021. "The long shadow of a large scale education interruption: The intergenerational effect," Labour Economics, Elsevier, vol. 71(C).
    17. Joshua D. Angrist & Erich Battistin & Daniela Vuri, 2017. "In a Small Moment: Class Size and Moral Hazard in the Italian Mezzogiorno," American Economic Journal: Applied Economics, American Economic Association, vol. 9(4), pages 216-249, October.
    18. Derek Neal, 2011. "The Design of Performance Pay in Education," NBER Working Papers 16710, National Bureau of Economic Research, Inc.
    19. Bahadır Dursun & Resul Cesur, 2016. "Transforming lives: the impact of compulsory schooling on hope and happiness," Journal of Population Economics, Springer;European Society for Population Economics, vol. 29(3), pages 911-956, July.
    20. Isaac Mbiti & Mauricio Romero & Youdi Schipper, 2023. "Designing Effective Teacher Performance Pay Programs: Experimental Evidence from Tanzania," The Economic Journal, Royal Economic Society, vol. 133(653), pages 1968-2000.

    More about this item

    JEL classification:

    • I20 - Health, Education, and Welfare - - Education - - - General
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • I24 - Health, Education, and Welfare - - Education - - - Education and Inequality

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:22165. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.