IDEAS home Printed from https://ideas.repec.org/a/tpr/edfpol/v12y2017i2p197-223.html
   My bibliography  Save this article

Screen Twice, Cut Once: Assessing the Predictive Validity of Applicant Selection Tools

Author

Listed:
  • Dan Goldhaber

    (Center for Education Data & Research University of Washington– Bothell Seattle, WA 98103)

  • Cyrus Grout

    (Center for Education Data & Research University of Washington-- Bothell Seattle, WA 98103)

  • Nick Huntington-Klein

    (Department of Economics California State University, Fullerton Fullerton, CA 92831)

Abstract

Despite their widespread use, there is little academic evidence on whether applicant selection instruments can improve teacher hiring. We examine the relationship between two screening instruments used by Spokane Public Schools to select classroom teachers and three teacher outcomes: value added, absences, and attrition. We observe all applicants to the district (not only those who are hired), allowing us to estimate sample selection-corrected models using random tally errors and variation in the level of competition across job postings as instruments. Ratings on the screening instruments significantly predict value added in math and teacher attrition, but not absences—an increase of one standard deviation in screening scores is associated with an increase of about 0.06 standard deviations of student math achievement, and a decrease in teacher attrition of 3 percentage points. Hence the use of selection instruments appears to be a key means of improving the quality of the teacher workforce.

Suggested Citation

  • Dan Goldhaber & Cyrus Grout & Nick Huntington-Klein, 2017. "Screen Twice, Cut Once: Assessing the Predictive Validity of Applicant Selection Tools," Education Finance and Policy, MIT Press, vol. 12(2), pages 197-223, Spring.
    • Handle: RePEc:tpr:edfpol:v:12:y:2017:i:2:p:197-223
    as

    Download full text from publisher

    File URL: http://www.mitpressjournals.org/doi/pdf/10.1162/EDFP_a_00200
    Download Restriction: Access to PDF is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Steven G. Rivkin & Eric A. Hanushek & John F. Kain, 2005. "Teachers, Schools, and Academic Achievement," Econometrica, Econometric Society, vol. 73(2), pages 417-458, March.
    2. Erik Hanushek & Stephen Machin & Ludger Woessmann (ed.), 2011. "Handbook of the Economics of Education," Handbook of the Economics of Education, Elsevier, edition 1, volume 4, number 4, June.
    3. Steven Glazerman & Daniel Mayer & Paul Decker, 2006. "Alternative routes to teaching: The impacts of Teach for America on student achievement and other outcomes," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 25(1), pages 75-96.
    4. Thomas J. Kane & Douglas O. Staiger, 2008. "Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation," NBER Working Papers 14607, National Bureau of Economic Research, Inc.
    5. repec:mpr:mprres:4761 is not listed on IDEAS
    6. Charles T. Clotfelter & Helen F. Ladd & Jacob L. Vigdor, 2011. "Teacher Mobility, School Segregation, and Pay-Based Policies to Level the Playing Field," Education Finance and Policy, MIT Press, vol. 6(3), pages 399-438, July.
    7. Don Boyd & Hamp Lankford & Susanna Loeb & Matthew Ronfeldt & Jim Wyckoff, 2011. "The role of teacher quality in retention and hiring: Using applications to transfer to uncover preferences of teachers and schools," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 30(1), pages 88-110, December.
    8. Olsen, Randall J, 1980. "A Least Squares Correction for Selectivity Bias," Econometrica, Econometric Society, vol. 48(7), pages 1815-1820, November.
    9. Edward P. Lazear & Kathryn L. Shaw, 2007. "Personnel Economics: The Economist's View of Human Resources," Journal of Economic Perspectives, American Economic Association, vol. 21(4), pages 91-114, Fall.
    10. Eric A. Hanushek & John F. Kain & Steven G. Rivkin & Daniel M. O'Brien, 2005. "The Market for Teacher Quality," Discussion Papers 04-025, Stanford Institute for Economic Policy Research.
    11. Daniel Aaronson & Lisa Barrow & William Sander, 2007. "Teachers and Student Achievement in the Chicago Public High Schools," Journal of Labor Economics, University of Chicago Press, vol. 25(1), pages 95-135.
    12. Hinrichs, Peter, 2021. "What kind of teachers are schools looking for? Evidence from a randomized field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 186(C), pages 395-411.
    13. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates," American Economic Review, American Economic Association, vol. 104(9), pages 2593-2632, September.
    14. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood," American Economic Review, American Economic Association, vol. 104(9), pages 2633-2679, September.
    15. Dan Goldhaber & Michael Hansen, 2013. "Is it Just a Bad Class? Assessing the Long-term Stability of Estimated Teacher Performance," Economica, London School of Economics and Political Science, vol. 80(319), pages 589-612, July.
    16. Mariesa A. Herrmann & Jonah E. Rockoff, 2012. "Worker Absence and Productivity: Evidence from Teaching," Journal of Labor Economics, University of Chicago Press, vol. 30(4), pages 749-782.
    17. Donald Boyd & Hamilton Lankford & Susanna Loeb & James Wyckoff, 2013. "Analyzing the Determinants of the Matching of Public School Teachers to Jobs: Disentangling the Preferences of Teachers and Employers," Journal of Labor Economics, University of Chicago Press, vol. 31(1), pages 83-117.
    18. Jonah E. Rockoff & Brian A. Jacob & Thomas J. Kane & Douglas O. Staiger, 2011. "Can You Recognize an Effective Teacher When You Recruit One?," Education Finance and Policy, MIT Press, vol. 6(1), pages 43-74, January.
    19. Andrew Bacher-Hicks & Thomas J. Kane & Douglas O. Staiger, 2014. "Validating Teacher Effect Estimates Using Changes in Teacher Assignments in Los Angeles," NBER Working Papers 20657, National Bureau of Economic Research, Inc.
    20. Oyer, Paul & Schaefer, Scott, 2011. "Personnel Economics: Hiring and Incentives," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 4, chapter 20, pages 1769-1823, Elsevier.
    21. Jonah E. Rockoff, 2004. "The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data," American Economic Review, American Economic Association, vol. 94(2), pages 247-252, May.
    22. Richard Blundell & Monica Costa Dias, 2000. "Evaluation methods for non-experimental data," Fiscal Studies, Institute for Fiscal Studies, vol. 21(4), pages 427-468, January.
    23. Neal, Derek, 2011. "The Design of Performance Pay in Education," Handbook of the Economics of Education, in: Erik Hanushek & Stephen Machin & Ludger Woessmann (ed.), Handbook of the Economics of Education, edition 1, volume 4, chapter 0, pages 495-550, Elsevier.
    24. Thomas S. Dee & James Wyckoff, 2015. "Incentives, Selection, and Teacher Performance: Evidence from IMPACT," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 34(2), pages 267-297, March.
    25. Douglas O. Staiger & Jonah E. Rockoff, 2010. "Searching for Effective Teachers with Imperfect Information," Journal of Economic Perspectives, American Economic Association, vol. 24(3), pages 97-118, Summer.
    26. Eric A. Hanushek & Steven G. Rivkin, 2010. "Generalizations about Using Value-Added Measures of Teacher Quality," American Economic Review, American Economic Association, vol. 100(2), pages 267-271, May.
    27. Dan Goldhaber & Roddy Theobald, 2013. "Managing the Teacher Workforce in Austere Times: The Determinants and Implications of Teacher Layoffs," Education Finance and Policy, MIT Press, vol. 8(4), pages 494-527, October.
    28. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    29. Koedel, Cory & Mihaly, Kata & Rockoff, Jonah E., 2015. "Value-added modeling: A review," Economics of Education Review, Elsevier, vol. 47(C), pages 180-195.
    30. Joshua D. Angrist, 1998. "Estimating the Labor Market Impact of Voluntary Military Service Using Social Security Data on Military Applicants," Econometrica, Econometric Society, vol. 66(2), pages 249-288, March.
    31. Charles T. Clotfelter & Helen F. Ladd & Jacob L. Vigdor, 2006. "Teacher-Student Matching and the Assessment of Teacher Effectiveness," Journal of Human Resources, University of Wisconsin Press, vol. 41(4).
    32. Strauss, Robert P. & Bowes, Lori R. & Marks, Mindy S. & Plesko, Mark R., 2000. "Improving teacher preparation and selection: lessons from the Pennsylvania experience," Economics of Education Review, Elsevier, vol. 19(4), pages 387-415, October.
    33. Derek Neal, 2011. "The Design of Performance Pay in Education," NBER Working Papers 16710, National Bureau of Economic Research, Inc.
    34. Douglas N. Harris & Stacey A. Rutledge & William K. Ingle & Cynthia C. Thompson, 2010. "Mix and Match: What Principals Really Look for When Hiring Teachers," Education Finance and Policy, MIT Press, vol. 5(2), pages 228-246, April.
    35. Dale Ballou, 1996. "Do Public Schools Hire the Best Applicants?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 111(1), pages 97-133.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Matthew A. Kraft & John P. Papay & Olivia L. Chi, 2020. "Teacher Skill Development: Evidence from Performance Ratings by Principals," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 315-347, March.
    2. Patrícia Martinková & Dan Goldhaber & Elena Erosheva, 2018. "Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-17, October.
    3. Brian Jacob & Jonah E. Rockoff & Eric S. Taylor & Benjamin Lindy & Rachel Rosen, 2016. "Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools," NBER Working Papers 22054, National Bureau of Economic Research, Inc.
    4. Brutti, Zelda & Sánchez Torres, Fabio, 2022. "Turning around teacher quality in Latin America: Renewed confidence and lessons from Colombia," Economic Analysis and Policy, Elsevier, vol. 73(C), pages 62-93.
    5. Dan Goldhaber, 2018. "Impact and Your Death Bed: Playing the Long Game," Education Finance and Policy, MIT Press, vol. 13(1), pages 1-18, Winter.
    6. Thijssen, Maximiliaan W.P. & Rege, Mari & Solheim, Oddny J., 2022. "Teacher relationship skills and student learning," Economics of Education Review, Elsevier, vol. 89(C).
    7. Jessalynn James & Matthew A. Kraft & John P. Papay, 2023. "Local supply, temporal dynamics, and unrealized potential in teacher hiring," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 42(4), pages 1010-1044, September.
    8. Jacob, Brian A. & Rockoff, Jonah E. & Taylor, Eric S. & Lindy, Benjamin & Rosen, Rachel, 2018. "Teacher applicant hiring and teacher performance: Evidence from DC public schools," Journal of Public Economics, Elsevier, vol. 166(C), pages 81-97.
    9. Goldhaber, Dan & Grout, Cyrus & Wolff, Malcolm & Martinková, Patrícia, 2021. "Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants," Economics of Education Review, Elsevier, vol. 83(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Dhushyanth Raju, 2017. "Public School Teacher Management in Sri Lanka," South Asia Economic Journal, Institute of Policy Studies of Sri Lanka, vol. 18(1), pages 39-63, March.
    2. Hinrichs, Peter, 2021. "What kind of teachers are schools looking for? Evidence from a randomized field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 186(C), pages 395-411.
    3. Gershenson, Seth, 2021. "Identifying and Producing Effective Teachers," IZA Discussion Papers 14096, Institute of Labor Economics (IZA).
    4. Jacob, Brian A. & Rockoff, Jonah E. & Taylor, Eric S. & Lindy, Benjamin & Rosen, Rachel, 2018. "Teacher applicant hiring and teacher performance: Evidence from DC public schools," Journal of Public Economics, Elsevier, vol. 166(C), pages 81-97.
    5. Michael Bates & Michael Dinerstein & Andrew C. Johnston & Isaac Sorkin, 2022. "Teacher Labor Market Equilibrium and Student Achievement," CESifo Working Paper Series 9551, CESifo.
    6. Hanushek, Eric A. & Rivkin, Steven G. & Schiman, Jeffrey C., 2016. "Dynamic effects of teacher turnover on the quality of instruction," Economics of Education Review, Elsevier, vol. 55(C), pages 132-148.
    7. Brian Jacob & Jonah E. Rockoff & Eric S. Taylor & Benjamin Lindy & Rachel Rosen, 2016. "Teacher Applicant Hiring and Teacher Performance: Evidence from DC Public Schools," NBER Working Papers 22054, National Bureau of Economic Research, Inc.
    8. Figlio, D. & Karbownik, K. & Salvanes, K.G., 2016. "Education Research and Administrative Data," Handbook of the Economics of Education,, Elsevier.
    9. Pope, Nolan G., 2019. "The effect of teacher ratings on teacher performance," Journal of Public Economics, Elsevier, vol. 172(C), pages 84-110.
    10. Koedel, Cory & Mihaly, Kata & Rockoff, Jonah E., 2015. "Value-added modeling: A review," Economics of Education Review, Elsevier, vol. 47(C), pages 180-195.
    11. M. Caridad Araujo & Pedro Carneiro & Yyannú Cruz-Aguayo & Norbert Schady, 2016. "Teacher Quality and Learning Outcomes in Kindergarten," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(3), pages 1415-1453.
    12. Filmer,Deon P. & Nahata,Vatsal & Sabarwal,Shwetlena, 2021. "Preparation, Practice, and Beliefs : A Machine Learning Approach to Understanding Teacher Effectiveness," Policy Research Working Paper Series 9847, The World Bank.
    13. Azam, Mehtabul & Kingdon, Geeta Gandhi, 2015. "Assessing teacher quality in India," Journal of Development Economics, Elsevier, vol. 117(C), pages 74-83.
    14. Johnston, Andrew C., 2021. "Preferences, Selection, and the Structure of Teacher Pay," IZA Discussion Papers 14831, Institute of Labor Economics (IZA).
    15. David Blazar, 2018. "Validating Teacher Effects on Students’ Attitudes and Behaviors: Evidence from Random Assignment of Teachers to Students," Education Finance and Policy, MIT Press, vol. 13(3), pages 281-309, Summer.
    16. Marc Steeg & Sander Gerritsen, 2016. "Teacher Evaluations and Pupil Achievement Gains: Evidence from Classroom Observations," De Economist, Springer, vol. 164(4), pages 419-443, December.
    17. Goel, Deepti & Barooah, Bidisha, 2018. "Drivers of Student Performance: Evidence from Higher Secondary Public Schools in Delhi," GLO Discussion Paper Series 231, Global Labor Organization (GLO).
    18. Allison Atteberry & Susanna Loeb & James Wyckoff, 2013. "Do First Impressions Matter? Improvement in Early Career Teacher Effectiveness," NBER Working Papers 19096, National Bureau of Economic Research, Inc.
    19. Canales, Andrea & Maldonado, Luis, 2018. "Teacher quality and student achievement in Chile: Linking teachers' contribution and observable characteristics," International Journal of Educational Development, Elsevier, vol. 60(C), pages 33-50.
    20. Goldhaber, Dan & Krieg, John & Theobald, Roddy, 2020. "Effective like me? Does having a more productive mentor improve the productivity of mentees?," Labour Economics, Elsevier, vol. 63(C).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tpr:edfpol:v:12:y:2017:i:2:p:197-223. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Kelly McDougall (email available below). General contact details of provider: https://direct.mit.edu/journals .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.