IDEAS home Printed from https://ideas.repec.org/a/eee/ecoedu/v83y2021ics0272775721000492.html
   My bibliography  Save this article

Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants

Author

Listed:
  • Goldhaber, Dan
  • Grout, Cyrus
  • Wolff, Malcolm
  • Martinková, Patrícia

Abstract

There is growing interest in using measures of teacher applicant quality to improve hiring decisions, but the statistical properties of such measures are not well understood. We use unique data on structured ratings solicited from the references of teacher applicants to explore the dimensionality of measures of teacher applicant quality and the inter-rater reliability of the reference ratings. Despite questions about applicants designed to capture multiple dimensions of quality, factor analysis suggests that the reference ratings only capture one underlying dimension. Point estimates of inter-rater reliability range between 0.23 and 0.31 and are significantly lower for novice applicants. It is difficult to judge whether these levels of reliability are high or low in the current context given so little evidence on applicant assessment tools.

Suggested Citation

  • Goldhaber, Dan & Grout, Cyrus & Wolff, Malcolm & Martinková, Patrícia, 2021. "Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants," Economics of Education Review, Elsevier, vol. 83(C).
    • Handle: RePEc:eee:ecoedu:v:83:y:2021:i:c:s0272775721000492
    DOI: 10.1016/j.econedurev.2021.102130
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0272775721000492
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.econedurev.2021.102130?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bates, Douglas & Mächler, Martin & Bolker, Ben & Walker, Steve, 2015. "Fitting Linear Mixed-Effects Models Using lme4," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 67(i01).
    2. Dan Goldhaber & Cyrus Grout & Nick Huntington-Klein, 2017. "Screen Twice, Cut Once: Assessing the Predictive Validity of Applicant Selection Tools," Education Finance and Policy, MIT Press, vol. 12(2), pages 197-223, Spring.
    3. Jonah E. Rockoff & Brian A. Jacob & Thomas J. Kane & Douglas O. Staiger, 2011. "Can You Recognize an Effective Teacher When You Recruit One?," Education Finance and Policy, MIT Press, vol. 6(1), pages 43-74, January.
    4. Jacob, Brian A. & Rockoff, Jonah E. & Taylor, Eric S. & Lindy, Benjamin & Rosen, Rachel, 2018. "Teacher applicant hiring and teacher performance: Evidence from DC public schools," Journal of Public Economics, Elsevier, vol. 166(C), pages 81-97.
    5. Patrícia Martinková & Dan Goldhaber & Elena Erosheva, 2018. "Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-17, October.
    6. Matthew A. Kraft, 2019. "Teacher Effects on Complex Cognitive Skills and Social-Emotional Competencies," Journal of Human Resources, University of Wisconsin Press, vol. 54(1), pages 1-36.
    7. Brian A. Jacob & Lars Lefgren, 2005. "Principals as Agents: Subjective Performance Measurement in Education," NBER Working Papers 11463, National Bureau of Economic Research, Inc.
    8. Douglas N. Harris & Stacey A. Rutledge & William K. Ingle & Cynthia C. Thompson, 2010. "Mix and Match: What Principals Really Look for When Hiring Teachers," Education Finance and Policy, MIT Press, vol. 5(2), pages 228-246, April.
    9. Jacob, Brian A. & Lefgren, Lars, 2005. "Principals as Agents: Subjective Performance Measures in Education," Working Paper Series rwp05-040, Harvard University, John F. Kennedy School of Government.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Patrícia Martinková & František Bartoš & Marek Brabec, 2023. "Assessing Inter-rater Reliability With Heterogeneous Variance Components Models: Flexible Approach Accounting for Contextual Variables," Journal of Educational and Behavioral Statistics, , vol. 48(3), pages 349-383, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jessalynn James & Matthew A. Kraft & John P. Papay, 2023. "Local supply, temporal dynamics, and unrealized potential in teacher hiring," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 42(4), pages 1010-1044, September.
    2. Vegas, E & Ganimian, A. J., 2013. "Theory and Evidence on Teacher Policies in Developed and Developing Countries," Working Paper 104291, Harvard University OpenScholar.
    3. Thijssen, Maximiliaan W.P. & Rege, Mari & Solheim, Oddny J., 2022. "Teacher relationship skills and student learning," Economics of Education Review, Elsevier, vol. 89(C).
    4. Matthew A. Kraft & John P. Papay & Olivia L. Chi, 2020. "Teacher Skill Development: Evidence from Performance Ratings by Principals," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 315-347, March.
    5. NISHIHATA Masaya & TAHARA Hidenori & KOBAYASHI Yohei, 2023. "Does Sending Teachers Abroad Enhance Their Quality and Ability?," Discussion papers 23055, Research Institute of Economy, Trade and Industry (RIETI).
    6. Eric Parsons, 2014. "Does Attending a Low-Achieving School Affect High-Performing Student Outcomes?," Working Papers 1407, Department of Economics, University of Missouri, revised 18 Feb 2015.
    7. Naven, Matthew, 2019. "Human-Capital Formation During Childhood and Adolescence: Evidence from School Quality and Postsecondary Success in California," MPRA Paper 97716, University Library of Munich, Germany.
    8. Johnston, Andrew C., 2021. "Preferences, Selection, and the Structure of Teacher Pay," IZA Discussion Papers 14831, Institute of Labor Economics (IZA).
    9. Maria De Paola, 2009. "Does Teacher Quality Affect Student Performance? Evidence From An Italian University," Bulletin of Economic Research, Wiley Blackwell, vol. 61(4), pages 353-377, October.
    10. Katina Pollock & Fei Wang & David Cameron Hauseman, 2015. "Complexity and Volume: An Inquiry into Factors that Drive Principals’ Work," Societies, MDPI, vol. 5(2), pages 1-29, June.
    11. Grönqvist, Erik & Hensvik, Lena & Thoresson, Anna, 2022. "Teacher career opportunities and school quality," Labour Economics, Elsevier, vol. 77(C).
    12. Terrier, Camille, 2020. "Boys lag behind: How teachers’ gender biases affect student achievement," Economics of Education Review, Elsevier, vol. 77(C).
    13. Dan Goldhaber & Cyrus Grout & Nick Huntington-Klein, 2017. "Screen Twice, Cut Once: Assessing the Predictive Validity of Applicant Selection Tools," Education Finance and Policy, MIT Press, vol. 12(2), pages 197-223, Spring.
    14. Hinrichs, Peter, 2021. "What kind of teachers are schools looking for? Evidence from a randomized field experiment," Journal of Economic Behavior & Organization, Elsevier, vol. 186(C), pages 395-411.
    15. Gershenson, Seth, 2021. "Identifying and Producing Effective Teachers," IZA Discussion Papers 14096, Institute of Labor Economics (IZA).
    16. Burgess, Simon & Greaves, Ellen & Murphy, Richard, 2022. "Deregulating Teacher Labor Markets," Economics of Education Review, Elsevier, vol. 88(C).
    17. Jacob, Brian A. & Rockoff, Jonah E. & Taylor, Eric S. & Lindy, Benjamin & Rosen, Rachel, 2018. "Teacher applicant hiring and teacher performance: Evidence from DC public schools," Journal of Public Economics, Elsevier, vol. 166(C), pages 81-97.
    18. Cassandra M. Guarino & Michelle Maxfield & Mark D. Reckase & Paul N. Thompson & Jeffrey M. Wooldridge, 2015. "An Evaluation of Empirical Bayes’s Estimation of Value-Added Teacher Performance Measures," Journal of Educational and Behavioral Statistics, , vol. 40(2), pages 190-222, April.
    19. Jason A. Grissom & Susanna Loeb, 2017. "Assessing Principals’ Assessments: Subjective Evaluations of Teacher Effectiveness in Low- and High-Stakes Environments," Education Finance and Policy, MIT Press, vol. 12(3), pages 369-395, Summer.
    20. W. Bentley MacLeod & Miguel Urquiola, 2018. "Is Education Consumption or Investment? Implications for the Effect of School Competition," NBER Working Papers 25117, National Bureau of Economic Research, Inc.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecoedu:v:83:y:2021:i:c:s0272775721000492. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/econedurev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.