IDEAS home Printed from https://ideas.repec.org/a/eee/ecoedu/v83y2021ics0272775721000492.html
   My bibliography  Save this article

Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants

Author

Listed:
  • Goldhaber, Dan
  • Grout, Cyrus
  • Wolff, Malcolm
  • Martinková, Patrícia

Abstract

There is growing interest in using measures of teacher applicant quality to improve hiring decisions, but the statistical properties of such measures are not well understood. We use unique data on structured ratings solicited from the references of teacher applicants to explore the dimensionality of measures of teacher applicant quality and the inter-rater reliability of the reference ratings. Despite questions about applicants designed to capture multiple dimensions of quality, factor analysis suggests that the reference ratings only capture one underlying dimension. Point estimates of inter-rater reliability range between 0.23 and 0.31 and are significantly lower for novice applicants. It is difficult to judge whether these levels of reliability are high or low in the current context given so little evidence on applicant assessment tools.

Suggested Citation

  • Goldhaber, Dan & Grout, Cyrus & Wolff, Malcolm & Martinková, Patrícia, 2021. "Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants," Economics of Education Review, Elsevier, vol. 83(C).
    • Handle: RePEc:eee:ecoedu:v:83:y:2021:i:c:s0272775721000492
    DOI: 10.1016/j.econedurev.2021.102130
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0272775721000492
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.econedurev.2021.102130?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Bates, Douglas & Mächler, Martin & Bolker, Ben & Walker, Steve, 2015. "Fitting Linear Mixed-Effects Models Using lme4," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 67(i01).
    2. Dan Goldhaber & Cyrus Grout & Nick Huntington-Klein, 2017. "Screen Twice, Cut Once: Assessing the Predictive Validity of Applicant Selection Tools," Education Finance and Policy, MIT Press, vol. 12(2), pages 197-223, Spring.
    3. Jonah E. Rockoff & Brian A. Jacob & Thomas J. Kane & Douglas O. Staiger, 2011. "Can You Recognize an Effective Teacher When You Recruit One?," Education Finance and Policy, MIT Press, vol. 6(1), pages 43-74, January.
    4. Jacob, Brian A. & Rockoff, Jonah E. & Taylor, Eric S. & Lindy, Benjamin & Rosen, Rachel, 2018. "Teacher applicant hiring and teacher performance: Evidence from DC public schools," Journal of Public Economics, Elsevier, vol. 166(C), pages 81-97.
    5. Patrícia Martinková & Dan Goldhaber & Elena Erosheva, 2018. "Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-17, October.
    6. Matthew A. Kraft, 2019. "Teacher Effects on Complex Cognitive Skills and Social-Emotional Competencies," Journal of Human Resources, University of Wisconsin Press, vol. 54(1), pages 1-36.
    7. Brian A. Jacob & Lars Lefgren, 2005. "Principals as Agents: Subjective Performance Measurement in Education," NBER Working Papers 11463, National Bureau of Economic Research, Inc.
    8. Douglas N. Harris & Stacey A. Rutledge & William K. Ingle & Cynthia C. Thompson, 2010. "Mix and Match: What Principals Really Look for When Hiring Teachers," Education Finance and Policy, MIT Press, vol. 5(2), pages 228-246, April.
    9. Jacob, Brian A. & Lefgren, Lars, 2005. "Principals as Agents: Subjective Performance Measures in Education," Working Paper Series rwp05-040, Harvard University, John F. Kennedy School of Government.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Patrícia Martinková & František Bartoš & Marek Brabec, 2023. "Assessing Inter-rater Reliability With Heterogeneous Variance Components Models: Flexible Approach Accounting for Contextual Variables," Journal of Educational and Behavioral Statistics, , vol. 48(3), pages 349-383, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jessalynn James & Matthew A. Kraft & John P. Papay, 2023. "Local supply, temporal dynamics, and unrealized potential in teacher hiring," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 42(4), pages 1010-1044, September.
    2. Vegas, E & Ganimian, A. J., 2013. "Theory and Evidence on Teacher Policies in Developed and Developing Countries," Working Paper 104291, Harvard University OpenScholar.
    3. Thijssen, Maximiliaan W.P. & Rege, Mari & Solheim, Oddny J., 2022. "Teacher relationship skills and student learning," Economics of Education Review, Elsevier, vol. 89(C).
    4. Matthew A. Kraft & John P. Papay & Olivia L. Chi, 2020. "Teacher Skill Development: Evidence from Performance Ratings by Principals," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 315-347, March.
    5. Jenifer Ruiz-Valenzuela & Camille Terrier & Clémentine Van Effenterre, 2017. "Effectiveness of CEOs in the public sector: evidence from further education institutions," CVER Briefing Notes 005, Centre for Vocational Education Research.
    6. NISHIHATA Masaya & TAHARA Hidenori & KOBAYASHI Yohei, 2023. "Does Sending Teachers Abroad Enhance Their Quality and Ability?," Discussion papers 23055, Research Institute of Economy, Trade and Industry (RIETI).
    7. Patrícia Martinková & Dan Goldhaber & Elena Erosheva, 2018. "Disparities in ratings of internal and external applicants: A case for model-based inter-rater reliability," PLOS ONE, Public Library of Science, vol. 13(10), pages 1-17, October.
    8. Burgess, Simon & Greaves, Ellen & Murphy, Richard, 2022. "Deregulating Teacher Labor Markets," Economics of Education Review, Elsevier, vol. 88(C).
    9. Brutti, Zelda & Sánchez Torres, Fabio, 2022. "Turning around teacher quality in Latin America: Renewed confidence and lessons from Colombia," Economic Analysis and Policy, Elsevier, vol. 73(C), pages 62-93.
    10. Victor Lavy, 2011. "What Makes an Effective Teacher? Quasi-Experimental Evidence," NBER Working Papers 16885, National Bureau of Economic Research, Inc.
    11. Victor Lavy & Rigissa Megalokonomou, 2019. "Persistency in Teachers’ Grading Bias and Effects on Longer-Term Outcomes: University Admissions Exams and Choice of Field of Study," NBER Working Papers 26021, National Bureau of Economic Research, Inc.
    12. Eric Parsons, 2014. "Does Attending a Low-Achieving School Affect High-Performing Student Outcomes?," Working Papers 1407, Department of Economics, University of Missouri, revised 18 Feb 2015.
    13. Naven, Matthew, 2019. "Human-Capital Formation During Childhood and Adolescence: Evidence from School Quality and Postsecondary Success in California," MPRA Paper 97716, University Library of Munich, Germany.
    14. Johnston, Andrew C., 2021. "Preferences, Selection, and the Structure of Teacher Pay," IZA Discussion Papers 14831, Institute of Labor Economics (IZA).
    15. Maria De Paola, 2009. "Does Teacher Quality Affect Student Performance? Evidence From An Italian University," Bulletin of Economic Research, Wiley Blackwell, vol. 61(4), pages 353-377, October.
    16. Katina Pollock & Fei Wang & David Cameron Hauseman, 2015. "Complexity and Volume: An Inquiry into Factors that Drive Principals’ Work," Societies, MDPI, vol. 5(2), pages 1-29, June.
    17. David Blazar, 2018. "Validating Teacher Effects on Students’ Attitudes and Behaviors: Evidence from Random Assignment of Teachers to Students," Education Finance and Policy, MIT Press, vol. 13(3), pages 281-309, Summer.
    18. Terrier, Camille, 2020. "Boys lag behind: How teachers’ gender biases affect student achievement," Economics of Education Review, Elsevier, vol. 77(C).
    19. Jesse Rothstein, 2007. "Do Value-Added Models Add Value? Tracking, Fixed Effects, and Causal Inference," Working Papers 1036, Princeton University, Department of Economics, Center for Economic Policy Studies..
    20. Grönqvist, Erik & Hensvik, Lena & Thoresson, Anna, 2022. "Teacher career opportunities and school quality," Labour Economics, Elsevier, vol. 77(C).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:ecoedu:v:83:y:2021:i:c:s0272775721000492. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/econedurev .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.