IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v40y2016i5p383-409.html
   My bibliography  Save this article

Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes

Author

Listed:
  • Geoffrey Phelps
  • Benjamin Kelcey
  • Nathan Jones
  • Shuangshuang Liu

Abstract

Mathematics professional development is widely offered, typically with the goal of improving teachers’ content knowledge, the quality of teaching, and ultimately students’ achievement. Recently, new assessments focused on mathematical knowledge for teaching (MKT) have been developed to assist in the evaluation and improvement of mathematics professional development. This study presents empirical estimates of average program change in MKT and its variation with the goal of supporting the design of experimental trials that are adequately powered to detect a specified program effect. The study drew on a large database representing five different assessments of MKT and collectively 326 professional development programs and 9,365 teachers. Results from cross-classified hierarchical growth models found that standardized average change estimates across the five assessments ranged from a low of 0.16 standard deviations ( SD s) to a high of 0.26 SD s. Power analyses using the estimated pre- and posttest change estimates indicated that hundreds of teachers are needed to detect changes in knowledge at the lower end of the distribution. Even studies powered to detect effects at the higher end of the distribution will require substantial resources to conduct rigorous experimental trials. Empirical benchmarks that describe average program change and its variation provide a useful preliminary resource for interpreting the relative magnitude of effect sizes associated with professional development programs and for designing adequately powered trials.

Suggested Citation

  • Geoffrey Phelps & Benjamin Kelcey & Nathan Jones & Shuangshuang Liu, 2016. "Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes," Evaluation Review, , vol. 40(5), pages 383-409, October.
    • Handle: RePEc:sae:evarev:v:40:y:2016:i:5:p:383-409
    DOI: 10.1177/0193841X16665024
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X16665024
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X16665024?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. repec:mpr:mprres:5863 is not listed on IDEAS
    2. Jonah E. Rockoff & Brian A. Jacob & Thomas J. Kane & Douglas O. Staiger, 2011. "Can You Recognize an Effective Teacher When You Recruit One?," Education Finance and Policy, MIT Press, vol. 6(1), pages 43-74, January.
    3. Peter Z. Schochet, "undated". "Statistical Power for Random Assignment Evaluations of Education Programs," Mathematica Policy Research Reports 6749d31ad72d4acf988f7dce5, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. repec:mpr:mprres:6721 is not listed on IDEAS
    2. Peter Z. Schochet & Hanley S. Chiang, "undated". "Error Rates in Measuring Teacher and School Performance Based on Student Test Score Gains," Mathematica Policy Research Reports d415285f980b4d64b7e75f40b, Mathematica Policy Research.
    3. Matthew A. Kraft & John P. Papay & Olivia L. Chi, 2020. "Teacher Skill Development: Evidence from Performance Ratings by Principals," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 315-347, March.
    4. Jesse Rothstein, 2015. "Teacher Quality Policy When Supply Matters," American Economic Review, American Economic Association, vol. 105(1), pages 100-130, January.
    5. Papay, John P. & Kraft, Matthew A., 2015. "Productivity returns to experience in the teacher labor market: Methodological challenges and new evidence on long-term career improvement," Journal of Public Economics, Elsevier, vol. 130(C), pages 105-119.
    6. Seth Gershenson, 2016. "Performance Standards and Employee Effort: Evidence From Teacher Absences," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 35(3), pages 615-638, June.
    7. Catherine Haeck & Pierre Lefebvre, 2020. "The Evolution of Cognitive Skills Inequalities by Socioeconomic Status across Canada," Working Papers 20-04, Research Group on Human Capital, University of Quebec in Montreal's School of Management.
    8. World Bank, 2017. "Pre-Primary Education in Mongolia," World Bank Publications - Reports 26402, The World Bank Group.
    9. NISHIHATA Masaya & TAHARA Hidenori & KOBAYASHI Yohei, 2023. "Does Sending Teachers Abroad Enhance Their Quality and Ability?," Discussion papers 23055, Research Institute of Economy, Trade and Industry (RIETI).
    10. Roland G. Fryer, 2013. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," Journal of Labor Economics, University of Chicago Press, vol. 31(2), pages 373-407.
    11. Busso, Matias & Montaño, Sebastián & Muñoz-Morales, Juan & Pope, Nolan G., 2024. "The unintended consequences of merit-based teacher selection: Evidence from a large-scale reform in Colombia," Journal of Public Economics, Elsevier, vol. 239(C).
    12. Vegas, E & Ganimian, A. J., 2013. "Theory and Evidence on Teacher Policies in Developed and Developing Countries," Working Paper 104291, Harvard University OpenScholar.
    13. Will Dobbie & Roland G. Fryer Jr., 2013. "Getting beneath the Veil of Effective Schools: Evidence from New York City," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 28-60, October.
    14. Jan Bietenbeck & Marc Piopiunik & Simon Wiederhold, 2018. "Africa’s Skill Tragedy: Does Teachers’ Lack of Knowledge Lead to Low Student Performance?," Journal of Human Resources, University of Wisconsin Press, vol. 53(3), pages 553-578.
    15. repec:mpr:mprres:6286 is not listed on IDEAS
    16. Elizabeth Tipton & Robert B. Olsen, "undated". "Enhancing the Generalizability of Impact Studies in Education," Mathematica Policy Research Reports 35d5625333dc480aba9765b3b, Mathematica Policy Research.
    17. Bohdana Kurylo, 2021. "The Impact of Same-Race Teachers on Student Behavioral Outcomes," CERGE-EI Working Papers wp695, The Center for Economic Research and Graduate Education - Economics Institute, Prague.
    18. Goldhaber, Dan & Grout, Cyrus & Wolff, Malcolm & Martinková, Patrícia, 2021. "Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants," Economics of Education Review, Elsevier, vol. 83(C).
    19. Dhushyanth Raju, 2017. "Public School Teacher Management in Sri Lanka," South Asia Economic Journal, Institute of Policy Studies of Sri Lanka, vol. 18(1), pages 39-63, March.
    20. Robin Jacob & Marie-Andree Somers & Pei Zhu & Howard Bloom, 2016. "The Validity of the Comparative Interrupted Time Series Design for Evaluating the Effect of School-Level Interventions," Evaluation Review, , vol. 40(3), pages 167-198, June.
    21. Hanley S. Chiang & Melissa A. Clark & Sheena McConnell, "undated". "Supplying Disadvantaged Schools with Effective Teachers: Experimental Evidence on Secondary Math Teachers from Teach For America," Mathematica Policy Research Reports 4623b3b72e154403be21e3878, Mathematica Policy Research.
    22. Sean Tanner & Jenna Terrell & Emily Vislosky & Jonathan Gellar & Brian Gill, "undated". "Predicting Early Fall Student Enrollment in the School District of Philadelphia," Mathematica Policy Research Reports 63a18bf538bd41f98d72ff91d, Mathematica Policy Research.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:40:y:2016:i:5:p:383-409. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.