IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v40y2016i5p383-409.html
   My bibliography  Save this article

Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes

Author

Listed:
  • Geoffrey Phelps
  • Benjamin Kelcey
  • Nathan Jones
  • Shuangshuang Liu

Abstract

Mathematics professional development is widely offered, typically with the goal of improving teachers’ content knowledge, the quality of teaching, and ultimately students’ achievement. Recently, new assessments focused on mathematical knowledge for teaching (MKT) have been developed to assist in the evaluation and improvement of mathematics professional development. This study presents empirical estimates of average program change in MKT and its variation with the goal of supporting the design of experimental trials that are adequately powered to detect a specified program effect. The study drew on a large database representing five different assessments of MKT and collectively 326 professional development programs and 9,365 teachers. Results from cross-classified hierarchical growth models found that standardized average change estimates across the five assessments ranged from a low of 0.16 standard deviations ( SD s) to a high of 0.26 SD s. Power analyses using the estimated pre- and posttest change estimates indicated that hundreds of teachers are needed to detect changes in knowledge at the lower end of the distribution. Even studies powered to detect effects at the higher end of the distribution will require substantial resources to conduct rigorous experimental trials. Empirical benchmarks that describe average program change and its variation provide a useful preliminary resource for interpreting the relative magnitude of effect sizes associated with professional development programs and for designing adequately powered trials.

Suggested Citation

  • Geoffrey Phelps & Benjamin Kelcey & Nathan Jones & Shuangshuang Liu, 2016. "Informing Estimates of Program Effects for Studies of Mathematics Professional Development Using Teacher Content Knowledge Outcomes," Evaluation Review, , vol. 40(5), pages 383-409, October.
    • Handle: RePEc:sae:evarev:v:40:y:2016:i:5:p:383-409
    DOI: 10.1177/0193841X16665024
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X16665024
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X16665024?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Peter Z. Schochet, "undated". "Statistical Power for Random Assignment Evaluations of Education Programs," Mathematica Policy Research Reports 6749d31ad72d4acf988f7dce5, Mathematica Policy Research.
    2. repec:mpr:mprres:5863 is not listed on IDEAS
    3. Jonah E. Rockoff & Brian A. Jacob & Thomas J. Kane & Douglas O. Staiger, 2011. "Can You Recognize an Effective Teacher When You Recruit One?," Education Finance and Policy, MIT Press, vol. 6(1), pages 43-74, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Peter Z. Schochet & Hanley S. Chiang, "undated". "Error Rates in Measuring Teacher and School Performance Based on Student Test Score Gains," Mathematica Policy Research Reports d415285f980b4d64b7e75f40b, Mathematica Policy Research.
    2. repec:mpr:mprres:6721 is not listed on IDEAS
    3. Matthew A. Kraft & John P. Papay & Olivia L. Chi, 2020. "Teacher Skill Development: Evidence from Performance Ratings by Principals," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 39(2), pages 315-347, March.
    4. Papay, John P. & Kraft, Matthew A., 2015. "Productivity returns to experience in the teacher labor market: Methodological challenges and new evidence on long-term career improvement," Journal of Public Economics, Elsevier, vol. 130(C), pages 105-119.
    5. Seth Gershenson, 2016. "Performance Standards and Employee Effort: Evidence From Teacher Absences," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 35(3), pages 615-638, June.
    6. Catherine Haeck & Pierre Lefebvre, 2020. "The Evolution of Cognitive Skills Inequalities by Socioeconomic Status across Canada," Working Papers 20-04, Research Group on Human Capital, University of Quebec in Montreal's School of Management.
    7. World Bank, 2017. "Pre-Primary Education in Mongolia," World Bank Publications - Reports 26402, The World Bank Group.
    8. NISHIHATA Masaya & TAHARA Hidenori & KOBAYASHI Yohei, 2023. "Does Sending Teachers Abroad Enhance Their Quality and Ability?," Discussion papers 23055, Research Institute of Economy, Trade and Industry (RIETI).
    9. Vegas, E & Ganimian, A. J., 2013. "Theory and Evidence on Teacher Policies in Developed and Developing Countries," Working Paper 104291, Harvard University OpenScholar.
    10. Will Dobbie & Roland G. Fryer Jr., 2013. "Getting beneath the Veil of Effective Schools: Evidence from New York City," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 28-60, October.
    11. Jan Bietenbeck & Marc Piopiunik & Simon Wiederhold, 2018. "Africa’s Skill Tragedy: Does Teachers’ Lack of Knowledge Lead to Low Student Performance?," Journal of Human Resources, University of Wisconsin Press, vol. 53(3), pages 553-578.
    12. repec:mpr:mprres:6286 is not listed on IDEAS
    13. Bohdana Kurylo, 2021. "The Impact of Same-Race Teachers on Student Behavioral Outcomes," CERGE-EI Working Papers wp695, The Center for Economic Research and Graduate Education - Economics Institute, Prague.
    14. Goldhaber, Dan & Grout, Cyrus & Wolff, Malcolm & Martinková, Patrícia, 2021. "Evidence on the Dimensionality and Reliability of Professional References’ Ratings of Teacher Applicants," Economics of Education Review, Elsevier, vol. 83(C).
    15. C. Kirabo Jackson, 2013. "Match Quality, Worker Productivity, and Worker Mobility: Direct Evidence from Teachers," The Review of Economics and Statistics, MIT Press, vol. 95(4), pages 1096-1116, October.
    16. Martin Schlotter, 2011. "Age at Preschool Entrance and Noncognitive Skills before School - An Instrumental Variable Approach," ifo Working Paper Series 112, ifo Institute - Leibniz Institute for Economic Research at the University of Munich.
    17. Kenneth Fortson & Natalya Verbitsky-Savitz & Emma Kopa & Philip Gleason, 2012. "Using an Experimental Evaluation of Charter Schools to Test Whether Nonexperimental Comparison Group Methods Can Replicate Experimental Impact Estimates," Mathematica Policy Research Reports 27f871b5b7b94f3a80278a593, Mathematica Policy Research.
    18. Deborah Peikes & Stacy Dale & Eric Lundquist & Janice Genevro & David Meyers, 2011. "Building the Evidence Base for the Medical Home: What Sample and Sample Size Do Studies Need?," Mathematica Policy Research Reports 5814eb8219b24982af7f7536c, Mathematica Policy Research.
    19. Jones, Michael D., 2013. "Teacher behavior under performance pay incentives," Economics of Education Review, Elsevier, vol. 37(C), pages 148-164.
    20. Pierre Lefebvre & Philip Merrigan, 2020. "Les inégalités provinciales aux tests internationaux-nationaux de littéracie : Québec, Ontario et autres provinces canadiennes 1993-2018 (Version révisée et augmentée octobre 2020)," CIRANO Working Papers 2020s-29, CIRANO.
    21. Andrew P. Jaciw & Li Lin & Boya Ma, 2016. "An Empirical Study of Design Parameters for Assessing Differential Impacts for Students in Group Randomized Trials," Evaluation Review, , vol. 40(5), pages 410-443, October.
    22. M. Caridad Araujo & Pedro Carneiro & Yyannú Cruz-Aguayo & Norbert Schady, 2016. "Teacher Quality and Learning Outcomes in Kindergarten," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(3), pages 1415-1453.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:40:y:2016:i:5:p:383-409. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.