IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v44y2019i6p648-670.html
   My bibliography  Save this article

Research on Psychometric Modeling, Analysis, and Reporting of the National Assessment of Educational Progress

Author

Listed:
  • Andreas Oranje

    (Educational Testing Service)

  • Andrew Kolstad

    (P20 Strategies, LLC)

Abstract

The design and psychometric methodology of the National Assessment of Educational Progress (NAEP) is constantly evolving to meet the changing interests and demands stemming from a rapidly shifting educational landscape. NAEP has been built on strong research foundations that include conducting extensive evaluations and comparisons before new approaches are adopted. During those evaluations, many lessons are learned and discoveries surface that do not often find their way into widely accessible outlets. This article discusses a number of those insights with the goal to provide an integrated and accessible perspective on the strengths and limitations of NAEP’s psychometric methodology and statistical reporting practices. Drawing from a range of technical reports and memoranda, presentations, and published literature, the following topics are covered: calibration, estimation of proficiency, data reduction, standard error estimation, statistical inference, and standard setting.

Suggested Citation

  • Andreas Oranje & Andrew Kolstad, 2019. "Research on Psychometric Modeling, Analysis, and Reporting of the National Assessment of Educational Progress," Journal of Educational and Behavioral Statistics, , vol. 44(6), pages 648-670, December.
    • Handle: RePEc:sae:jedbes:v:44:y:2019:i:6:p:648-670
    DOI: 10.3102/1076998619867105
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/1076998619867105
    Download Restriction: no
    File URL: https://libkey.io/10.3102/1076998619867105?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Robert Gibbons & Donald Hedeker, 1992. "Full-information item bi-factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 57(3), pages 423-436, September.
    2. Johnson, Matthew S., 2007. "Marginal Maximum Likelihood Estimation of Item Response Models in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 20(i10).
    3. Neal Thomas, 2002. "The role of secondary covariates when estimating latent trait population distributions," Psychometrika, Springer;The Psychometric Society, vol. 67(1), pages 33-48, March.
    4. Robert Mislevy, 1991. "Randomization-based inference about latent variables from complex samples," Psychometrika, Springer;The Psychometric Society, vol. 56(2), pages 177-196, June.
    5. Deping Li & Andreas Oranje & Yanlin Jiang, 2009. "On the Estimation of Hierarchical Latent Regression Models for Large-Scale Assessments," Journal of Educational and Behavioral Statistics, , vol. 34(4), pages 433-463, December.
    6. Robert Mislevy, 1984. "Estimating latent distributions," Psychometrika, Springer;The Psychometric Society, vol. 49(3), pages 359-381, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Minjeong Jeon & Sophia Rabe-Hesketh, 2012. "Profile-Likelihood Approach for Estimating Generalized Linear Mixed Models With Factor Structures," Journal of Educational and Behavioral Statistics, , vol. 37(4), pages 518-542, August.
    2. Yan Huo & Jimmy de la Torre & Eun-Young Mun & Su-Young Kim & Anne Ray & Yang Jiao & Helene White, 2015. "A Hierarchical Multi-Unidimensional IRT Approach for Analyzing Sparse, Multi-Group Data for Integrative Data Analysis," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 834-855, September.
    3. Simon Grund & Oliver Lüdtke & Alexander Robitzsch, 2021. "On the Treatment of Missing Data in Background Questionnaires in Educational Large-Scale Assessments: An Evaluation of Different Procedures," Journal of Educational and Behavioral Statistics, , vol. 46(4), pages 430-465, August.
    4. Brian Junker & Lynne Schofield & Lowell Taylor, 2012. "The use of cognitive ability measures as explanatory variables in regression analysis," IZA Journal of Labor Economics, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 1(1), pages 1-19, December.
    5. Matthias von Davier & Sandip Sinharay, 2010. "Stochastic Approximation Methods for Latent Regression Item Response Models," Journal of Educational and Behavioral Statistics, , vol. 35(2), pages 174-193, April.
    6. Lynne Schofield & Brian Junker & Lowell Taylor & Dan Black, 2015. "Predictive Inference Using Latent Variables with Covariates," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 727-747, September.
    7. Scott Monroe, 2021. "Testing Latent Variable Distribution Fit in IRT Using Posterior Residuals," Journal of Educational and Behavioral Statistics, , vol. 46(3), pages 374-398, June.
    8. repec:jss:jstsof:34:i03 is not listed on IDEAS
    9. Matthias von Davier & Lale Khorramdel & Qiwei He & Hyo Jeong Shin & Haiwen Chen, 2019. "Developments in Psychometric Population Models for Technology-Based Large-Scale Assessments: An Overview of Challenges and Opportunities," Journal of Educational and Behavioral Statistics, , vol. 44(6), pages 671-705, December.
    10. Li Cai & Carrie R. Houts, 2021. "Longitudinal Analysis of Patient-Reported Outcomes in Clinical Trials: Applications of Multilevel and Multidimensional Item Response Theory," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 754-777, September.
    11. Melissa Gladstone & Gillian Lancaster & Gareth McCray & Vanessa Cavallera & Claudia R. L. Alves & Limbika Maliwichi & Muneera A. Rasheed & Tarun Dua & Magdalena Janus & Patricia Kariger, 2021. "Validation of the Infant and Young Child Development (IYCD) Indicators in Three Countries: Brazil, Malawi and Pakistan," IJERPH, MDPI, vol. 18(11), pages 1-19, June.
    12. Aristidis Nikoloulopoulos & Harry Joe, 2015. "Factor Copula Models for Item Response Data," Psychometrika, Springer;The Psychometric Society, vol. 80(1), pages 126-150, March.
    13. Victoria T. Tanaka & George Engelhard & Matthew P. Rabbitt, 2020. "Using a Bifactor Model to Measure Food Insecurity in Households with Children," Journal of Family and Economic Issues, Springer, vol. 41(3), pages 492-504, September.
    14. Chun Wang & Steven W. Nydick, 2020. "On Longitudinal Item Response Theory Models: A Didactic," Journal of Educational and Behavioral Statistics, , vol. 45(3), pages 339-368, June.
    15. C. Glas & Anna Dagohoy, 2007. "A Person Fit Test For Irt Models For Polytomous Items," Psychometrika, Springer;The Psychometric Society, vol. 72(2), pages 159-180, June.
    16. Michael Edwards, 2010. "A Markov Chain Monte Carlo Approach to Confirmatory Item Factor Analysis," Psychometrika, Springer;The Psychometric Society, vol. 75(3), pages 474-497, September.
    17. Lire Ersado & Jérémie Gignoux, 2017. "Egypt: inequality of opportunity in education," Middle East Development Journal, Taylor & Francis Journals, vol. 9(1), pages 22-54, January.
    18. Yajuan Si & Jerome P. Reiter, 2013. "Nonparametric Bayesian Multiple Imputation for Incomplete Categorical Variables in Large-Scale Assessment Surveys," Journal of Educational and Behavioral Statistics, , vol. 38(5), pages 499-521, October.
    19. Robitzsch, Alexander, 2020. "About Still Nonignorable Consequences of (Partially) Ignoring Missing Item Responses in Large-scale Assessment," OSF Preprints hmy45, Center for Open Science.
    20. Lire Ersado & Jérémie Gignoux, 2014. "Inequality of Educational Opportunities in Egypt," PSE Working Papers halshs-01064510, HAL.
    21. Laine Bradshaw & Jonathan Templin, 2014. "Combining Item Response Theory and Diagnostic Classification Models: A Psychometric Model for Scaling Ability and Diagnosing Misconceptions," Psychometrika, Springer;The Psychometric Society, vol. 79(3), pages 403-425, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:44:y:2019:i:6:p:648-670. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.