IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v78y2013i3p557-575.html
   My bibliography  Save this article

Monitoring Scale Scores over Time via Quality Control Charts, Model-Based Approaches, and Time Series Techniques

Author

Listed:
  • Yi-Hsuan Lee
  • Alina Davier

Abstract

Maintaining a stable score scale over time is critical for all standardized educational assessments. Traditional quality control tools and approaches for assessing scale drift either require special equating designs, or may be too time-consuming to be considered on a regular basis with an operational test that has a short time window between an administration and its score reporting. Thus, the traditional methods are not sufficient to catch unusual testing outcomes in a timely manner. This paper presents a new approach for score monitoring and assessment of scale drift. It involves quality control charts, model-based approaches, and time series techniques to accommodate the following needs of monitoring scale scores: continuous monitoring, adjustment of customary variations, identification of abrupt shifts, and assessment of autocorrelation. Performance of the methodologies is evaluated using manipulated data based on real responses from 71 administrations of a large-scale high-stakes language assessment. Copyright The Psychometric Society 2013

Suggested Citation

  • Yi-Hsuan Lee & Alina Davier, 2013. "Monitoring Scale Scores over Time via Quality Control Charts, Model-Based Approaches, and Time Series Techniques," Psychometrika, Springer;The Psychometric Society, vol. 78(3), pages 557-575, July.
    • Handle: RePEc:spr:psycho:v:78:y:2013:i:3:p:557-575
    DOI: 10.1007/s11336-013-9317-5
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1007/s11336-013-9317-5
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1007/s11336-013-9317-5?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Achim Zeileis, 2005. "A Unified Approach to Structural Change Tests Based on ML Scores, F Statistics, and OLS Residuals," Econometric Reviews, Taylor & Francis Journals, vol. 24(4), pages 445-466.
    2. Yi-Hsuan Lee & Shelby Haberman, 2013. "Harmonic Regression and Scale Stability," Psychometrika, Springer;The Psychometric Society, vol. 78(4), pages 815-829, October.
    3. Zeileis, Achim & Leisch, Friedrich & Hornik, Kurt & Kleiber, Christian, 2002. "strucchange: An R Package for Testing for Structural Change in Linear Regression Models," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 7(i02).
    4. Visser, Ingmar & Speekenbrink, Maarten, 2010. "depmixS4: An R Package for Hidden Markov Models," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 36(i07).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yi-Hsuan Lee & Shelby Haberman, 2013. "Harmonic Regression and Scale Stability," Psychometrika, Springer;The Psychometric Society, vol. 78(4), pages 815-829, October.
    2. Sandip Sinharay, 2017. "Some Remarks on Applications of Tests for Detecting A Change Point to Psychometric Problems," Psychometrika, Springer;The Psychometric Society, vol. 82(4), pages 1149-1161, December.
    3. Amirali Kani & Wayne S. DeSarbo & Duncan K. H. Fong, 2018. "A Factorial Hidden Markov Model for the Analysis of Temporal Change in Choice Models," Customer Needs and Solutions, Springer;Institute for Sustainable Innovation and Growth (iSIG), vol. 5(3), pages 162-177, December.
    4. Yunxiao Chen & Yi-Hsuan Lee & Xiaoou Li, 2022. "Item Pool Quality Control in Educational Testing: Change Point Model, Compound Risk, and Sequential Detection," Journal of Educational and Behavioral Statistics, , vol. 47(3), pages 322-352, June.
    5. Yi-Hsuan Lee & Charles Lewis, 2021. "Monitoring Item Performance With CUSUM Statistics in Continuous Testing," Journal of Educational and Behavioral Statistics, , vol. 46(5), pages 611-648, October.
    6. Björn Andersson & Alina Davier, 2015. "Book Review," Psychometrika, Springer;The Psychometric Society, vol. 80(3), pages 856-858, September.
    7. Chen, Yunxiao & Lee, Yi-Hsuan & Li, Xiaoou, 2022. "Item pool quality control in educational testing: change point model, compound risk, and sequential detection," LSE Research Online Documents on Economics 112498, London School of Economics and Political Science, LSE Library.
    8. Alina Davier, 2013. "Observed-Score Equating: An Overview," Psychometrika, Springer;The Psychometric Society, vol. 78(4), pages 605-623, October.
    9. Sandip Sinharay, 2016. "Person Fit Analysis in Computerized Adaptive Testing Using Tests for a Change Point," Journal of Educational and Behavioral Statistics, , vol. 41(5), pages 521-549, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Edgar Merkle & Achim Zeileis, 2013. "Tests of Measurement Invariance Without Subgroups: A Generalization of Classical Methods," Psychometrika, Springer;The Psychometric Society, vol. 78(1), pages 59-82, January.
    2. Jan Verbesselt & Achim Zeileis & Martin Herold, 2011. "Near Real-Time Disturbance Detection in Terrestrial Ecosystems Using Satellite Image Time Series: Drought Detection in Somalia," Working Papers 2011-18, Faculty of Economics and Statistics, Universität Innsbruck.
    3. Fantazzini, Dean & Shangina, Tamara, 2019. "The importance of being informed: forecasting market risk measures for the Russian RTS index future using online data and implied volatility over two decades," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 55, pages 5-31.
    4. Kleiber, Christian, 2016. "Structural Change in (Economic) Time Series," Working papers 2016/06, Faculty of Business and Economics - University of Basel.
    5. Piotr Kotlarz & Michael Hanke & Sebastian Stöckl, 2023. "Regime-dependent drivers of the EUR/CHF exchange rate," Swiss Journal of Economics and Statistics, Springer;Swiss Society of Economics and Statistics, vol. 159(1), pages 1-18, December.
    6. Thomas Windberger & Achim Zeileis, 2011. "Structural Breaks in Inflation Dynamics within the European Monetary Union," Working Papers 2011-12, Faculty of Economics and Statistics, Universität Innsbruck.
    7. Zeileis, Achim & Shah, Ajay & Patnaik, Ila, 2010. "Testing, monitoring, and dating structural changes in exchange rate regimes," Computational Statistics & Data Analysis, Elsevier, vol. 54(6), pages 1696-1706, June.
    8. Achim Zeileis & Torsten Hothorn, 2013. "A toolbox of permutation tests for structural change," Statistical Papers, Springer, vol. 54(4), pages 931-954, November.
    9. Yazgan, M. Ege & Özkan, Harun, 2015. "Detecting structural changes using wavelets," Finance Research Letters, Elsevier, vol. 12(C), pages 23-37.
    10. Dean Fantazzini, 2014. "Nowcasting and Forecasting the Monthly Food Stamps Data in the US Using Online Search Data," PLOS ONE, Public Library of Science, vol. 9(11), pages 1-27, November.
    11. Maria Lourdes Ordoñez Olivo & Zoltán Lakner, 2023. "Shaping the Knowledge Base of Bioeconomy Sectors Development in Latin American and Caribbean Countries: A Bibliometric Analysis," Sustainability, MDPI, vol. 15(6), pages 1-18, March.
    12. Magdalena Mikolajek-Gocejna, 2021. "Estimation, Instability, and Non-Stationarity of Beta Coefficients for Twenty-four Emerging Markets in 2005-2021," European Research Studies Journal, European Research Studies Journal, vol. 0(4), pages 370-395.
    13. Akaev, Askar & Devezas, Tessaleno & Ichkitidze, Yuri & Sarygulov, Askar, 2021. "Forecasting the labor intensity and labor income share for G7 countries in the digital age," Technological Forecasting and Social Change, Elsevier, vol. 167(C).
    14. Grün, Bettina & Kosmidis, Ioannis & Zeileis, Achim, 2012. "Extended Beta Regression in R: Shaken, Stirred, Mixed, and Partitioned," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i11).
    15. Du, Xiaodong & Yu, Cindy L. & Hayes, Dermot J., 2011. "Speculation and volatility spillover in the crude oil and agricultural commodity markets: A Bayesian analysis," Energy Economics, Elsevier, vol. 33(3), pages 497-503, May.
    16. Les Oxley & Chris Price & William Rea & Marco Reale, 2008. "A New Procedure to Test for H Self-Similarity," Working Papers in Economics 08/16, University of Canterbury, Department of Economics and Finance.
    17. Mensi, Walid & Lee, Yun-Jung & Al-Yahyaee, Khamis Hamed & Sensoy, Ahmet & Yoon, Seong-Min, 2019. "Intraday downward/upward multifractality and long memory in Bitcoin and Ethereum markets: An asymmetric multifractal detrended fluctuation analysis," Finance Research Letters, Elsevier, vol. 31(C), pages 19-25.
    18. Ruggieri, Eric & Antonellis, Marcus, 2016. "An exact approach to Bayesian sequential change point detection," Computational Statistics & Data Analysis, Elsevier, vol. 97(C), pages 71-86.
    19. Dionne, Georges & Saissi-Hassani, Samir, 2016. "Hidden Markov Regimes in Operational Loss Data: Application to the Recent Financial Crisis," Working Papers 15-3, HEC Montreal, Canada Research Chair in Risk Management.
    20. Patrik Nosil & Zachariah Gompert & Daniel J. Funk, 2024. "Divergent dynamics of sexual and habitat isolation at the transition between stick insect populations and species," Nature Communications, Nature, vol. 15(1), pages 1-15, December.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:78:y:2013:i:3:p:557-575. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.