IDEAS home Printed from https://ideas.repec.org/a/sae/jedbes/v48y2023i6p889-913.html
   My bibliography  Save this article

Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions

Author

Listed:
  • Joshua B. Gilbert
  • James S. Kim
  • Luke W. Miratrix

    (Harvard University Graduate School of Education)

Abstract

Analyses that reveal how treatment effects vary allow researchers, practitioners, and policymakers to better understand the efficacy of educational interventions. In practice, however, standard statistical methods for addressing heterogeneous treatment effects (HTE) fail to address the HTE that may exist within outcome measures. In this study, we present a novel application of the explanatory item response model (EIRM) for assessing what we term “item-level†HTE (IL-HTE), in which a unique treatment effect is estimated for each item in an assessment. Results from data simulation reveal that when IL-HTE is present but ignored in the model, standard errors can be underestimated and false positive rates can increase. We then apply the EIRM to assess the impact of a literacy intervention focused on promoting transfer in reading comprehension on a digital assessment delivered online to approximately 8,000 third-grade students. We demonstrate that allowing for IL-HTE can reveal treatment effects at the item-level masked by a null average treatment effect, and the EIRM can thus provide fine-grained information for researchers and policymakers on the potentially heterogeneous causal effects of educational interventions.

Suggested Citation

  • Joshua B. Gilbert & James S. Kim & Luke W. Miratrix, 2023. "Modeling Item-Level Heterogeneous Treatment Effects With the Explanatory Item Response Model: Leveraging Large-Scale Online Assessments to Pinpoint the Impact of Educational Interventions," Journal of Educational and Behavioral Statistics, , vol. 48(6), pages 889-913, December.
    • Handle: RePEc:sae:jedbes:v:48:y:2023:i:6:p:889-913
    DOI: 10.3102/10769986231171710
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.3102/10769986231171710
    Download Restriction: no
    File URL: https://libkey.io/10.3102/10769986231171710?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sun-Joo Cho & Paul Boeck & Susan Embretson & Sophia Rabe-Hesketh, 2014. "Additive Multilevel Item Structure Models with Random Residuals: Item Modeling for Explanation and Item Generation," Psychometrika, Springer;The Psychometric Society, vol. 79(1), pages 84-104, January.
    2. Bates, Douglas & Mächler, Martin & Bolker, Ben & Walker, Steve, 2015. "Fitting Linear Mixed-Effects Models Using lme4," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 67(i01).
    3. Aeilko Zwinderman, 1991. "A generalized rasch model for manifest predictors," Psychometrika, Springer;The Psychometric Society, vol. 56(4), pages 589-600, December.
    4. De Boeck, Paul & Bakker, Marjan & Zwitser, Robert & Nivard, Michel & Hofman, Abe & Tuerlinckx, Francis & Partchev, Ivailo, 2011. "The Estimation of Item Response Models with the lmer Function from the lme4 Package in R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 39(i12).
    5. José H. Lozano & Javier Revuelta, 2021. "A Bayesian Generalized Explanatory Item Response Model to Account for Learning During the Test," Psychometrika, Springer;The Psychometric Society, vol. 86(4), pages 994-1015, December.
    6. Andrew Bell & Malcolm Fairbrother & Kelvyn Jones, 2019. "Fixed and random effects models: making an informed choice," Quality & Quantity: International Journal of Methodology, Springer, vol. 53(2), pages 1051-1074, March.
    7. Minjeong Jeon & Sophia Rabe-Hesketh, 2012. "Profile-Likelihood Approach for Estimating Generalized Linear Mixed Models With Factor Structures," Journal of Educational and Behavioral Statistics, , vol. 37(4), pages 518-542, August.
    8. Peng Ding & Avi Feller & Luke Miratrix, 2019. "Decomposing Treatment Effect Variation," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(525), pages 304-317, January.
    9. Paul Boeck, 2008. "Random Item IRT Models," Psychometrika, Springer;The Psychometric Society, vol. 73(4), pages 533-559, December.
    10. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Benjamin W. Domingue & Klint Kanopka & Radhika Kapoor & Steffi Pohl & R. Philip Chalmers & Charles Rahal & Mijke Rhemtulla, 2024. "The InterModel Vigorish as a Lens for Understanding (and Quantifying) the Value of Item Response Models for Dichotomously Coded Items," Psychometrika, Springer;The Psychometric Society, vol. 89(3), pages 1034-1054, September.
    2. Joshua B. Gilbert & Zachary Himmelsbach & James Soland & Mridul Joshi & Benjamin W. Domingue, 2024. "Estimating Heterogeneous Treatment Effects with Item-Level Outcome Data: Insights from Item Response Theory," Papers 2405.00161, arXiv.org, revised Aug 2024.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alexander Robitzsch, 2021. "A Comprehensive Simulation Study of Estimation Methods for the Rasch Model," Stats, MDPI, vol. 4(4), pages 1-23, October.
    2. Joshua B. Gilbert & Zachary Himmelsbach & James Soland & Mridul Joshi & Benjamin W. Domingue, 2024. "Estimating Heterogeneous Treatment Effects with Item-Level Outcome Data: Insights from Item Response Theory," Papers 2405.00161, arXiv.org, revised Aug 2024.
    3. Sun-Joo Cho & Jennifer Gilbert & Amanda Goodwin, 2013. "Explanatory Multidimensional Multilevel Random Item Response Model: An Application to Simultaneous Investigation of Word and Person Contributions to Multidimensional Lexical Representations," Psychometrika, Springer;The Psychometric Society, vol. 78(4), pages 830-855, October.
    4. Ting Wang & Benjamin Graves & Yves Rosseel & Edgar C. Merkle, 2022. "Computation and application of generalized linear mixed model derivatives using lme4," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 1173-1193, September.
    5. Quinn N. Lathrop & Ying Cheng, 2017. "Item Cloning Variation and the Impact on the Parameters of Response Models," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 245-263, March.
    6. Antonio Caronni & Marina Ramella & Pietro Arcuri & Claudia Salatino & Lucia Pigini & Maurizio Saruggia & Chiara Folini & Stefano Scarano & Rosa Maria Converti, 2023. "The Rasch Analysis Shows Poor Construct Validity and Low Reliability of the Quebec User Evaluation of Satisfaction with Assistive Technology 2.0 (QUEST 2.0) Questionnaire," IJERPH, MDPI, vol. 20(2), pages 1-19, January.
    7. Sun-Joo Cho & Sarah Brown-Schmidt & Woo-yeol Lee, 2018. "Autoregressive Generalized Linear Mixed Effect Models with Crossed Random Effects: An Application to Intensive Binary Time Series Eye-Tracking Data," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 751-771, September.
    8. Cervantes, Víctor H., 2017. "DFIT: An R Package for Raju's Differential Functioning of Items and Tests Framework," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 76(i05).
    9. Li, Kai & Chen, Pei-Ying & Yan, Erjia, 2019. "Challenges of measuring software impact through citations: An examination of the lme4 R package," Journal of Informetrics, Elsevier, vol. 13(1), pages 449-461.
    10. Øystein Sørensen & Anders M. Fjell & Kristine B. Walhovd, 2023. "Longitudinal Modeling of Age-Dependent Latent Traits with Generalized Additive Latent and Mixed Models," Psychometrika, Springer;The Psychometric Society, vol. 88(2), pages 456-486, June.
    11. Harold Doran, 2023. "A Collection of Numerical Recipes Useful for Building Scalable Psychometric Applications," Journal of Educational and Behavioral Statistics, , vol. 48(1), pages 37-69, February.
    12. Tibor András Marton & Anna Kis & Anna Zubor-Nemes & Anikó Kern & Nándor Fodor, 2020. "Human Impact Promotes Sustainable Corn Production in Hungary," Sustainability, MDPI, vol. 12(17), pages 1-16, August.
    13. Seth Nana Kwame Appiah-Kubi & Karel Malec & Joseph Phiri & Mansoor Maitah & Zdeňka Gebeltová & Luboš Smutka & Vojtech Blazek & Kamil Maitah & Jitka Sirohi, 2021. "Impact of Tax Incentives on Foreign Direct Investment: Evidence from Africa," Sustainability, MDPI, vol. 13(15), pages 1-12, August.
    14. Stewart, Fraser, 2021. "All for sun, sun for all: Can community energy help to overcome socioeconomic inequalities in low-carbon technology subsidies?," Energy Policy, Elsevier, vol. 157(C).
    15. Minjeong Jeon & Sophia Rabe-Hesketh, 2016. "An autoregressive growth model for longitudinal item analysis," Psychometrika, Springer;The Psychometric Society, vol. 81(3), pages 830-850, September.
    16. Youmi Suk & Hyunseung Kang, 2022. "Robust Machine Learning for Treatment Effects in Multilevel Observational Studies Under Cluster-level Unmeasured Confounding," Psychometrika, Springer;The Psychometric Society, vol. 87(1), pages 310-343, March.
    17. Alexia Stock & Rachel A. Davidson & James Kendra & V. Nuno Martins & Bradley Ewing & Linda K. Nozick & Kate Starbird & Maggie Leon-Corwin, 2023. "Household impacts of interruption to electric power and water services," Natural Hazards: Journal of the International Society for the Prevention and Mitigation of Natural Hazards, Springer;International Society for the Prevention and Mitigation of Natural Hazards, vol. 115(3), pages 2279-2306, February.
    18. Minjeong Jeon & Ick Hoon Jin & Michael Schweinberger & Samuel Baugh, 2021. "Mapping Unobserved Item–Respondent Interactions: A Latent Space Item Response Model with Interaction Map," Psychometrika, Springer;The Psychometric Society, vol. 86(2), pages 378-403, June.
    19. Izolda Pristojkovic Suko & Magdalena Holter & Erwin Stolz & Elfriede Renate Greimel & Wolfgang Freidl, 2022. "Acculturation, Adaptation, and Health among Croatian Migrants in Austria and Ireland: A Cross-Sectional Study," IJERPH, MDPI, vol. 19(24), pages 1-15, December.
    20. JANSSENS, Jochen & DE CORTE, Annelies & SÖRENSEN, Kenneth, 2016. "Water distribution network design optimisation with respect to reliability," Working Papers 2016007, University of Antwerp, Faculty of Business and Economics.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:jedbes:v:48:y:2023:i:6:p:889-913. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.