IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v86y2021i3d10.1007_s11336-021-09775-0.html
   My bibliography  Save this article

Differential Item Functioning Analyses of the Patient-Reported Outcomes Measurement Information System (PROMIS®) Measures: Methods, Challenges, Advances, and Future Directions

Author

Listed:
  • Jeanne A. Teresi

    (Columbia University Stroud Center
    Hebrew Home at Riverdale; RiverSpring Health
    Weill Cornell Medical Center
    New York State Psychiatric Institute)

  • Chun Wang

    (University of Washington College of Education)

  • Marjorie Kleinman

    (New York State Psychiatric Institute)

  • Richard N. Jones

    (Brown University)

  • David J. Weiss

    (University of Minnesota)

Abstract

Several methods used to examine differential item functioning (DIF) in Patient-Reported Outcomes Measurement Information System (PROMIS®) measures are presented, including effect size estimation. A summary of factors that may affect DIF detection and challenges encountered in PROMIS DIF analyses, e.g., anchor item selection, is provided. An issue in PROMIS was the potential for inadequately modeled multidimensionality to result in false DIF detection. Section 1 is a presentation of the unidimensional models used by most PROMIS investigators for DIF detection, as well as their multidimensional expansions. Section 2 is an illustration that builds on previous unidimensional analyses of depression and anxiety short-forms to examine DIF detection using a multidimensional item response theory (MIRT) model. The Item Response Theory-Log-likelihood Ratio Test (IRT-LRT) method was used for a real data illustration with gender as the grouping variable. The IRT-LRT DIF detection method is a flexible approach to handle group differences in trait distributions, known as impact in the DIF literature, and was studied with both real data and in simulations to compare the performance of the IRT-LRT method within the unidimensional IRT (UIRT) and MIRT contexts. Additionally, different effect size measures were compared for the data presented in Section 2. A finding from the real data illustration was that using the IRT-LRT method within a MIRT context resulted in more flagged items as compared to using the IRT-LRT method within a UIRT context. The simulations provided some evidence that while unidimensional and multidimensional approaches were similar in terms of Type I error rates, power for DIF detection was greater for the multidimensional approach. Effect size measures presented in Section 1 and applied in Section 2 varied in terms of estimation methods, choice of density function, methods of equating, and anchor item selection. Despite these differences, there was considerable consistency in results, especially for the items showing the largest values. Future work is needed to examine DIF detection in the context of polytomous, multidimensional data. PROMIS standards included incorporation of effect size measures in determining salient DIF. Integrated methods for examining effect size measures in the context of IRT-based DIF detection procedures are still in early stages of development.

Suggested Citation

  • Jeanne A. Teresi & Chun Wang & Marjorie Kleinman & Richard N. Jones & David J. Weiss, 2021. "Differential Item Functioning Analyses of the Patient-Reported Outcomes Measurement Information System (PROMIS®) Measures: Methods, Challenges, Advances, and Future Directions," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 674-711, September.
    • Handle: RePEc:spr:psycho:v:86:y:2021:i:3:d:10.1007_s11336-021-09775-0
    DOI: 10.1007/s11336-021-09775-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-021-09775-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11336-021-09775-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Robin Shealy & William Stout, 1993. "A model-based standardization approach that separates true bias/DIF from group ability differences and detects test bias/DTF as well as item bias/DIF," Psychometrika, Springer;The Psychometric Society, vol. 58(2), pages 159-194, June.
    2. William Meredith, 1964. "Notes on factorial invariance," Psychometrika, Springer;The Psychometric Society, vol. 29(2), pages 177-185, June.
    3. William Stout, 1987. "A nonparametric approach for assessing latent trait unidimensionality," Psychometrika, Springer;The Psychometric Society, vol. 52(4), pages 589-617, December.
    4. Alexandra Rouquette & Jean-Benoit Hardouin & Alexis Vanhaesebrouck & Véronique Sébille & Joël Coste, 2019. "Differential Item Functioning (DIF) in composite health measurement scale: Recommendations for characterizing DIF with meaningful consequences within the Rasch model framework," PLOS ONE, Public Library of Science, vol. 14(4), pages 1-16, April.
    5. Ting Wang & Carolin Strobl & Achim Zeileis & Edgar C. Merkle, 2018. "Score-Based Tests of Differential Item Functioning via Pairwise Maximum Likelihood Estimation," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 132-155, March.
    6. Qingzhao Yu & Kaelen L. Medeiros & Xiaocheng Wu & Roxanne E. Jensen, 2018. "Nonlinear Predictive Models for Multiple Mediation Analysis: With an Application to Explore Ethnic Disparities in Anxiety and Depression Among Cancer Survivors," Psychometrika, Springer;The Psychometric Society, vol. 83(4), pages 991-1006, December.
    7. Yu-Wei Chang & Nan-Jung Hsu & Rung-Ching Tsai, 2017. "Unifying Differential Item Functioning in Factor Analysis for Categorical Data Under a Discretization of a Normal Variant," Psychometrika, Springer;The Psychometric Society, vol. 82(2), pages 382-406, June.
    8. William Meredith, 1993. "Measurement invariance, factor analysis and factorial invariance," Psychometrika, Springer;The Psychometric Society, vol. 58(4), pages 525-543, December.
    9. Robert J. Zwitser & S. Sjoerd F. Glaser & Gunter Maris, 2017. "Monitoring Countries in a Changing World: A New Look at DIF in International Surveys," Psychometrika, Springer;The Psychometric Society, vol. 82(1), pages 210-232, March.
    10. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    11. Steven Andrew Culpepper & Herman Aguinis & Justin L. Kern & Roger Millsap, 2019. "High-Stakes Testing Case Study: A Latent Variable Approach for Assessing Measurement and Prediction Invariance," Psychometrika, Springer;The Psychometric Society, vol. 84(1), pages 285-309, March.
    12. Yoshio Takane & Jan Leeuw, 1987. "On the relationship between item response theory and factor analysis of discretized variables," Psychometrika, Springer;The Psychometric Society, vol. 52(3), pages 393-408, September.
    13. Nambury Raju, 1988. "The area between two item characteristic curves," Psychometrika, Springer;The Psychometric Society, vol. 53(4), pages 495-502, December.
    14. Carolin Strobl & Julia Kopf & Achim Zeileis, 2015. "Rasch Trees: A New Method for Detecting Differential Item Functioning in the Rasch Model," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 289-316, June.
    15. R. Philip Chalmers, 2018. "Model-Based Measures for Detecting and Quantifying Response Bias," Psychometrika, Springer;The Psychometric Society, vol. 83(3), pages 696-732, September.
    16. Choi, Seung W. & Gibbons, Laura E. & Crane, Paul K., 2011. "lordif: An R Package for Detecting Differential Item Functioning Using Iterative Hybrid Ordinal Logistic Regression/Item Response Theory and Monte Carlo Simulations," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 39(i08).
    17. Robert Mislevy, 1986. "Bayes modal estimation in item response models," Psychometrika, Springer;The Psychometric Society, vol. 51(2), pages 177-195, June.
    18. Brian Junker, 1991. "Essential independence and likelihood-based ability estimation for polytomous items," Psychometrika, Springer;The Psychometric Society, vol. 56(2), pages 255-278, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wallin, Gabriel & Chen, Yunxiao & Moustaki, Irini, 2024. "DIF analysis with unknown groups and anchor items," LSE Research Online Documents on Economics 121991, London School of Economics and Political Science, LSE Library.
    2. Paul Boeck & Sun-Joo Cho, 2021. "Not all DIF is shaped similarly," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 712-716, September.
    3. Bryce B. Reeve & Ron D. Hays, 2021. "Guest Editors’ Introduction to the Invited Special Section," Psychometrika, Springer;The Psychometric Society, vol. 86(3), pages 671-673, September.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ke-Hai Yuan & Hongyun Liu & Yuting Han, 2021. "Differential Item Functioning Analysis Without A Priori Information on Anchor Items: QQ Plots and Graphical Test," Psychometrika, Springer;The Psychometric Society, vol. 86(2), pages 345-377, June.
    2. Ting Wang & Carolin Strobl & Achim Zeileis & Edgar C. Merkle, 2018. "Score-Based Tests of Differential Item Functioning via Pairwise Maximum Likelihood Estimation," Psychometrika, Springer;The Psychometric Society, vol. 83(1), pages 132-155, March.
    3. Cervantes, Víctor H., 2017. "DFIT: An R Package for Raju's Differential Functioning of Items and Tests Framework," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 76(i05).
    4. Minjeong Jeon & Frank Rijmen & Sophia Rabe-Hesketh, 2013. "Modeling Differential Item Functioning Using a Generalization of the Multiple-Group Bifactor Model," Journal of Educational and Behavioral Statistics, , vol. 38(1), pages 32-60, February.
    5. Chen, Yunxiao & Li, Chengcheng & Ouyang, Jing & Xu, Gongjun, 2023. "DIF statistical inference without knowing anchoring items," LSE Research Online Documents on Economics 119923, London School of Economics and Political Science, LSE Library.
    6. Roger Millsap, 2007. "Invariance in Measurement and Prediction Revisited," Psychometrika, Springer;The Psychometric Society, vol. 72(4), pages 461-473, December.
    7. Ting Wang & Benjamin Graves & Yves Rosseel & Edgar C. Merkle, 2022. "Computation and application of generalized linear mixed model derivatives using lme4," Psychometrika, Springer;The Psychometric Society, vol. 87(3), pages 1173-1193, September.
    8. Cees Glas, 1999. "Modification indices for the 2-PL and the nominal response model," Psychometrika, Springer;The Psychometric Society, vol. 64(3), pages 273-294, September.
    9. Johan Oud & Manuel Voelkle, 2014. "Do missing values exist? Incomplete data handling in cross-national longitudinal studies by means of continuous time modeling," Quality & Quantity: International Journal of Methodology, Springer, vol. 48(6), pages 3271-3288, November.
    10. Wilson, Christopher J. & Bowden, Stephen C. & Byrne, Linda K. & Joshua, Nicole R. & Marx, Wolfgang & Weiss, Lawrence G., 2023. "The cross-cultural generalizability of cognitive ability measures: A systematic literature review," Intelligence, Elsevier, vol. 98(C).
    11. A. Béguin & C. Glas, 2001. "MCMC estimation and some model-fit analysis of multidimensional IRT models," Psychometrika, Springer;The Psychometric Society, vol. 66(4), pages 541-561, December.
    12. Edward Ip & Yuchung Wang & Paul Boeck & Michel Meulders, 2004. "Locally dependent latent trait model for polytomous responses with application to inventory of hostility," Psychometrika, Springer;The Psychometric Society, vol. 69(2), pages 191-216, June.
    13. Yong Luo & Khaleel Al-Harbi, 2016. "The Utility of the Bifactor Method for Unidimensionality Assessment When Other Methods Disagree," SAGE Open, , vol. 6(4), pages 21582440166, October.
    14. Youmi Suk & Kyung T. Han, 2024. "A Psychometric Framework for Evaluating Fairness in Algorithmic Decision Making: Differential Algorithmic Functioning," Journal of Educational and Behavioral Statistics, , vol. 49(2), pages 151-172, April.
    15. Adam Carle, 2010. "Interpreting the results of studies using latent variable models to assess data quality: an empirical example using confirmatory factor analysis," Quality & Quantity: International Journal of Methodology, Springer, vol. 44(3), pages 483-497, April.
    16. Jochen Ranger & Kay Brauer, 2022. "On the Generalized S − X 2 –Test of Item Fit: Some Variants, Residuals, and a Graphical Visualization," Journal of Educational and Behavioral Statistics, , vol. 47(2), pages 202-230, April.
    17. Kai Liu & Longfei Zhang & Dongbo Tu & Yan Cai, 2022. "Developing an Item Bank of Computerized Adaptive Testing for Eating Disorders in Chinese University Students," SAGE Open, , vol. 12(4), pages 21582440221, December.
    18. Kano, Yutaka & Takai, Keiji, 2011. "Analysis of NMAR missing data without specifying missing-data mechanisms in a linear latent variate model," Journal of Multivariate Analysis, Elsevier, vol. 102(9), pages 1241-1255, October.
    19. Jeanne A. Teresi & Katja Ocepek-Welikson & John A. Toner & Marjorie Kleinman & Mildred Ramirez & Joseph P. Eimicke & Barry J. Gurland & Albert Siu, 2017. "Methodological Issues in Measuring Subjective Well-Being and Quality-of-Life: Applications to Assessment of Affect in Older, Chronically and Cognitively Impaired, Ethnically Diverse Groups Using the F," Applied Research in Quality of Life, Springer;International Society for Quality-of-Life Studies, vol. 12(2), pages 251-288, June.
    20. Alexander Robitzsch, 2020. "L p Loss Functions in Invariance Alignment and Haberman Linking with Few or Many Groups," Stats, MDPI, vol. 3(3), pages 1-38, August.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:86:y:2021:i:3:d:10.1007_s11336-021-09775-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.