Enhancing analytic rigor in qualitative analysis: developing and testing code scheme using Many Facet Rasch Model

Performance assessments that include multiple raters using rating scales to evaluate the quality of a performance are frequently seen and reported. However, the establishment of intercoder reliability in analysing content validity of codes applied to data from transcripts in a qualitative study is not widely practiced. This paper presents intercoder reliability using Many Facet Rasch Model (MFRM) in developing, testing, and enhancing the code scheme of a qualitative study. The results suggest that although the raters were in the same field of expertise and were given the same set of code scheme, their independent characteristics were still observable because unlike the standard approach, MFRM allowed for comparability of each expert’s rating operation as opposed to group homogeneity. While all raters agreed to assign 70% of the codings in the category of very good and good, the researchers were still able to determine the raters’ severity or leniency based on the ratings they assigned. Moreover, MFRM’s ability to provide independent piece of information helped the researchers to identify which codings were accepted and which coding needed review on the researchers’ part. The evaluation has increased trust in the quality of codings and this has created a sense of confidence to the researchers in establishing the code scheme for the qualitative study.