Evaluation of open items using the many-facet Rasch model

The goal of this study is to analyze the quality of ratings assigned to two constructed response questions for evaluating the written ability of essays in Portuguese language from the perspective of the many-facet Rasch (MFR [15]) model. The analyzed data set comes from 350 written tests with two open-item tasks that were developed based on a rating process independently marked by two rater coordinators and a group of 42 raters. The MFR model analysis shows the measurement quality related to the examinees, raters, tasks and items, and classification scale that has been used for the task rating process. The findings indicate significant differences amongst the rater severities and show that the raters cannot be interchanged. The results also suggest that the comparison between the two task difficulties needs further investigation. An additional study has been done on the scale structure of the classification used by each rater for each item. The result suggests that there have been some similarities amongst the tasks and a need of revision for some criteria of the rating process. Overall, the scale of evaluation has shown to be efficient for a classification of the examinees.