Correcting for sampling problems in PISA and the improvement in Portuguese students' performance

PISA uses a complex sampling procedure based on stratification variables chosen by the participating countries’ authorities. In this paper we analyse the representativeness of the samples used in terms of the distribution of students per grade and track of studies for Portugal. For the three exam years under analysis (2006, 2009 and 2012) a meaningful bias between the effective student distribution and PISA samples was found. We provide recalculated PISA scores that correct for the sample bias. We find that from 2009 to 2012, contrary to the stagnation denoted in the PISA results, the recalculated scores show an improvement in the Portuguese student performance. We also decompose the evolution of the scores into two effects: (i) change in the student population distribution by grade and track, and type of school; and (ii) evolution in the performance of each type of student. The results show that for the whole period the evolution of the students’ scores is the main driver of the increase in PISA results, with the change in the population structure playing a weaker role. Although we focus on the Portuguese case, similar problems of representativeness are expected to arise in other countries with high retention rates, affecting the grade of study of 15-year-olds.