Predicting Skill Shortages in Labor Markets: A Machine Learning Approach

Skill shortages are a drain on society. They hamper economic opportunities for individuals, slow growth for firms, and impede labor productivity in aggregate. Therefore, the ability to understand and predict skill shortages in advance is critical for policy-makers and educators to help alleviate their adverse effects. This research implements a high-performing Machine Learning approach to predict occupational skill shortages. In addition, we demonstrate methods to analyze the underlying skill demands of occupations in shortage and the most important features for predicting skill shortages. For this work, we compile a unique dataset of both Labor Demand and Labor Supply occupational data in Australia from 2012 to 2018. This includes data from 7.7 million job advertisements (ads) and 20 official labor force measures. We use these data as explanatory variables and leverage the XGBoost classifier to predict yearly skills shortage classifications for 132 standardized occupations. The models we construct achieve macro-F1 average performance scores of up to 83 per cent. Our results show that job ads data and employment statistics were the highest performing feature sets for predicting year-to-year skills shortage changes for occupations. We also find that features such as 'Hours Worked', years of 'Education', years of 'Experience', and median 'Salary' are highly important features for predicting occupational skill shortages. This research provides a robust data-driven approach for predicting and analyzing skill shortages, which can assist policy-makers, educators, and businesses to prepare for the future of work.