Metrics for Evaluating the Performance of Automated Valuation Models

Automated Valuation Models (AVMs) based on Machine Learning (ML) algorithms are widely used for the prediction of house prices. While there is consensus in the literature that cross-validation (CV) should be used for model selection in this context, the question of which performance metrics to use is generally neglected. Here we collect the most commonly used metrics from the AVM literature and elsewhere, and evaluate them with respect to two symmetry conditions: symmetry with respect to prediction error rates and symmetry with respect to the treatment of actual and predicted values. While none of the commonly used metrics satisfy both conditions, we propose a number of new metrics that do. We also show how popular existing metrics can be altered so that they adhere to these conditions. To illustrate our findings we compare the performance of 5 ML-based AVMs and find, that the most popular metrics in the AVM literature can generate misleading results. A different picture emerges when the full set of metrics is considered, and especially when we focus on four key metrics with the best symmetry properties.