Model Performance Measures For Expected Utility Maximizing Investors

We examine model performance measures in four contexts: Discrete Probability, Continuous Probability, Conditional Discrete Probability and Conditional Probability Density Models. We consider the model performance question from the point of view of an investor who evaluates models based on the performance of the (optimal) strategies that the models suggest. Under this new paradigm, the investor selects the model with the highest estimated expected utility. We interpret our performance measures in information theoretic terms and provide new generalizations of entropy and Kullback-Leibler relative entropy. We show that the relative performance measure is independent of the market prices if and only if the investor's utility function is a member of a logarithmic family that admits a wide range of possible risk aversions. In this case, we show that the relative performance measure is equivalent to the (easily understood) differential expected growth of wealth or the (familiar) likelihood ratio. We state conditions under which relative performance measures for general utilities are well approximated by logarithmic-family-based relative performance measures. Some popular probability model performance measures (including ROC methods) are not consistent with our framework. We demonstrate that rank based performance measures can suggest model selections that are disastrous under various popular utilities.