The Difference Between Instability and Uncertainty: Comment on Young and Holsteen (2017)

Young and Holsteen (YH) introduce a number of tools for evaluating model uncertainty. In so doing, they are careful to differentiate their method from existing forms of model averaging. The fundamental difference lies in the way in which the underlying estimates are weighted. Whereas standard approaches to model averaging assign higher weight to better fitting models, the YH method weights all models equally. As I show, this is a nontrivial distinction, in that the two sets of procedures tend to produce radically different results. Drawing on both simulation and real-world examples, I demonstrate that in failing to distinguish between numerical variation and statistical uncertainty, the procedure proposed by YH will tend to overstate the amount of uncertainty resulting from variation across models. In standard circumstances, the quality of estimates produced using this method will tend to be objectively worse than that of conventional alternatives.