Harnessing uncertainty in clinical prediction models using Stata

Development of new clinical prediction models is in vogue, with many showing off their ill-fitting wares on journal runways. The vast majority of these models ultimately aim to inform care for the individual, based on the probability of their outcome as calculated by the prediction model. Therefore, we should all be concerned about the reliability of such models. Unfortunately, most models are ill-fitting and developed using small samples, exacerbating overfitting and leading to large uncertainty in model predictions for an individual. This issue makes internal validation nonnegotiable in the development of any new model, and its reporting is mandated by the recent TRIPOD+AI guidelines. At the development stage, we know that our model and any estimates of performance are optimistic – our model is fit to our data and so should perform well. Therefore, we commonly assess the internal validity of our model using bootstrapping, allowing us to quantify the optimism in our development process and uncertainty in the model predictions, giving a better feel for how accurate and reliable our model is. In this talk, I will discuss the concept of model uncertainty and demonstrate how our new Stata packages allow developers to estimate uncertainty in their model and harness this information to inform the next steps in the pipeline of their model.