Moving Beyond Comparative Validation: Predictive Abilities of APPSIM's Health Module

In the development of a dynamic microsimulation model, validation lends credibility to the model, making it more likely to be accepted by policy makers. In the validation of multi-module models such as the Australian Population and Policy Simulation model (APPSIM) currently under development at the National Centre for Social and Economic Modelling (NATSEM), one is faced with additional validation challenges. Not only do the specific characteristics of the module under consideration need validating but so too do the data inputs from earlier developed modules. Moreover, validation needs to account for dynamic interactions with the other modules in the model system. This paper considers the validation of the health module of APPSIM, which is itself highly reliant on prerequisite modules such as education and earnings. Commonly used comparative validation methods are used. In addition, as part of the validation of the health module, issues related to variable inputs are disentangled from the health prediction models. To achieve this, a version of the APPSIM health module is developed in SAS and run against the Australian longitudinal HILDA dataset between 2006 and 2008. This provides an alternate data source for the socio-economic variables used in the model, providing health outcomes to assess the equation?s predictive quality. Since these data are not used in the development of the prediction equations, the proposed method provides an independent dataset for testing the module?s generalisability. Using this method, it is possible to verify both the baseline imputation and the transition equations used in the dynamic simulation model. The validation of the equations? predictive qualities includes „confusion tables? to consider at a simple level the predictive accuracy of the model, precision measures looking at the ratio of true positive results to all positive results, and ROC curves to consider the true positives against the false positives. As an illustration, a validation of predictions of being overweight or obese from the APPSIM health module is presented.