Model Evaluation Using Stochastic Simulations: The Case of the Econometric Model KOSMOS

One aspect of model behaviour that is of interest to the model builder is sensitivity to different forms of errors. This can be investigated using stochastic simulations, as shown by Gajda [1995]. The method involves generating random numbers from a given (usually normal) distribution and introducing them as shocks to the model. In particular, stochastic simulations can be employed to make an empirical investigation into the following aspects of the model: the effects on the forecast of random disturbances, the effects on the forecast of random variation in equation parameters (sampling errors), error propagation and accumulation patterns in the model, the effects on the forecast of (random) errors in the exogenous variables. The results of stochastic simulations can provide information on - inter alia - the sampling distribution of the model forecast. In particular, the model builder may be interested in the shape of this distribution. If it is not symmetric, the mean (stochastic) forecast will be different from the median forecast (which under certain conditions is equal to the deterministic forecast). Furthermore, if the distribution is skewed, a typical stochastic forecast (represented by the mode) will systematically underestimate (or overestimate) both the mean and the median forecasts. The purpose of the present paper is to investigate KOSMOS, the econometric model of the National Institute of Economic Research in Stockholm, from the point of view of the first three aspects mentioned above. The main aim of this exercise is to look for ”weak links” in the model, i.e. to find out which equations introduce most uncertainty and at the same time are crucial for the forecast because of their strong influence on it. Thus, our interest is not only in assessing the forecast error variance as a descriptive statistic, but also - and primarily - in finding those equations that are important for error propagation and those coefficients whose values are crucial for the model. The outline of the paper is as follows. Section 2 discusses the analysis of expected forecast errors (for linear models) based on analytical formulae. In Section 3, model simulations and stochastic simulations are defined. The two subsequent sections discuss the purpose of our experiments and their design, respectively. Section 6 gives a brief description of the econometric model KOSMOS, whose equations are subject to our investigation. Section 7 describes the results of stochastic simulations with additive equation disturbances. Section 8 presents the results of stochastic simulations with both equation disturbances and disturbances to the estimated coefficients. Sampling distributions of forecasts are discussed in Section 9. Section 10 concludes.