IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v38y1992i8p1164-1185.html
   My bibliography  Save this article

Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals

Author

Listed:
  • Jack P. C. Kleijnen

    (Department of Information Systems and Auditing, School of Business and Economics, Katholieke Universiteit Brabant (Tilburg University), 5000 LE Tilburg, Netherlands)

Abstract

Linear regression analysis is important in many fields. In the analysis of simulation results, a regression (meta)model can be applied, even when common pseudorandom numbers are used. To test the validity of the specified regression model, Rao (1959) generalized the F statistic for lack of fit, whereas Kleijnen (1983) proposed a cross-validation procedure using a Student's t statistic combined with Bonferroni's inequality. This paper reports on an extensive Monte Carlo experiment designed to compare these two methods. Under the normality assumption, cross-validation is conservative, whereas Rao's test realizes its nominal type I error and has high power. Robustness is investigated through lognormal and uniform distributions. When simulation responses are distributed lognormally, then cross-validation using Ordinary Least Squares is the only technique that has acceptable type I error. Uniform distributions give results similar to the normal case. Once the regression model is validated, confidence intervals for the individual regression parameters are computed. The Monte Carlo experiment compares several confidence interval procedures. Under normality, Rao's procedure is preferred since it has good coverage probability and acceptable half-length. Under lognormality, Ordinary Least Squares achieves nominal coverage probability. Uniform distributions again give results similar to the normal case.

Suggested Citation

  • Jack P. C. Kleijnen, 1992. "Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals," Management Science, INFORMS, vol. 38(8), pages 1164-1185, August.
    • Handle: RePEc:inm:ormnsc:v:38:y:1992:i:8:p:1164-1185
    DOI: 10.1287/mnsc.38.8.1164
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.38.8.1164
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.38.8.1164?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Kleijnen, J.P.C., 2007. "Simulation Experiments in Practice : Statistical Design and Regression Analysis," Discussion Paper 2007-09, Tilburg University, Center for Economic Research.
    2. Kleijnen, J.P.C., 2001. "Experimental Design for Sensitivity Analysis of Simulation Models," Other publications TiSEM 1c716145-1d34-41b5-b965-7, Tilburg University, School of Economics and Management.
    3. Kleijnen, Jack P. C., 2005. "An overview of the design and analysis of simulation experiments for sensitivity analysis," European Journal of Operational Research, Elsevier, vol. 164(2), pages 287-300, July.
    4. Kleijnen, J.P.C., 2006. "White Noise Assumptions Revisited : Regression Models and Statistical Designs for Simulation Practice," Discussion Paper 2006-50, Tilburg University, Center for Economic Research.
    5. Cheng, R.C.H. & Kleijnen, J.P.C., 1995. "Optimal design of simulation experiments with nearly saturated queues," Discussion Paper 1995-67, Tilburg University, Center for Economic Research.
    6. Leutscher, K. J. & Renkema, J. A. & Challa, H., 1999. "Modelling operational adaptations of tactical production plans on pot plant nurseries: a simulation approach," Agricultural Systems, Elsevier, vol. 59(1), pages 67-78, January.
    7. Kleijnen, Jack P.C. & Deflandre, David, 2006. "Validation of regression metamodels in simulation: Bootstrap approach," European Journal of Operational Research, Elsevier, vol. 170(1), pages 120-131, April.
    8. Kleijnen, J.P.C., 1997. "Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models," Discussion Paper 1997-52, Tilburg University, Center for Economic Research.
    9. Reis dos Santos, M. Isabel & Porta Nova, Acacio M.O., 2006. "Statistical fitting and validation of non-linear simulation metamodels: A case study," European Journal of Operational Research, Elsevier, vol. 171(1), pages 53-63, May.
    10. Ali Kokangul & Serap Akcan & Mufide Narli, 2017. "Optimizing nurse capacity in a teaching hospital neonatal intensive care unit," Health Care Management Science, Springer, vol. 20(2), pages 276-285, June.
    11. Shi, Wen & Kleijnen, J.P.C., 2017. "Testing the Assumptions of Sequential Bifurcation for Factor Screening (revision of CentER DP 2015-034)," Discussion Paper 2017-006, Tilburg University, Center for Economic Research.
    12. Russell C. H. Cheng & Jack P. C. Kleijnen, 1999. "Improved Design of Queueing Simulation Experiments with Highly Heteroscedastic Responses," Operations Research, INFORMS, vol. 47(5), pages 762-777, October.
    13. Kleijnen, Jack P. C. & Sargent, Robert G., 2000. "A methodology for fitting and validating metamodels in simulation," European Journal of Operational Research, Elsevier, vol. 120(1), pages 14-29, January.
    14. Kleijnen, J.P.C., 2001. "Experimental designs for sensitivity analysis of simulation models," Other publications TiSEM bcb932d5-6abe-4da6-977e-5, Tilburg University, School of Economics and Management.
    15. Kleijnen, J.P.C., 1995. "Sensitivity analysis and optimization of system dynamics models : Regression analysis and statistical design of experiments," Other publications TiSEM 87ee6ee0-592c-4204-ac50-6, Tilburg University, School of Economics and Management.
    16. van Groenendaal, W.J.H. & Kleijnen, J.P.C., 1997. "On the assessment of economic risk : Factorial design versus Monte Carlo methods," Other publications TiSEM fd2a2307-0812-4543-8151-7, Tilburg University, School of Economics and Management.
    17. Safizadeh, M. Hossein, 2002. "Minimizing the bias and variance of the gradient estimate in RSM simulation studies," European Journal of Operational Research, Elsevier, vol. 136(1), pages 121-135, January.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:38:y:1992:i:8:p:1164-1185. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.