IDEAS home Printed from https://ideas.repec.org/p/tiu/tiucen/d4bee6b0-7a00-4695-85ed-6975fe7cb1f2.html
   My bibliography  Save this paper

Simulation Experiments in Practice : Statistical Design and Regression Analysis

Author

Listed:
  • Kleijnen, J.P.C.

    (Tilburg University, Center For Economic Research)

Abstract

In practice, simulation analysts often change only one factor at a time, and use graphical analysis of the resulting Input/Output (I/O) data. The goal of this article is to change these traditional, naïve methods of design and analysis, because statistical theory proves that more information is obtained when applying Design Of Experiments (DOE) and linear regression analysis. Unfortunately, classic DOE and regression analysis assume a single simulation response that is normally and independently distributed with a constant variance; moreover, the regression (meta)model of the simulation model’s I/O behaviour is assumed to have residuals with zero means. This article addresses the following practical questions: (i) How realistic are these assumptions, in practice? (ii) How can these assumptions be tested? (iii) If assumptions are violated, can the simulation's I/O data be transformed such that the assumptions do hold? (iv) If not, which alternative statistical methods can then be applied?
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(This abstract was borrowed from another version of this item.)
(Th
(This abstract was borrowed from another version of this item.)

Suggested Citation

  • Kleijnen, J.P.C., 2007. "Simulation Experiments in Practice : Statistical Design and Regression Analysis," Discussion Paper 2007-30, Tilburg University, Center for Economic Research.
    • Handle: RePEc:tiu:tiucen:d4bee6b0-7a00-4695-85ed-6975fe7cb1f2
    as

    Download full text from publisher

    File URL: https://pure.uvt.nl/ws/portalfiles/portal/825496/dp2007-30.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Jack P. C. Kleijnen & Russell C. H. Cheng & Bert Bettonvil, 2001. "Validation of Trace-Driven Simulation Models: Bootstrap Tests," Management Science, INFORMS, vol. 47(11), pages 1533-1538, November.
    2. Kleijnen, Jack P.C. & Deflandre, David, 2006. "Validation of regression metamodels in simulation: Bootstrap approach," European Journal of Operational Research, Elsevier, vol. 170(1), pages 120-131, April.
    3. Kleijnen, J.P.C. & van Groenendaal, W.J.H., 1992. "Two-stage versus sequential sample-size determination in regression analysis of simulation experiments," Other publications TiSEM cac61c9f-0967-4f5a-aab1-9, Tilburg University, School of Economics and Management.
    4. van Beers, Wim C.M. & Kleijnen, Jack P.C., 2008. "Customized sequential designs for random simulation experiments: Kriging metamodeling and bootstrapping," European Journal of Operational Research, Elsevier, vol. 186(3), pages 1099-1113, May.
    5. Arcones, Miguel A. & Wang, Yishi, 2006. "Some new tests for normality based on U-processes," Statistics & Probability Letters, Elsevier, vol. 76(1), pages 69-82, January.
    6. J.P.C. Kleijnen & J. Kriens & H. Timmermans & H. van den Wildenberg, 1989. "Regression Sampling in Statistical Auditing: A Practical Survey and Evaluation," Statistica Neerlandica, Netherlands Society for Statistics and Operations Research, vol. 43(4), pages 193-209, December.
    7. Jack P. C. Kleijnen, 1992. "Regression Metamodels for Simulation with Common Random Numbers: Comparison of Validation Tests and Confidence Intervals," Management Science, INFORMS, vol. 38(8), pages 1164-1185, August.
    8. Kleijnen, J.P.C. & Cheng, R.C.H. & Bettonvil, B.W.M., 2001. "Validation of trace-driven simulation models : Bootstrapped tests," Other publications TiSEM 13d22f9a-c1e2-45a7-b96a-9, Tilburg University, School of Economics and Management.
    9. Davidson, Russell & MacKinnon, James G., 2007. "Improving the reliability of bootstrap tests with the fast double bootstrap," Computational Statistics & Data Analysis, Elsevier, vol. 51(7), pages 3259-3281, April.
    10. Kleijnen, J.P.C., 2006. "White Noise Assumptions Revisited : Regression Models and Statistical Designs for Simulation Practice," Other publications TiSEM d8c37ad3-f9a5-4824-986d-2, Tilburg University, School of Economics and Management.
    11. J P C Kleijnen & W C M van Beers, 2004. "Application-driven sequential designs for simulation experiments: Kriging metamodelling," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 55(8), pages 876-883, August.
    12. Ayanso, Anteneh & Diaby, Moustapha & Nair, Suresh K., 2006. "Inventory rationing via drop-shipping in Internet retailing: A sensitivity analysis," European Journal of Operational Research, Elsevier, vol. 171(1), pages 135-152, May.
    13. Ruud, Paul A., 2000. "An Introduction to Classical Econometric Theory," OUP Catalogue, Oxford University Press, number 9780195111644.
    14. V C Ivănescu & J W M Bertrand & J C Fransoo & J P C Kleijnen, 2006. "Bootstrapping to solve the limited data problem in production control: an application in batch process industries," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 57(1), pages 2-9, January.
    15. Freeman, Jade & Modarres, Reza, 2006. "Inverse Box-Cox: The power-normal distribution," Statistics & Probability Letters, Elsevier, vol. 76(8), pages 764-772, April.
    16. Matias Salibian-Barrera, 2006. "The Asymptotics of MM-Estimators for Linear Regression with Fixed Designs," Metrika: International Journal for Theoretical and Applied Statistics, Springer, vol. 63(3), pages 283-294, June.
    17. Godfrey, L.G., 2006. "Tests for regression models with heteroskedasticity of unknown form," Computational Statistics & Data Analysis, Elsevier, vol. 50(10), pages 2715-2733, June.
    18. Helton, J.C. & Johnson, J.D. & Sallaberry, C.J. & Storlie, C.B., 2006. "Survey of sampling-based methods for uncertainty and sensitivity analysis," Reliability Engineering and System Safety, Elsevier, vol. 91(10), pages 1175-1209.
    19. Salibian-Barrera, Matias, 2006. "Bootstrapping MM-estimators for linear regression with fixed designs," Statistics & Probability Letters, Elsevier, vol. 76(12), pages 1287-1297, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Kleijnen, J.P.C., 2006. "White Noise Assumptions Revisited : Regression Models and Statistical Designs for Simulation Practice," Other publications TiSEM d8c37ad3-f9a5-4824-986d-2, Tilburg University, School of Economics and Management.
    2. Scott L. Rosen & Christopher P. Saunders & Samar K Guharay, 2015. "A Structured Approach for Rapidly Mapping Multilevel System Measures via Simulation Metamodeling," Systems Engineering, John Wiley & Sons, vol. 18(1), pages 87-101, January.
    3. Kleijnen, J.P.C. & van Beers, W.C.M. & van Nieuwenhuyse, I., 2008. "Constrained Optimization in Simulation : A Novel Approach," Discussion Paper 2008-95, Tilburg University, Center for Economic Research.
    4. Kleijnen, Jack P.C. & Deflandre, David, 2006. "Validation of regression metamodels in simulation: Bootstrap approach," European Journal of Operational Research, Elsevier, vol. 170(1), pages 120-131, April.
    5. Kleijnen, Jack P.C., 2009. "Kriging metamodeling in simulation: A review," European Journal of Operational Research, Elsevier, vol. 192(3), pages 707-716, February.
    6. Kleijnen, Jack P.C., 2017. "Regression and Kriging metamodels with their experimental designs in simulation: A review," European Journal of Operational Research, Elsevier, vol. 256(1), pages 1-16.
    7. Russell C. H. Cheng & Jack P. C. Kleijnen, 1999. "Improved Design of Queueing Simulation Experiments with Highly Heteroscedastic Responses," Operations Research, INFORMS, vol. 47(5), pages 762-777, October.
    8. Camponovo, Lorenzo & Scaillet, Olivier & Trojani, Fabio, 2012. "Robust subsampling," Journal of Econometrics, Elsevier, vol. 167(1), pages 197-210.
    9. Kleijnen, J.P.C., 2006. "Generalized Response Surface Methodology : A New Metaheuristic," Other publications TiSEM 9a11fb77-d2e1-4c6e-8bba-0, Tilburg University, School of Economics and Management.
    10. Plischke, Elmar & Borgonovo, Emanuele & Smith, Curtis L., 2013. "Global sensitivity measures from given data," European Journal of Operational Research, Elsevier, vol. 226(3), pages 536-550.
    11. Kleijnen, Jack P.C. & Beers, Wim van & Nieuwenhuyse, Inneke van, 2010. "Constrained optimization in expensive simulation: Novel approach," European Journal of Operational Research, Elsevier, vol. 202(1), pages 164-174, April.
    12. Kleijnen, Jack P.C., 2011. "Ethical issues in engineering models : An operations researcher's reflections," Other publications TiSEM 88caba9e-1d78-47d7-9a0b-b, Tilburg University, School of Economics and Management.
    13. Deflandre, D. & Kleijnen, J.P.C., 2002. "Statistical Analysis of Random Simulations : Bootstrap Tutorial," Other publications TiSEM 351056e6-c63d-4fec-8c00-a, Tilburg University, School of Economics and Management.
    14. M I Reis dos Santos & P M Reis dos Santos, 2011. "Construction and validation of distribution-based regression simulation metamodels," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 62(7), pages 1376-1384, July.
    15. Song, Kunling & Zhang, Yugang & Shen, Linjie & Zhao, Qingyan & Song, Bifeng, 2021. "A failure boundary exploration and exploitation framework combining adaptive Kriging model and sample space partitioning strategy for efficient reliability analysis," Reliability Engineering and System Safety, Elsevier, vol. 216(C).
    16. Ali Kokangul & Serap Akcan & Mufide Narli, 2017. "Optimizing nurse capacity in a teaching hospital neonatal intensive care unit," Health Care Management Science, Springer, vol. 20(2), pages 276-285, June.
    17. Kleijnen, J.P.C., 1997. "Experimental Design for Sensitivity Analysis, Optimization and Validation of Simulation Models," Discussion Paper 1997-52, Tilburg University, Center for Economic Research.
    18. Klein, Ingo, 2012. "Quasi-arithmetische Mittelwerte und Normalverteilung," Discussion Papers 89/2010, Friedrich-Alexander University Erlangen-Nuremberg, Chair of Statistics and Econometrics.
    19. Schick, Anton & Wang, Yishi & Wefelmeyer, Wolfgang, 2011. "Tests for normality based on density estimators of convolutions," Statistics & Probability Letters, Elsevier, vol. 81(2), pages 337-343, February.
    20. Jan R. Magnus & Wendun Wang & Xinyu Zhang, 2016. "Weighted-Average Least Squares Prediction," Econometric Reviews, Taylor & Francis Journals, vol. 35(6), pages 1040-1074, June.

    More about this item

    Keywords

    metamodel; experimental design; jackknife; bootstrap; common random numbers; validation;
    All these keywords.

    JEL classification:

    • C0 - Mathematical and Quantitative Methods - - General
    • C1 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:tiu:tiucen:d4bee6b0-7a00-4695-85ed-6975fe7cb1f2. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Richard Broekman (email available below). General contact details of provider: http://center.uvt.nl .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.