IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v47y2001i11p1533-1538.html
   My bibliography  Save this article

Validation of Trace-Driven Simulation Models: Bootstrap Tests

Author

Listed:
  • Jack P. C. Kleijnen

    (Department of Information Systems (BIK)/Center for Economic Research (CentER), Tilburg University (KUB), 5000 LE Tilburg, Netherlands)

  • Russell C. H. Cheng

    (Department of Mathematical Sciences, University of Southampton, Highfield, Southampton, SO17 1BJ, United Kingdom)

  • Bert Bettonvil

    (Department of Information Systems (BIK)/Center for Economic Research (CentER), Tilburg University (KUB), 5000 LE Tilburg, Netherlands)

Abstract

Trace-driven (or correlated inspection) simulation means that the simulated and the real systems have some common inputs (say, historical arrival times) so that the two systems' outputs are cross-correlated. To validate such a simulation, this paper focuses on the difference between the average simulated and real responses. To evaluate this validation statistic, the paper develops a novel bootstrap technique---based on replicated runs. This validation statistic and the bootstrap technique are evaluated in extensive Monte Carlo experiments with specific single-server queues. These experiments show acceptable Type-I and Type-II error probabilities.

Suggested Citation

  • Jack P. C. Kleijnen & Russell C. H. Cheng & Bert Bettonvil, 2001. "Validation of Trace-Driven Simulation Models: Bootstrap Tests," Management Science, INFORMS, vol. 47(11), pages 1533-1538, November.
    • Handle: RePEc:inm:ormnsc:v:47:y:2001:i:11:p:1533-1538
    DOI: 10.1287/mnsc.47.11.1533.10255
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.47.11.1533.10255
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.47.11.1533.10255?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Kleijnen, J.P.C., 2000. "Strategic directions in verification, validation and accreditation research : A personal view," Other publications TiSEM eee4931d-abef-4137-8112-7, Tilburg University, School of Economics and Management.
    2. Jack P. C. Kleijnen & Bert Bettonvil & Willem Van Groenendaal, 1998. "Validation of Trace-Driven Simulation Models: A Novel Regression Test," Management Science, INFORMS, vol. 44(6), pages 812-819, June.
    3. Bruce Schmeiser, 1982. "Batch Size Effects in the Analysis of Simulation Output," Operations Research, INFORMS, vol. 30(3), pages 556-568, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Deflandre, D. & Kleijnen, J.P.C., 2002. "Statistical Analysis of Random Simulations : Bootstrap Tutorial," Other publications TiSEM 351056e6-c63d-4fec-8c00-a, Tilburg University, School of Economics and Management.
    2. Kleijnen, J.P.C., 2007. "Simulation Experiments in Practice : Statistical Design and Regression Analysis," Other publications TiSEM d4bee6b0-7a00-4695-85ed-6, Tilburg University, School of Economics and Management.
    3. Kleijnen, J.P.C., 2007. "Simulation Experiments in Practice : Statistical Design and Regression Analysis," Other publications TiSEM 66449de2-3d9c-4d19-a3e4-a, Tilburg University, School of Economics and Management.
    4. Kleijnen, Jack P.C., 2011. "Ethical issues in engineering models : An operations researcher's reflections," Other publications TiSEM 88caba9e-1d78-47d7-9a0b-b, Tilburg University, School of Economics and Management.
    5. Kleijnen, J.P.C., 2007. "Simulation Experiments in Practice : Statistical Design and Regression Analysis," Discussion Paper 2007-09, Tilburg University, Center for Economic Research.
    6. Kleijnen, Jack P.C. & Deflandre, David, 2006. "Validation of regression metamodels in simulation: Bootstrap approach," European Journal of Operational Research, Elsevier, vol. 170(1), pages 120-131, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Jack P. C. Kleijnen & Susan M. Sanchez & Thomas W. Lucas & Thomas M. Cioppa, 2005. "State-of-the-Art Review: A User’s Guide to the Brave New World of Designing Simulation Experiments," INFORMS Journal on Computing, INFORMS, vol. 17(3), pages 263-289, August.
    2. David Goldsman & Keebom Kang & Seong‐Hee Kim & Andrew F. Seila & Gamze Tokol, 2007. "Combining standardized time series area and Cramér–von Mises variance estimators," Naval Research Logistics (NRL), John Wiley & Sons, vol. 54(4), pages 384-396, June.
    3. Mingchang Chih, 2019. "An Insight into the Data Structure of the Dynamic Batch Means Algorithm with Binary Tree Code," Mathematics, MDPI, vol. 7(9), pages 1-8, August.
    4. van Schaik, F.D.J. & Kleijnen, J.P.C., 2001. "Sealed-Bid Auctions : Case Study," Other publications TiSEM dd93bf40-d790-4fd1-ac2c-f, Tilburg University, School of Economics and Management.
    5. Driessen, M.A. & van Houtum, G.J. & Zijm, W.H.M. & Rustenburg, W.D., 2020. "Capacity assignment in repair shops with high material uncertainty," International Journal of Production Economics, Elsevier, vol. 221(C).
    6. Kleijnen, J.P.C., 2001. "Experimental Design for Sensitivity Analysis of Simulation Models," Other publications TiSEM 1c716145-1d34-41b5-b965-7, Tilburg University, School of Economics and Management.
    7. Srinagesh Gavirneni & Douglas J. Morrice & Peter Mullarkey, 2004. "Simulation Helps Maxager Shorten Its Sales Cycle," Interfaces, INFORMS, vol. 34(2), pages 87-96, April.
    8. Kenneth W. Bauer & James R. Wilson, 1992. "Control‐variate selection criteria," Naval Research Logistics (NRL), John Wiley & Sons, vol. 39(3), pages 307-321, April.
    9. Legato, Pasquale & Mazza, Rina M., 2001. "Berth planning and resources optimisation at a container terminal via discrete event simulation," European Journal of Operational Research, Elsevier, vol. 133(3), pages 537-547, September.
    10. Song, Wheyming Tina & Chih, Mingchang, 2013. "Run length not required: Optimal-mse dynamic batch means estimators for steady-state simulations," European Journal of Operational Research, Elsevier, vol. 229(1), pages 114-123.
    11. Kleijnen, J.P.C., 2001. "Experimental designs for sensitivity analysis of simulation models," Other publications TiSEM bcb932d5-6abe-4da6-977e-5, Tilburg University, School of Economics and Management.
    12. Karen T. Hicklin & Julie S. Ivy & James R. Wilson & Fay Cobb Payton & Meera Viswanathan & Evan R. Myers, 2019. "Simulation model of the relationship between cesarean section rates and labor duration," Health Care Management Science, Springer, vol. 22(4), pages 635-657, December.
    13. Kleijnen, Jack P. C. & Sargent, Robert G., 2000. "A methodology for fitting and validating metamodels in simulation," European Journal of Operational Research, Elsevier, vol. 120(1), pages 14-29, January.
    14. Ilan Halachmi & Yitzhak Simon & Noam Mozes, 2014. "Simulation of the shift from marine netcages to inland recirculating aquaculture systems," Annals of Operations Research, Springer, vol. 219(1), pages 85-99, August.
    15. Muñoz Negrón, David F. & Muñoz Medina, Diego F., 2009. "Bayesian Forecastings For Automobile Parts Using Stochastic Simulation," Journal of Economics, Finance and Administrative Science, Universidad ESAN, vol. 14(27), pages 7-20.
    16. Kleijnen, Jack P.C. & van Schaik, Frans D.J., 2011. "Sealed-bid auction of Netherlands mussels: Statistical analysis," International Journal of Production Economics, Elsevier, vol. 132(1), pages 154-161, July.
    17. Tûba Aktaran‐Kalaycı & Christos Alexopoulos & Nilay Tanık Argon & David Goldsman & James R. Wilson, 2007. "Exact expected values of variance estimators for simulation," Naval Research Logistics (NRL), John Wiley & Sons, vol. 54(4), pages 397-410, June.
    18. Chaitra H. Nagaraja & Haikady N. Nagaraja, 2020. "Distribution‐free Approximate Methods for Constructing Confidence Intervals for Quantiles," International Statistical Review, International Statistical Institute, vol. 88(1), pages 75-100, April.
    19. Munoz, F.D. & Hobbs, B.F. & Watson, J.-P., 2016. "New bounding and decomposition approaches for MILP investment problems: Multi-area transmission and generation planning under policy constraints," European Journal of Operational Research, Elsevier, vol. 248(3), pages 888-898.
    20. Vonk Noordegraaf, Antonie & Nielen, Mirjam & Kleijnen, Jack P. C., 2003. "Sensitivity analysis by experimental design and metamodelling: Case study on simulation in national animal disease control," European Journal of Operational Research, Elsevier, vol. 146(3), pages 433-443, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:47:y:2001:i:11:p:1533-1538. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.