IDEAS home Printed from https://ideas.repec.org/a/inm/ormnsc/v48y2002i12p1569-1586.html
   My bibliography  Save this article

An Improved Batch Means Procedure for Simulation Output Analysis

Author

Listed:
  • Natalie M. Steiger

    (Maine Business School, University of Maine, Orono, Maine 04469)

  • James R. Wilson

    (Department of Industrial Engineering, North Carolina State University, Raleigh, North Carolina 27695)

Abstract

We formulate and evaluate the Automated Simulation Analysis Procedure (ASAP), an algorithm for steady-state simulation output analysis based on the method of nonover-lapping batch means (NOBM). ASAP delivers a confidence interval for an expected response that is centered on the sample mean of a portion of a simulation-generated time series and satisfies a user-specified absolute or relative precision requirement. ASAP operates as follows: The batch size is progressively increased until either (a) the batch means pass the von Neumann test for independence, and then ASAP delivers a classical NOBM confidence interval; or (b) the batch means pass the Shapiro-Wilk test for multivariate normality, and then ASAP delivers a correlation-adjusted confidence interval. The latter adjustment is based on an inverted Cornish-Fisher expansion for the classical NOBM t-ratio, where the terms of the expansion are estimated via an autoregressive-moving average time series model of the batch means. After determining the batch size and confidence-interval type, ASAP sequentially increases the number of batches until the precision requirement is satisfied. An extensive experimental study demonstrates the performance improvements achieved by ASAP versus well-known batch means procedures, especially in confidence-interval coverage probability.

Suggested Citation

  • Natalie M. Steiger & James R. Wilson, 2002. "An Improved Batch Means Procedure for Simulation Output Analysis," Management Science, INFORMS, vol. 48(12), pages 1569-1586, December.
    • Handle: RePEc:inm:ormnsc:v:48:y:2002:i:12:p:1569-1586
    DOI: 10.1287/mnsc.48.12.1569.438
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.1287/mnsc.48.12.1569.438
    Download Restriction: no
    File URL: https://libkey.io/10.1287/mnsc.48.12.1569.438?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. R. W. Conway, 1963. "Some Tactical Problems in Digital Simulation," Management Science, INFORMS, vol. 10(1), pages 47-61, October.
    2. George S. Fishman, 1978. "Grouping Observations in Digital Simulation," Management Science, INFORMS, vol. 24(5), pages 510-521, January.
    3. Averill M. Law & John S. Carson, 1979. "A Sequential Procedure for Determining the Length of a Steady-State Simulation," Operations Research, INFORMS, vol. 27(5), pages 1011-1025, October.
    4. George S. Fishman & L. Stephen Yarberry, 1997. "An Implementation of the Batch Means Method," INFORMS Journal on Computing, INFORMS, vol. 9(3), pages 296-310, August.
    5. Natalie M. Steiger & James R. Wilson, 2001. "Convergence Properties of the Batch Means Method for Simulation Output Analysis," INFORMS Journal on Computing, INFORMS, vol. 13(4), pages 277-293, November.
    6. Jeffrey D. Tew & James R. Wilson, 1992. "Validation of Simulation Analysis Methods for the Schruben-Margolin Correlation-Induction Strategy," Operations Research, INFORMS, vol. 40(1), pages 87-103, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Lada, Emily K. & Wilson, James R., 2006. "A wavelet-based spectral procedure for steady-state simulation analysis," European Journal of Operational Research, Elsevier, vol. 174(3), pages 1769-1801, November.
    2. Ali Tafazzoli & James R. Wilson & Emily K. Lada & Natalie M. Steiger, 2011. "Performance of Skart: A Skewness- and Autoregression-Adjusted Batch Means Procedure for Simulation Analysis," INFORMS Journal on Computing, INFORMS, vol. 23(2), pages 297-314, May.
    3. Emily K. Lada & James R. Wilson & Natalie M. Steiger & Jeffrey A. Joines, 2007. "Performance of a Wavelet-Based Spectral Procedure for Steady-State Simulation Analysis," INFORMS Journal on Computing, INFORMS, vol. 19(2), pages 150-160, May.
    4. Yuanhui Zhang & Haipeng Wu & Brian T. Denton & James R. Wilson & Jennifer M. Lobo, 2019. "Probabilistic sensitivity analysis on Markov models with uncertain transition probabilities: an application in evaluating treatment decisions for type 2 diabetes," Health Care Management Science, Springer, vol. 22(1), pages 34-52, March.
    5. Barry L. Nelson, 2004. "50th Anniversary Article: Stochastic Simulation Research in Management Science," Management Science, INFORMS, vol. 50(7), pages 855-868, July.
    6. Dashi I. Singham & Lee W. Schruben, 2012. "Finite-Sample Performance of Absolute Precision Stopping Rules," INFORMS Journal on Computing, INFORMS, vol. 24(4), pages 624-635, November.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Song, Wheyming Tina & Chih, Mingchang, 2013. "Run length not required: Optimal-mse dynamic batch means estimators for steady-state simulations," European Journal of Operational Research, Elsevier, vol. 229(1), pages 114-123.
    2. Song, Wheyming T. & Chih, Mingchang, 2010. "Extended dynamic partial-overlapping batch means estimators for steady-state simulations," European Journal of Operational Research, Elsevier, vol. 203(3), pages 640-651, June.
    3. John R. Birge, 2023. "Uses of Sub-sample Estimates to Reduce Errors in Stochastic Optimization Models," Papers 2310.07052, arXiv.org.
    4. Lada, Emily K. & Wilson, James R., 2006. "A wavelet-based spectral procedure for steady-state simulation analysis," European Journal of Operational Research, Elsevier, vol. 174(3), pages 1769-1801, November.
    5. Vandin, Andrea & Giachini, Daniele & Lamperti, Francesco & Chiaromonte, Francesca, 2022. "Automated and distributed statistical analysis of economic agent-based models," Journal of Economic Dynamics and Control, Elsevier, vol. 143(C).
    6. Andrea Vandin & Daniele Giachini & Francesco Lamperti & Francesca Chiaromonte, 2021. "Automated and Distributed Statistical Analysis of Economic Agent-Based Models," Papers 2102.05405, arXiv.org, revised Nov 2023.
    7. Ali Tafazzoli & James R. Wilson & Emily K. Lada & Natalie M. Steiger, 2011. "Performance of Skart: A Skewness- and Autoregression-Adjusted Batch Means Procedure for Simulation Analysis," INFORMS Journal on Computing, INFORMS, vol. 23(2), pages 297-314, May.
    8. Andrea Vandin & Daniele Giachini & Francesco Lamperti & Francesca Chiaromonte, 2020. "Automated and Distributed Statistical Analysis of Economic Agent-Based Models," LEM Papers Series 2020/31, Laboratory of Economics and Management (LEM), Sant'Anna School of Advanced Studies, Pisa, Italy.
    9. Mingchang Chih, 2019. "An Insight into the Data Structure of the Dynamic Batch Means Algorithm with Binary Tree Code," Mathematics, MDPI, vol. 7(9), pages 1-8, August.
    10. Munoz, F.D. & Hobbs, B.F. & Watson, J.-P., 2016. "New bounding and decomposition approaches for MILP investment problems: Multi-area transmission and generation planning under policy constraints," European Journal of Operational Research, Elsevier, vol. 248(3), pages 888-898.
    11. Enver Yücesan, 1993. "Randomization tests for initialization bias in simulation output," Naval Research Logistics (NRL), John Wiley & Sons, vol. 40(5), pages 643-663, August.
    12. Robinson, Stewart, 2007. "A statistical process control approach to selecting a warm-up period for a discrete-event simulation," European Journal of Operational Research, Elsevier, vol. 176(1), pages 332-346, January.
    13. Kleijnen, J.P.C. & Sanchez, S.M. & Lucas, T.W. & Cioppa, T.M., 2003. "A User's Guide to the Brave New World of Designing Simulation Experiments," Discussion Paper 2003-1, Tilburg University, Center for Economic Research.
    14. Driessen, M.A. & van Houtum, G.J. & Zijm, W.H.M. & Rustenburg, W.D., 2020. "Capacity assignment in repair shops with high material uncertainty," International Journal of Production Economics, Elsevier, vol. 221(C).
    15. Logendran, Rasaratnam & Talkington, Diane, 1997. "Analysis of cellular and functional manufacturing systems in the presence of machine breakdown," International Journal of Production Economics, Elsevier, vol. 53(3), pages 239-256, December.
    16. Srinagesh Gavirneni & Douglas J. Morrice & Peter Mullarkey, 2004. "Simulation Helps Maxager Shorten Its Sales Cycle," Interfaces, INFORMS, vol. 34(2), pages 87-96, April.
    17. K Hoad & S Robinson & R Davies, 2010. "Automating warm-up length estimation," Journal of the Operational Research Society, Palgrave Macmillan;The OR Society, vol. 61(9), pages 1389-1403, September.
    18. Kleijnen, Jack P.C., 1992. "Sensitivity analysis of simulation experiments: regression analysis and statistical design," Mathematics and Computers in Simulation (MATCOM), Elsevier, vol. 34(3), pages 297-315.
    19. Joshi, Shirish & Tew, Jeffrey D., 1995. "Validation and statistical analysis procedures under the common random number correlation-induction strategy for multipopulation simulation experiments," European Journal of Operational Research, Elsevier, vol. 85(1), pages 205-220, August.
    20. David Goldsman & Seong-Hee Kim & William S. Marshall & Barry L. Nelson, 2002. "Ranking and Selection for Steady-State Simulation: Procedures and Perspectives," INFORMS Journal on Computing, INFORMS, vol. 14(1), pages 2-19, February.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:inm:ormnsc:v:48:y:2002:i:12:p:1569-1586. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Asher (email available below). General contact details of provider: https://edirc.repec.org/data/inforea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.