IDEAS home Printed from https://ideas.repec.org/a/spr/metcap/v20y2018i4d10.1007_s11009-017-9599-7.html
   My bibliography  Save this article

Uncertainty Quantification of Stochastic Simulation for Black-box Computer Experiments

Author

Listed:
  • Youngjun Choe

    (University of Washington)

  • Henry Lam

    (Columbia University)

  • Eunshin Byon

    (University of Michigan)

Abstract

Stochastic simulations applied to black-box computer experiments are becoming more widely used to evaluate the reliability of systems. Yet, the reliability evaluation or computer experiments involving many replications of simulations can take significant computational resources as simulators become more realistic. To speed up, importance sampling coupled with near-optimal sampling allocation for these experiments is recently proposed to efficiently estimate the probability associated with the stochastic system output. In this study, we establish the central limit theorem for the probability estimator from such procedure and construct an asymptotically valid confidence interval to quantify estimation uncertainty. We apply the proposed approach to a numerical example and present a case study for evaluating the structural reliability of a wind turbine.

Suggested Citation

  • Youngjun Choe & Henry Lam & Eunshin Byon, 2018. "Uncertainty Quantification of Stochastic Simulation for Black-box Computer Experiments," Methodology and Computing in Applied Probability, Springer, vol. 20(4), pages 1155-1172, December.
    • Handle: RePEc:spr:metcap:v:20:y:2018:i:4:d:10.1007_s11009-017-9599-7
    DOI: 10.1007/s11009-017-9599-7
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11009-017-9599-7
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11009-017-9599-7?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Reuven Rubinstein, 1999. "The Cross-Entropy Method for Combinatorial and Continuous Optimization," Methodology and Computing in Applied Probability, Springer, vol. 1(2), pages 127-190, September.
    2. Jose Blanchet & Henry Lam, 2014. "Rare-Event Simulation for Many-Server Queues," Mathematics of Operations Research, INFORMS, vol. 39(4), pages 1142-1178, November.
    3. Andrea Staid & Seth Guikema & Roshanak Nateghi & Steven Quiring & Michael Gao, 2014. "Simulation of tropical cyclone impacts to the U.S. power system under climate change scenarios," Climatic Change, Springer, vol. 127(3), pages 535-546, December.
    4. Bruce Ankenman & Barry L. Nelson & Jeremy Staum, 2010. "Stochastic Kriging for Simulation Metamodeling," Operations Research, INFORMS, vol. 58(2), pages 371-382, April.
    5. H. Kahn & A. W. Marshall, 1953. "Methods of Reducing Sample Size in Monte Carlo Computations," Operations Research, INFORMS, vol. 1(5), pages 263-278, November.
    6. Paul Glasserman & Jingyi Li, 2005. "Importance Sampling for Portfolio Credit Risk," Management Science, INFORMS, vol. 51(11), pages 1643-1656, November.
    7. Koopman, Siem Jan & Shephard, Neil & Creal, Drew, 2009. "Testing the assumptions behind importance sampling," Journal of Econometrics, Elsevier, vol. 149(1), pages 2-11, April.
    8. Reiichiro Kawai, 2008. "Adaptive Monte Carlo Variance Reduction for Lévy Processes with Two-Time-Scale Stochastic Approximation," Methodology and Computing in Applied Probability, Springer, vol. 10(2), pages 199-223, June.
    9. Mathieu Balesdent & Jérôme Morio & Loïc Brevault, 2016. "Rare Event Probability Estimation in the Presence of Epistemic Uncertainty on Input Probability Distribution Parameters," Methodology and Computing in Applied Probability, Springer, vol. 18(1), pages 197-216, March.
    10. Roussas, G. G., 1994. "Asymptotic Normality of Random Fields of Positively or Negatively Associated Processes," Journal of Multivariate Analysis, Elsevier, vol. 50(1), pages 152-173, July.
    11. Ning Zhang & Daniel W. Apley, 2016. "Brownian Integrated Covariance Functions for Gaussian Process Modeling: Sigmoidal Versus Localized Basis Functions," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(515), pages 1182-1195, July.
    12. Peter W. Glynn & Donald L. Iglehart, 1989. "Importance Sampling for Stochastic Simulations," Management Science, INFORMS, vol. 35(11), pages 1367-1392, November.
    13. Yunpeng Sun & Daniel W. Apley & Jeremy Staum, 2011. "Efficient Nested Simulation for Estimating the Variance of a Conditional Expectation," Operations Research, INFORMS, vol. 59(4), pages 998-1007, August.
    14. R. A. Rigby & D. M. Stasinopoulos, 2005. "Generalized additive models for location, scale and shape," Journal of the Royal Statistical Society Series C, Royal Statistical Society, vol. 54(3), pages 507-554, June.
    15. Zdravko I. Botev & Dirk P. Kroese, 2008. "An Efficient Algorithm for Rare-event Probability Estimation, Combinatorial Optimization, and Counting," Methodology and Computing in Applied Probability, Springer, vol. 10(4), pages 471-505, December.
    16. L. Jeff Hong & Sandeep Juneja & Guangwu Liu, 2017. "Kernel Smoothing for Nested Estimation with Application to Portfolio Risk Measurement," Operations Research, INFORMS, vol. 65(3), pages 657-673, June.
    17. Riccardo Gatto & Benjamin Baumgartner, 2016. "Saddlepoint Approximations to the Probability of Ruin in Finite Time for the Compound Poisson Risk Process Perturbed by Diffusion," Methodology and Computing in Applied Probability, Springer, vol. 18(1), pages 217-235, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Jeong, Cheoljoon & Byon, Eunshin, 2024. "Calibration of building energy computer models via bias-corrected iteratively reweighted least squares method," Applied Energy, Elsevier, vol. 360(C).
    2. Matieyendou Lamboni, 2023. "On Exact Distribution for Multivariate Weighted Distributions and Classification," Methodology and Computing in Applied Probability, Springer, vol. 25(1), pages 1-26, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cao, Quoc Dung & Choe, Youngjun, 2019. "Cross-entropy based importance sampling for stochastic simulation models," Reliability Engineering and System Safety, Elsevier, vol. 191(C).
    2. Qiyun Pan & Eunshin Byon & Young Myoung Ko & Henry Lam, 2020. "Adaptive importance sampling for extreme quantile estimation with stochastic black box computer models," Naval Research Logistics (NRL), John Wiley & Sons, vol. 67(7), pages 524-547, October.
    3. Tito Homem-de-Mello, 2007. "A Study on the Cross-Entropy Method for Rare-Event Probability Estimation," INFORMS Journal on Computing, INFORMS, vol. 19(3), pages 381-394, August.
    4. Guangxin Jiang & L. Jeff Hong & Barry L. Nelson, 2020. "Online Risk Monitoring Using Offline Simulation," INFORMS Journal on Computing, INFORMS, vol. 32(2), pages 356-375, April.
    5. David J. Eckman & Shane G. Henderson & Sara Shashaani, 2023. "Diagnostic Tools for Evaluating and Comparing Simulation-Optimization Algorithms," INFORMS Journal on Computing, INFORMS, vol. 35(2), pages 350-367, March.
    6. Mingbin Ben Feng & Eunhye Song, 2020. "Efficient Nested Simulation Experiment Design via the Likelihood Ratio Method," Papers 2008.13087, arXiv.org, revised May 2024.
    7. Kamiński, Bogumił, 2015. "A method for the updating of stochastic kriging metamodels," European Journal of Operational Research, Elsevier, vol. 247(3), pages 859-866.
    8. Kun Zhang & Ben Mingbin Feng & Guangwu Liu & Shiyu Wang, 2022. "Sample Recycling for Nested Simulation with Application in Portfolio Risk Measurement," Papers 2203.15929, arXiv.org.
    9. Ali Eshragh & Jerzy Filar & Michael Haythorpe, 2011. "A hybrid simulation-optimization algorithm for the Hamiltonian cycle problem," Annals of Operations Research, Springer, vol. 189(1), pages 103-125, September.
    10. Liu, Xiaoyu & Yan, Xing & Zhang, Kun, 2024. "Kernel quantile estimators for nested simulation with application to portfolio value-at-risk measurement," European Journal of Operational Research, Elsevier, vol. 312(3), pages 1168-1177.
    11. Hatem Çoban & İpek Deveci Kocakoç & Şemsettin Erken & Mehmet Akif Aksoy, 2019. "Reducing Variation of Risk Estimation by Using Importance Sampling," Alphanumeric Journal, Bahadir Fatih Yildirim, vol. 7(2), pages 173-184, December.
    12. M. Garvels, 2011. "A combined splitting—cross entropy method for rare-event probability estimation of queueing networks," Annals of Operations Research, Springer, vol. 189(1), pages 167-185, September.
    13. Peter Salemi & Jeremy Staum & Barry L. Nelson, 2019. "Generalized Integrated Brownian Fields for Simulation Metamodeling," Operations Research, INFORMS, vol. 67(3), pages 874-891, May.
    14. Matti Vihola & Jouni Helske & Jordan Franks, 2020. "Importance sampling type estimators based on approximate marginal Markov chain Monte Carlo," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 47(4), pages 1339-1376, December.
    15. Hsieh, Ming-Hua & Lee, Yi-Hsi & Shyu, So-De & Chiu, Yu-Fen, 2019. "Estimating multifactor portfolio credit risk: A variance reduction approach," Pacific-Basin Finance Journal, Elsevier, vol. 57(C).
    16. Joshua Chan & Dirk Kroese, 2011. "Rare-event probability estimation with conditional Monte Carlo," Annals of Operations Research, Springer, vol. 189(1), pages 43-61, September.
    17. Wang, Tianxiang & Xu, Jie & Hu, Jian-Qiang & Chen, Chun-Hung, 2023. "Efficient estimation of a risk measure requiring two-stage simulation optimization," European Journal of Operational Research, Elsevier, vol. 305(3), pages 1355-1365.
    18. Han, Chulwoo & Park, Frank C., 2022. "A geometric framework for covariance dynamics," Journal of Banking & Finance, Elsevier, vol. 134(C).
    19. Yixuan Wang & Jianzhu Li & Ping Feng & Rong Hu, 2015. "A Time-Dependent Drought Index for Non-Stationary Precipitation Series," Water Resources Management: An International Journal, Published for the European Water Resources Association (EWRA), Springer;European Water Resources Association (EWRA), vol. 29(15), pages 5631-5647, December.
    20. Gerardo Manzo & Antonio Picca, 2020. "The Impact of Sovereign Shocks," Management Science, INFORMS, vol. 66(7), pages 3113-3132, July.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:metcap:v:20:y:2018:i:4:d:10.1007_s11009-017-9599-7. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.